diff --git a/.dockerignore b/.dockerignore index 1e5cd1d7f..e8689d0b0 100644 --- a/.dockerignore +++ b/.dockerignore @@ -18,12 +18,11 @@ !.curlrc !LICENSE -# groovy cli +# CLI build !src !pom.xml !mvnw !.mvn -!compiler.groovy # Including .git is risky, but required so maven can read the build number. At least keep it to a minium. !.git/HEAD diff --git a/.editorconfig b/.editorconfig index f21bf4c48..96cde9a3b 100644 --- a/.editorconfig +++ b/.editorconfig @@ -1,27 +1,58 @@ [*] charset = utf-8 end_of_line = lf -ij_formatter_off_tag = @formatter:off,<#if -ij_formatter_on_tag = @formatter:on, ` branches for new functionality + (e.g. `feature/add-user-authentication`). +- Use `fix/` branches for bug fixes in existing code + (e.g. `fix/login-button-not-working`). +- Use `hotfix/` branches for critical fixes against `main` + (e.g. `hotfix/security-patch-for-cve-2024-1234`). + +## Commit Guidelines + +- **Write small, focused commits.** Each commit should contain a single, logical + change. Avoid bundling unrelated changes together. +- **Commit frequently.** Frequent commits with clear messages make rollbacks and + history easier to follow. +- **Write meaningful commit messages.** The diff already shows *what* changed; the + commit message should explain *why*. Write it so an outside developer can + understand it without additional context. + + ``` + # bad + "Add method validate" + + # good + "Validate feature configuration before enabling to prevent runtime errors in production" + ``` + +- **Write commit messages in English.** +- **Use semantic versioning for tags.** Follow [SemVer](https://semver.org/) for + release tags (e.g. `v1.0.0`, `v2.3.1`). +- **Run a linter before committing or pushing** (e.g. Checkstyle or PMD) to catch + style issues and common bugs automatically. + +## Pull Requests + +- **Keep PRs small.** Large pull requests are hard to merge and either stall in + review or get rubber-stamped without a real look. +- **Provide context in the description.** Explain the goal of the PR and how the + change can be tested — it helps reviewers understand the code faster. +- **Ensure CI is green** before requesting review and before merging. +- **Use descriptive PR titles.** PR titles are often used to generate changelogs, so + avoid vague titles like "fix bug" or "refactoring". Prefer precise titles such as + `fix: resolve memory leak in session management`. +- **Prefer merge commits (`--no-ff`) over squash** when merging, so the full history + of development steps stays traceable. Squashing is acceptable for hotfixes or small + changes to keep the history clean. + +## Code Style (Java) + +- Use `camelCase` for variable and method names, `PascalCase` for class and enum + names, and tabs for indentation. +- Use descriptive names for variables and methods — verbose, speaking names help the + reader understand code faster and should reveal *what* a method does, not *how*. +- Use explicit typing; avoid overusing `var`. Only use `var` when the type is already + obvious from the right-hand side (e.g. `var client = new HttpClientFactory()`); + spell out the type when it comes from a method call or generic expression whose + return type isn't visible at the call site. + + ```java + // bad + var result = repository.find(id); + + // good + Optional result = repository.find(id); + ``` + +- Use named lambda parameters instead of single letters, especially in nested streams. + + ```java + // bad + users.stream().filter(u -> u.isActive()).forEach(u -> u.sendNotification()); + + // good + users.stream() + .filter(user -> user.isActive()) + .forEach(activeUser -> activeUser.sendNotification()); + ``` + +- Use `Optional` only for genuinely optional values — reserve it for values that + are legitimately absent (e.g. a lookup that may find nothing), and use + `Objects.requireNonNull()` / fail-fast validation for values that must always be + present. Don't wrap required fields in `Optional` just to avoid a null check. Once a + value is an `Optional`, unwrap it with `orElse`/`orElseGet` (or a ternary for + plain nullable references) rather than calling `.get()` behind a null check. + + ```java + // bad (address must always be present) + String zip = user.getAddress() == null ? null : user.getAddress().getZipCode(); + + // good (address is genuinely optional) + Optional
address = user.getAddress(); + String zip = address.map(Address::getZipCode).orElse(null); + ``` + + ```java + // bad + String name = user.getName() != null ? user.getName() : "Default"; + + // good (Optional-based) + String name = Optional.ofNullable(user.getName()).orElse("Default"); + ``` + +- Prefer builders over long constructors once a type has more than 2-3 fields, so call + sites read like named arguments (we use Lombok's `@Builder`). Whenever a fluent + chain (builder or otherwise) exceeds 2-3 calls, put each call on its own line for + readability. + + ```java + // bad + Config config = new Config(host, port, true, false, null, retries); + config.setHost("localhost").setPort(8080).setEnabled(true).setDebug(false); + + // good + Config config = Config.builder() + .host(host) + .port(port) + .enabled(true) + .build(); + + config.setHost("localhost") + .setPort(8080) + .setEnabled(true) + .setDebug(false); + ``` + +- Use comments to explain *why* code does something, not *what* it does. What the + code does should already be self-explanatory. + + ```java + // bad + // set retry to 3 + int retryCount = 3; + + // good + // we use 3 retries because the external API is unstable + int retryCount = 3; + ``` + +- **Fail fast** and **use defensive programming** — validate inputs up front and + return safe defaults instead of propagating `null`. + + ```java + // fail fast + void processOrder(Order order) { + if (order == null) { + throw new IllegalArgumentException("Order must not be null"); + } + // ... logic + } + + // defensive programming + List getTags(User user) { + if (user.getTags() == null) { + return List.of(); + } + return user.getTags(); + } + ``` + +- Keep classes and methods small and focused, following the single responsibility + principle. + + ```java + // bad + class OrderManager { + void processOrder(Order order) { /* ... */ } + void sendEmail(String recipient, String message) { /* ... */ } + void saveToDatabase(Order order) { /* ... */ } + } + + // good + class OrderService { + void processOrder(Order order) { /* ... */ } + } + + class EmailService { + void sendEmail(String recipient, String message) { /* ... */ } + } + ``` + +- Use the right exceptions. Prefer specific exceptions over the generic + `RuntimeException`/`Exception`, and create custom exception classes where the + context calls for it. + + ```java + // bad + throw new RuntimeException("Order not found"); + + // good + throw new OrderNotFoundException("Order with ID " + orderId + " not found"); + ``` + +- Avoid deeply nested code. Use guard clauses and fail-fast to keep nesting depth low. + + ```java + // bad + void process(User user) { + if (user != null) { + if (user.isActive()) { + // ... a lot of logic + } + } + } + + // good + void process(User user) { + if (user == null || !user.isActive()) { + return; + } + // ... a lot of logic + } + ``` + +- Obey the boy scout rule: "Always leave the campground cleaner than you found it." + When you touch a file, fix small messes (typos, formatting) in the immediate area of + your change. + +- Prefer text blocks / `String.format` over ad hoc concatenation when building + strings — plain literals for static text, `String.format(...)` or text blocks + (`"""..."""`, Java 15+) when you actually need to build a string from parts. + + ```java + // good + String constant = "I am a static string"; + String dynamic = String.format("I am dynamic: %s", constant); + ``` + +- Avoid runtime metaprogramming and reflection. Reflection-based frameworks and + dynamic proxies bypass compile-time type checking, hurt performance, and are + fragile at runtime. Prefer direct API calls, interfaces, or + composition/polymorphism. + + ```java + // bad + Method method = UserService.class.getDeclaredMethod("doSomething"); + method.invoke(userService); + + // good + userService.doSomething(); + ``` + +- Use `@Slf4j` for logging. Lombok's `@Slf4j` annotation injects a + `private static final Logger log` field — don't hand-declare a `Logger` via + `LoggerFactory.getLogger(...)` for a class's own logging. (A deliberately-named, + cross-cutting logger not tied to the enclosing class name is a legitimate + exception, since `@Slf4j` can only produce a logger named after the class.) + + ```java + // bad + import org.slf4j.Logger; + import org.slf4j.LoggerFactory; + + class UserService { + private static final Logger log = LoggerFactory.getLogger(UserService.class); + } + + // good + import lombok.extern.slf4j.Slf4j; + + @Slf4j + class UserService { + void doSomething() { + log.info("Doing something..."); + } + } + ``` + +- Use uniform logging levels, consistently and deliberately, to keep log volume and + readability sane in production: + - `debug` / `trace`: detailed diagnostic info for development-time troubleshooting + (e.g. method parameters, loop iterations). + - `info`: important, business-critical or systemic milestones (e.g. successful + startup, completed transaction). Don't overuse. + - `warn`: unexpected situations that don't block the flow (e.g. fallbacks, use of + deprecated APIs, transient connection errors). + - `error`: errors that require aborting or manual intervention (e.g. caught + exceptions, system failures). + +## Testing + +We use JUnit 5 and Mockito. + +- Use descriptive test class names (e.g. `UserServiceTest`). +- Structure tests with given-when-then / arrange-act-assert comments. +- Use `@ParameterizedTest` (with `@ValueSource`, `@CsvSource` or `@MethodSource`) for + data-driven tests. +- Mock external dependencies using Mockito's `@Mock` / `Mockito.mock(...)`. + +```java +class CalculatorTest { + + private final Calculator calculator = new Calculator(); + + @ParameterizedTest + @CsvSource({ + "5, 10, 15", + "0, 0, 0", + "-3, 3, 0" + }) + void shouldCalculateSumCorrectly(int a, int b, int expected) { + // when + int result = calculator.add(a, b); + + // then + assertEquals(expected, result); + } +} +``` + +```java +@ExtendWith(MockitoExtension.class) +class OrderServiceTest { + + @Mock + private OrderRepository orderRepository; + + @InjectMocks + private OrderService orderService; + + @Test + void shouldThrowWhenOrderNotFound() { + // given + when(orderRepository.findById("123")).thenReturn(Optional.empty()); + + // when / then + assertThrows(OrderNotFoundException.class, () -> orderService.getOrder("123")); + } +} +``` + +## Code Review Etiquette + +- **Be constructive and respectful.** Criticize the code, not the author. Phrase + suggestions as questions or ideas (e.g. "Have you considered...?" instead of + "This is wrong"). +- **Praise good code.** If you see a particularly elegant solution, say so — reviews + are also a place to learn and to give positive feedback. diff --git a/Dockerfile b/Dockerfile index a2e036044..6535c85dd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,7 +2,7 @@ # BUILD ARGUMENTS # ============================================================================ # Keep in sync with the versions in pom.xml -ARG JDK_VERSION='17' +ARG JDK_VERSION='25' # ============================================================================ # STAGE 1: Maven Dependency Cache @@ -33,7 +33,6 @@ COPY --from=maven-cache /mvn/ /mvn/ COPY --from=maven-cache /app/ /app COPY src/main /app/src/main -COPY compiler.groovy /app COPY .git /app/.git WORKDIR /app @@ -58,8 +57,9 @@ RUN apk add curl grep # 3.1: Version Configuration # ----------------------------------------------------------------------------- -# When updating Helm, also upgrade helm image in Config.groovy -ARG HELM_VERSION=4.2.1 +# When updating Helm, also upgrade the helm chart version in Config.java +# renovate: depName=helm/helm datasource=github-releases +ARG HELM_VERSION=4.2.4 # Install additional tools required for downloads # bash curl unzip required for Jenkins downloader @@ -133,10 +133,10 @@ RUN /jenkins/download-plugins.sh /dist/gitops/jenkins-plugins # ----------------------------------------------------------------------------- # 3.7: Download Helm Charts # ----------------------------------------------------------------------------- -COPY src/main/groovy/com/cloudogu/gitops/config/Config.groovy /tmp/ -COPY src/main/groovy/com/cloudogu/gitops/config/scm/ScmTenantSchema.groovy /tmp/ +COPY src/main/java/com/cloudogu/gitops/config/Config.java /tmp/ +COPY src/main/java/com/cloudogu/gitops/config/scm/ScmTenantSchema.java /tmp/ COPY scripts/downloadHelmCharts.sh /tmp/ -RUN cd /dist/gitops && /tmp/downloadHelmCharts.sh /tmp/Config.groovy /tmp/ScmTenantSchema.groovy +RUN cd /dist/gitops && /tmp/downloadHelmCharts.sh /tmp/Config.java /tmp/ScmTenantSchema.java # ----------------------------------------------------------------------------- # 3.8: Prepare Application Files diff --git a/Jenkinsfile b/Jenkinsfile index d8c956bb2..307b2d537 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -16,7 +16,7 @@ pipeline { parameters { booleanParam(defaultValue: false, name: 'forcePushImage', description: 'Pushes the image with the current git commit as tag, even when it is on a branch') booleanParam(defaultValue: false, name: 'noCache', description: 'Builds the docker image without cache') - choice(name: 'chooseProfile', choices: ['full', 'minimal', 'all-profiles', 'full-prefix', 'content-examples', 'operator-full','operator-mandants'], description: 'Starts GOP with given profile only and execute tests which belongs to profile.') + choice(name: 'chooseProfile', choices: ['full', 'full-secrets', 'minimal', 'all-profiles', 'full-prefix', 'content-examples', 'operator-full', 'operator-mandants'], description: 'Starts GOP with given profile only and execute tests which belongs to profile.') } environment { @@ -24,7 +24,7 @@ pipeline { BUILD_GROUP = sh(script: 'getent group docker | cut -d: -f3', returnStdout: true).trim() DOCKER_REGISTRY_BASE_URL = 'ghcr.io' DOCKER_IMAGE_NAME = 'cloudogu/gitops-playground' - MAVEN_IMAGE = 'maven:3-eclipse-temurin-17' + MAVEN_IMAGE = 'maven:3-eclipse-temurin-25' GRYPE_IMAGE = 'anchore/grype:v0.109.1' SYFT_IMAGE = 'anchore/syft:v1.42.2' GOLANG_IMAGE = 'golang:1.25-alpine' @@ -43,19 +43,30 @@ pipeline { parallel { - stage("Build CLI") { - agent { docker { - image "${env.MAVEN_IMAGE}" - args "-v maven-cache:/root/.m2" - reuseNode true - }} + stage("Test & SonarScanner") { + agent { + docker { + image "${env.MAVEN_IMAGE}" + args "-e HOME=${env.WORKSPACE}/.maven-home" + reuseNode true + } + } steps { - sh 'mvn -B clean test' + withSonarQubeEnv('ces-sonar') { + sh ''' + mkdir -p "$WORKSPACE/.maven-home/.m2/repository" + + mvn -B \ + -Dmaven.repo.local="$WORKSPACE/.maven-home/.m2/repository" \ + clean verify sonar:sonar \ + -Dsonar.projectKey=gitops-playground \ + -Dsonar.branch.name="$BRANCH_NAME" + ''' + } } post { always { junit testResults: '**/target/surefire-reports/TEST-*.xml' - archiveArtifacts artifacts: "**/target/site/jacoco/**" } } } @@ -63,9 +74,9 @@ pipeline { stage("Build Image") { steps { script { - def buildArgs = "--no-cache " + - "--build-arg BUILD_DATE='${env.BUILD_DATE}' " + - "--build-arg VCS_REF='${env.GIT_COMMIT}' " + def buildArgs = (params.noCache ? "--no-cache " : "") + + "--build-arg BUILD_DATE='${env.BUILD_DATE}' " + + "--build-arg VCS_REF='${env.GIT_COMMIT}' " docker.build(env.FULL_IMAGE_TAG, "${buildArgs} .") } } @@ -84,14 +95,22 @@ pipeline { -u :$BUILD_GROUP \ -e NO_COLOR=1 \ $SYFT_IMAGE --output syft-table=/workspace/sbom.txt --output spdx-json=/workspace/sbom.json --quiet $FULL_IMAGE_TAG''' - sh '''docker run --rm -v $WORKSPACE:/workspace \ + + catchError( + buildResult: 'SUCCESS', + stageResult: 'UNSTABLE', + catchInterruptions: false + ) { + sh '''docker run --rm -v $WORKSPACE:/workspace \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ -u :$BUILD_GROUP \ -e NO_COLOR=1 \ $GRYPE_IMAGE sbom:/workspace/sbom.json \ --output table=/workspace/vulnerabilities.txt \ --output sarif=/workspace/vulnerabilities.sarif \ - --quiet --sort-by severity --fail-on critical''' + --sort-by severity --fail-on critical''' + } + archiveArtifacts artifacts: 'sbom.*, vulnerabilities.*' } } @@ -100,10 +119,9 @@ pipeline { steps { script { def profiles = [] - def isTriggeredByTimer = currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').size() > 0 - if (isTriggeredByTimer || params.chooseProfile == 'all-profiles' || env.BRANCH_NAME == 'main') { - profiles = ['minimal', 'full', 'full-prefix', 'content-examples', 'operator-full','operator-mandants'] + if (isTriggeredByTimer() || params.chooseProfile == 'all-profiles' || env.BRANCH_NAME == 'main') { + profiles = ['minimal', 'full', 'full-secrets', 'full-prefix', 'content-examples', 'operator-full', 'operator-mandants'] } else if (env.BRANCH_NAME == 'develop') { profiles = ['full-prefix', 'operator-mandants', 'operator-full'] } else { @@ -141,29 +159,42 @@ pipeline { kubectl logs -n "\${namespace}" "\${pod}" --all-containers=true --previous --tail=200 --prefix=true >> '${dumpDir}/container-logs.txt' 2>&1 echo >> '${dumpDir}/container-logs.txt' done + + chown -R ${env.BUILD_USER}:${env.BUILD_GROUP} target """, returnStatus: true) } - archiveArtifacts artifacts: "${dumpDir}/**", allowEmptyArchive: true - } + archiveArtifacts artifacts: "${dumpDir}/**", allowEmptyArchive: true + } def withK3dCluster = { profile, body -> try { sh "yes | KUBECONFIG=${env.WORKSPACE}/.kubeconfig.yaml ./scripts/init-cluster.sh --cluster-name=${env.K3D_CLUSTER_NAME}" body() - } catch(Throwable t) { + } catch (Throwable t) { dumpKubernetesDebugInfo(profile) throw t } finally { sh "KUBECONFIG=${env.WORKSPACE}/.kubeconfig.yaml $HOME/.local/bin/k3d cluster delete ${env.K3D_CLUSTER_NAME}" - }} + } + } profiles.each { profile -> withK3dCluster(profile) { + if (profile == 'full-secrets') { + docker.image("${env.GOLANG_IMAGE}").inside(env.INTEGRATION_TEST_DOCKER_ARGS) { + sh ''' + apk add --no-cache kubectl + kubectl create namespace gop-job --dry-run=client -o yaml | kubectl apply -f - + kubectl apply -f ./scripts/dev/gop-secrets.yaml + ''' + } + } + if (profile.startsWith('operator')) { docker.image("${env.GOLANG_IMAGE}").inside(env.INTEGRATION_TEST_DOCKER_ARGS) { - sh 'apk add --no-cache make bash curl git kubectl && ./scripts/local/install-argocd-operator.sh' + sh 'apk add --no-cache make bash curl git kubectl && make install-operator' } } @@ -171,7 +202,11 @@ pipeline { sh "java -jar /app/gitops-playground.jar --profile=${profile}" } docker.image("${env.MAVEN_IMAGE}").inside(env.INTEGRATION_TEST_DOCKER_ARGS) { - sh "mvn -B failsafe:integration-test failsafe:verify -Dmicronaut.environments=${profile} -Dsurefire.reportNameSuffix=${profile} && chown $BUILD_USER:$BUILD_GROUP ./* -R" + try { + sh "mvn -B failsafe:integration-test failsafe:verify -Dmicronaut.environments=${profile} -Dsurefire.reportNameSuffix=${profile}" + } finally { + sh '[ ! -e target ] || chown -R $BUILD_USER:$BUILD_GROUP target' + } } } @@ -223,16 +258,37 @@ pipeline { } post { + always { + script { + if (isTriggeredByTimer()) { + currentBuild.displayName = "#${env.BUILD_NUMBER} weekly" + emailext( + subject: "Weekly build ${currentBuild.currentResult}: ${env.JOB_NAME} #${env.BUILD_NUMBER}", + body: '${SCRIPT, template="groovy-html.template"}', + mimeType: 'text/html', + to: env.GOP_DEVELOPERS + ) + } + } + } changed { - emailext( - subject: "${currentBuild.result}: ${env.JOB_NAME} #${env.BUILD_NUMBER}", - body: '${SCRIPT, template="groovy-html.template"}', - mimeType: 'text/html', - recipientProviders: [ - [$class: 'DevelopersRecipientProvider'], - [$class: 'RequesterRecipientProvider'] - ] - ) + script { + if (!isTriggeredByTimer()) { + emailext( + subject: "${currentBuild.result}: ${env.JOB_NAME} #${env.BUILD_NUMBER}", + body: '${SCRIPT, template="groovy-html.template"}', + mimeType: 'text/html', + recipientProviders: [ + [$class: 'DevelopersRecipientProvider'], + [$class: 'RequesterRecipientProvider'] + ] + ) + } + } } } -} \ No newline at end of file +} + +boolean isTriggeredByTimer() { + return !currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').isEmpty() +} diff --git a/Makefile b/Makefile index f28672522..8bcc8d3e2 100644 --- a/Makefile +++ b/Makefile @@ -8,15 +8,19 @@ help: @grep -E '^[a-zA-Z_-]+:.*##' $(MAKEFILE_LIST) | \ awk -F '##' '{printf " %-15s %s\n", $$1, $$2}' -.PHONY: +.PHONY: prepare-airgapped-cluster prepare-airgapped-cluster: ## for airgapped-tests ./scripts/dev/prepare_airgapped_cluster.sh -.PHONY: +.PHONY: cluster cluster: ## creates a k3d cluster suitable for GOP ./scripts/init-cluster.sh $(RUN_ARGS) -.PHONY: +.PHONY: keycloak +keycloak: ## installs local Keycloak test instance for OIDC + bash ./scripts/keycloak/install-keycloak.sh + +.PHONY: prepare-two-registries prepare-two-registries: ## for testing with multiple registries ./scripts/dev/prepare_two_registries.sh @@ -30,5 +34,12 @@ image: ## builds the docker image for local testing docker buildx prune -f && docker build . -t local/gop echo "created docker image local/gop" +.PHONY: gop-config-in-secrets +gop-config-in-secrets: ## creates a local cluster with test credentials stored in Kubernetes Secrets + ./scripts/init-cluster.sh + kubectl create namespace gop-job --dry-run=client -o yaml | kubectl apply -f - + kubectl apply -f ./scripts/dev/gop-secrets.yaml + echo "created cluster with GOP test credentials in Kubernetes Secrets" + %: @: diff --git a/argocd/cluster-resources/apps/argocd/argocd/templates/allow-namespaces.ftl.yaml b/argocd/cluster-resources/apps/argocd/argocd/templates/allow-namespaces.ftl.yaml index 5fe61860f..7390eaa26 100644 --- a/argocd/cluster-resources/apps/argocd/argocd/templates/allow-namespaces.ftl.yaml +++ b/argocd/cluster-resources/apps/argocd/argocd/templates/allow-namespaces.ftl.yaml @@ -3,7 +3,7 @@ apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-argocd-access-scmm - namespace: ${config.application.namePrefix}${config.scm.scmManager.namespace} + namespace: ${config.scm.scmManager.namespace} spec: podSelector: matchLabels: diff --git a/argocd/cluster-resources/apps/argocd/argocd/values.ftl.yaml b/argocd/cluster-resources/apps/argocd/argocd/values.ftl.yaml index 78e253a3d..19aa15f75 100644 --- a/argocd/cluster-resources/apps/argocd/argocd/values.ftl.yaml +++ b/argocd/cluster-resources/apps/argocd/argocd/values.ftl.yaml @@ -4,8 +4,9 @@ argo-cd: crds: install: false -<#if config.application.netpols == true> global: + domain: ${argocd.host} +<#if config.application.netpols == true> networkPolicy: create: true @@ -61,7 +62,7 @@ argo-cd: application.namespaces: "${config.application.namePrefix}${config.features.argocd.namespace}" server.insecure: true # tls terminated in ingress - # Repo credential templates are created dynamically in groovy, so they are not stored in git + # Repo credential templates are created dynamically by GOP, so they are not stored in git #credentialTemplates: # scmm: # url: http://scmm.scm-manager.svc.cluster.local @@ -70,9 +71,31 @@ argo-cd: cm: timeout.reconciliation: 15s repository.check.interval: 30s + application.resourceTrackingMethod: annotation + <#if config.features.argocd.url?has_content && argocd.host?has_content> + url: ${config.features.argocd.url} + additionalUrls: | + - http://${argocd.host} + - https://${argocd.host} + + <#assign argocdOidc = (config.features.argocd.oidc)!{}> + <#if argocdOidc?has_content && argocdOidc.enabled> + oidc.config: | + name: ${(argocdOidc.providerName)!"Keycloak"} + issuer: ${argocdOidc.issuerUrl} + clientID: ${argocdOidc.clientId} + clientSecret: ${argocdOidc.clientSecret} + requestedScopes: [<#list (argocdOidc.scopes!["openid", "profile", "email"]) as scope>"${scope}"<#sep>, ] + + <#if argocdOidc?has_content && argocdOidc.enabled && argocdOidc.adminGroupName?has_content> + rbac: + policy.csv: | + g, ${argocdOidc.adminGroupName}, role:admin + scopes: '[groups]' + notifications: - # secrets are created dynamically in groovy, so they are not stored in git + # secrets are created dynamically by GOP, so they are not stored in git secret: create: false enabled: <#if config.features.mail.active == true>true<#else>false @@ -86,8 +109,8 @@ argo-cd: <#if config.features.mail.smtpAddress?has_content> host: ${config.features.mail.smtpAddress} <#if config.features.mail.smtpPort??>port: ${config.features.mail.smtpPort?c} - <#if config.features.mail.smtpUser?has_content>username: $email-username - <#if config.features.mail.smtpPassword?has_content>password: $email-password + <#if config.features.mail.smtpUserConfigured>username: $email-username + <#if config.features.mail.smtpPasswordConfigured>password: $email-password <#else> host: mail.${config.application.namePrefix}${config.features.monitoring.namespace}.svc.cluster.local port: 1025 @@ -190,4 +213,4 @@ argo-cd: - app-sync-status-longer-10s when: app.status.operationState.phase in ['Running'] and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Seconds() >= 10 - + \ No newline at end of file diff --git a/argocd/cluster-resources/apps/argocd/operator/argocd.ftl.yaml b/argocd/cluster-resources/apps/argocd/operator/argocd.ftl.yaml index 5b86a558d..c53e0d4ca 100644 --- a/argocd/cluster-resources/apps/argocd/operator/argocd.ftl.yaml +++ b/argocd/cluster-resources/apps/argocd/operator/argocd.ftl.yaml @@ -4,6 +4,14 @@ metadata: name: argocd namespace: "${config.application.namePrefix}${config.features.argocd.namespace}" spec: + extraConfig: + application.resourceTrackingMethod: annotation + <#if config.features.argocd.url?has_content && argocd.host?has_content> + url: ${config.features.argocd.url} + additionalUrls: | + - http://${argocd.host} + - https://${argocd.host} + applicationSet: enabled: true resources: diff --git a/argocd/cluster-resources/apps/argocd/projects/cluster-resources.ftl.yaml b/argocd/cluster-resources/apps/argocd/projects/cluster-resources.ftl.yaml index f48a8a4cf..49d3b595b 100644 --- a/argocd/cluster-resources/apps/argocd/projects/cluster-resources.ftl.yaml +++ b/argocd/cluster-resources/apps/argocd/projects/cluster-resources.ftl.yaml @@ -28,6 +28,9 @@ spec: - https://helm.releases.hashicorp.com - https://charts.external-secrets.io - https://charts.jetstack.io + - https://charts.jenkins.io + - https://twuni.github.io/docker-registry.helm + - https://packages.scm-manager.org/repository/helm-v2-releases/ <#-- NEW: allow Helm repos from content.helmReleases --> <#if config.content.helmReleases?? && (config.content.helmReleases?size > 0)> diff --git a/argocd/cluster-resources/apps/argocd/projects/default.ftl.yaml b/argocd/cluster-resources/apps/argocd/projects/default.ftl.yaml index 3c9396c1c..5bd48f3b5 100644 --- a/argocd/cluster-resources/apps/argocd/projects/default.ftl.yaml +++ b/argocd/cluster-resources/apps/argocd/projects/default.ftl.yaml @@ -9,7 +9,7 @@ metadata: spec: description: Default fallback AppProject if none other is specified. Is not allowed to do anything. - clusterResourceWhitelist: - destinations: - sourceRepos: - sourceNamespaces: + clusterResourceWhitelist: [ ] + destinations: [ ] + sourceRepos: [ ] + sourceNamespaces: [ ] diff --git a/argocd/cluster-resources/apps/cert-manager/templates/certManager-helm-values.ftl.yaml b/argocd/cluster-resources/apps/cert-manager/templates/values.ftl.yaml similarity index 100% rename from argocd/cluster-resources/apps/cert-manager/templates/certManager-helm-values.ftl.yaml rename to argocd/cluster-resources/apps/cert-manager/templates/values.ftl.yaml diff --git a/argocd/cluster-resources/apps/jenkins/values.ftl.yaml b/argocd/cluster-resources/apps/jenkins/templates/values.ftl.yaml similarity index 64% rename from argocd/cluster-resources/apps/jenkins/values.ftl.yaml rename to argocd/cluster-resources/apps/jenkins/templates/values.ftl.yaml index 9328140bd..bed20795b 100644 --- a/argocd/cluster-resources/apps/jenkins/values.ftl.yaml +++ b/argocd/cluster-resources/apps/jenkins/templates/values.ftl.yaml @@ -1,14 +1,39 @@ dockerClientVersion: ${config.jenkins.internalDockerClientVersion} +<#assign jenkinsOidc = config.jenkins.oidc> +<#if config.jenkins.ingress?has_content> +<#assign jenkinsOidcExternalUrl = "http://" + config.jenkins.ingress> +<#else> +<#assign jenkinsOidcExternalUrl = config.jenkins.url> + controller: +<#if config.jenkins.jenkinsImage?has_content> + <#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> + <#assign jenkinsImage = DockerImageParser.parse(config.jenkins.jenkinsImage)> + image: + registry: ${jenkinsImage.registry} + repository: ${jenkinsImage.repository} + tag: "${jenkinsImage.tag}" +<#else> image: registry: ghcr.io repository: cloudogu/jenkins-helm # The image corresponds to the helm version, # because it contains the default plugins for this particular chart version tag: "${config.jenkins.helm.version}" + +<#if config.registry.createImagePullSecrets == true> + imagePullSecretName: proxy-registry + +<#if jenkinsBootPlugins?has_content> + installPlugins: + <#list jenkinsBootPlugins as plugin> + - ${plugin} + +<#else> installPlugins: false - + + # to prevent the jenkins-ui-test pod being created testEnabled: false @@ -17,19 +42,19 @@ controller: jenkinsUrl: ${config.jenkins.url} -<#if config.application.baseUrl?has_content> + <#if config.application.baseUrl?has_content> ingress: enabled: true hostName: ${config.jenkins.ingress} -<#if config.features.certManager.active!false> - annotations: - cert-manager.io/cluster-issuer: ${config.features.certManager.issuer} - tls: - - secretName: jenkins-tls - hosts: - - ${config.jenkins.ingress} - - + <#if config.features.certManager.active!false> + annotations: + cert-manager.io/cluster-issuer: ${config.features.certManager.issuer} + tls: + - secretName: jenkins-tls + hosts: + - ${config.jenkins.ingress} + + # Don't use controller for builds numExecutors: 0 @@ -45,6 +70,16 @@ controller: existingSecret: jenkins-credentials containerEnv: + - name: GOP_JENKINS_ADMIN_USER + valueFrom: + secretKeyRef: + name: jenkins-credentials + key: jenkins-admin-user + - name: GOP_JENKINS_ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: jenkins-credentials + key: jenkins-admin-password - name: PATH # We already mounted this PATH on the controller-agent. Still, "docker.inside {}" fails in pipeline? # Why? The docker pipeline plugin seems to set an empty environment: https://github.com/jenkinsci/docker-workflow-plugin/blob/docker-workflow-1.25/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java#L261 @@ -78,11 +113,53 @@ controller: rm docker.tgz; find docker -type f -not -name 'docker' -delete; # Delete containerd, etc. We only need the docker CLI - # Note: "wget -O- | tar" leads to the folder being owned by root, even when creating it beforehand?! + # Note: "wget -O- | tar" leads to the folder being owned by root, even when creating it beforehand?! volumeMounts: - name: host-tmp mountPath: /host-tmp + <#if jenkinsBootPlugins?has_content> + JCasC: #<- can be used to configure jenkin + defaultConfig: true + configScripts: + oidc-auth: | + jenkins: + securityRealm: + oic: + clientId: "${jenkinsOidc.clientId}" + clientSecret: "${jenkinsOidc.clientSecret}" + serverConfiguration: + wellKnown: + wellKnownOpenIDConfigurationUrl: "${jenkinsOidc.issuerUrl}/.well-known/openid-configuration" + scopesOverride: "<#list (jenkinsOidc.scopes!["openid", "profile", "email"]) as scope>${scope}<#sep> " + userNameField: "preferred_username" + fullNameFieldName: "name" + emailFieldName: "email" + groupsFieldName: "groups" + logoutFromOpenidProvider: true + postLogoutRedirectUrl: "${jenkinsOidcExternalUrl}" + properties: + - escapeHatch: + username: "${r"${GOP_JENKINS_ADMIN_USER}"}" + <#if jenkinsOidc.adminGroupName?has_content> + group: "${jenkinsOidc.adminGroupName}" + + secret: "${r"${GOP_JENKINS_ADMIN_PASSWORD}"}" + authorizationStrategy: + globalMatrix: + entries: + - user: + name: "${r"${GOP_JENKINS_ADMIN_USER}"}" + permissions: + - "Overall/Administer" + <#if jenkinsOidc.adminGroupName?has_content> + - group: + name: "${jenkinsOidc.adminGroupName}" + permissions: + - "Overall/Administer" + + + persistence: volumes: # Needed for initContainer only diff --git a/argocd/cluster-resources/apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml b/argocd/cluster-resources/apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml index 336230b51..e6ff1ec6c 100644 --- a/argocd/cluster-resources/apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml +++ b/argocd/cluster-resources/apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml @@ -82,7 +82,7 @@ prometheusOperator: namespaces: releaseNamespace: false additional: - <#-- Note that the quotes in the final YAML here are created by groovy, not Freemarker--> + <#-- Note that the quotes in the final YAML here are added during YAML processing, not by Freemarker--> <#if namespaces?has_content> <#list namespaces as namespace> - ${namespace} @@ -141,7 +141,31 @@ kubeProxy: alertmanager: enabled: false grafana: + assertNoLeakedSecrets: false +<#assign grafanaOidc = (config.features.monitoring.oidc)!{}> grafana.ini: +<#if grafanaOidc?has_content && grafanaOidc.enabled> + server: + domain: ${monitoring.grafana.host} + root_url: ${config.features.monitoring.grafanaUrl} + auth.generic_oauth: + enabled: true + name: ${(grafanaOidc.providerName)!"Keycloak"} + allow_sign_up: true + client_id: ${grafanaOidc.clientId} + client_secret: ${grafanaOidc.clientSecret} + scopes: <#list (grafanaOidc.scopes!["openid", "profile", "email"]) as scope>${scope}<#sep> + auth_url: ${grafanaOidc.issuerUrl}/protocol/openid-connect/auth + token_url: ${grafanaOidc.issuerUrl}/protocol/openid-connect/token + api_url: ${grafanaOidc.issuerUrl}/protocol/openid-connect/userinfo + signout_redirect_url: ${config.features.monitoring.grafanaUrl} + groups_attribute_path: groups +<#if grafanaOidc.adminGroupName?has_content> + role_attribute_path: contains(groups[*], '${grafanaOidc.adminGroupName}') && 'Admin' || 'None' + role_attribute_strict: true + allow_assign_grafana_admin: true + + analytics: check_for_updates: false <#if config.application.openshift == true> @@ -156,8 +180,10 @@ grafana: create: false defaultDashboardsEnabled: false - adminUser: ${config.application["username"]} - adminPassword: ${config.application["password"]} + admin: + existingSecret: "grafana-admin-credentials" + userKey: "admin-user" + passwordKey: "admin-password" service: type: ClusterIP <#if monitoring?? && monitoring?is_hash && monitoring.grafana?? && monitoring.grafana.host?has_content> @@ -239,7 +265,7 @@ grafana: routes: - receiver: email group_by: ["grafana_folder", "alertname"] - <#if config.features.mail.smtpUser?has_content || config.features.mail.smtpPassword?has_content> + <#if config.features.mail.smtpCredentialsConfigured> smtp: # `existingSecret` is a reference to an existing secret containing the smtp configuration # for Grafana. @@ -339,14 +365,23 @@ prometheus: repository: ${prometheusImageObject.repository} tag : ${prometheusImageObject.tag} +<#assign hasScmMetrics = config.scm.scmProviderType?has_content + && config.scm.scmProviderType?lower_case == "scm_manager" + && scm.host?has_content + && scm.protocol?has_content + && scm.path?has_content> +<#assign hasJenkinsMetrics = config.jenkins.active == true> +<#if hasScmMetrics || hasJenkinsMetrics> secrets: +<#if hasScmMetrics> - prometheus-metrics-creds-scmm + +<#if hasJenkinsMetrics> - prometheus-metrics-creds-jenkins + + additionalScrapeConfigs: -<#if config.scm.scmProviderType?lower_case == "scm_manager" - && scm.host?has_content - && scm.protocol?has_content - && scm.path?has_content> +<#if hasScmMetrics> - job_name: 'scm-manager' static_configs: - targets: [ '${scm.host}' ] @@ -356,7 +391,7 @@ prometheus: username: '${config.application.namePrefix}metrics' password_file: '/etc/prometheus/secrets/prometheus-metrics-creds-scmm/password' -<#if config.jenkins.active == true> +<#if hasJenkinsMetrics> - job_name: 'jenkins' static_configs: - targets: [ '${jenkins.host}' ] @@ -365,4 +400,4 @@ prometheus: basic_auth: username: '${jenkins.metricsUsername}' password_file: '/etc/prometheus/secrets/prometheus-metrics-creds-jenkins/password' - + \ No newline at end of file diff --git a/argocd/cluster-resources/apps/scm-manager/templates/values.ftl.yaml b/argocd/cluster-resources/apps/scm-manager/templates/values.ftl.yaml index 5e7e13d86..faa46ff51 100644 --- a/argocd/cluster-resources/apps/scm-manager/templates/values.ftl.yaml +++ b/argocd/cluster-resources/apps/scm-manager/templates/values.ftl.yaml @@ -5,13 +5,11 @@ persistence: livenessProbe: initialDelaySeconds: 120 -fullnameOverride : ${releaseName} +fullnameOverride: ${releaseName} -extraEnv: | - - name: SCM_WEBAPP_INITIALUSER - value: "${username}" - - name: SCM_WEBAPP_INITIALPASSWORD - value: "${password}" +extraEnvFrom: | + - secretRef: + name: ${credentialsSecretName} service: type: NodePort @@ -30,4 +28,16 @@ ingress: hosts: - ${host} + +<#if config.scm.scmManager.scmmImage?has_content || config.registry.createImagePullSecrets == true> +image: + <#if config.scm.scmManager.scmmImage?has_content> + <#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> + <#assign scmmImage = DockerImageParser.parse(config.scm.scmManager.scmmImage)> + repository: ${scmmImage.registryAndRepositoryAsString} + tag: ${scmmImage.tag} + + <#if config.registry.createImagePullSecrets == true> + pullSecret: proxy-registry + \ No newline at end of file diff --git a/argocd/cluster-resources/apps/ingress/templates/ingress-helm-values.ftl.yaml b/argocd/cluster-resources/apps/traefik/templates/values.ftl.yaml similarity index 100% rename from argocd/cluster-resources/apps/ingress/templates/ingress-helm-values.ftl.yaml rename to argocd/cluster-resources/apps/traefik/templates/values.ftl.yaml diff --git a/argocd/cluster-resources/apps/vault/templates/dev-post-start.ftl.sh b/argocd/cluster-resources/apps/vault/templates/dev-post-start.ftl.sh index ac70ee084..8497fb0c3 100644 --- a/argocd/cluster-resources/apps/vault/templates/dev-post-start.ftl.sh +++ b/argocd/cluster-resources/apps/vault/templates/dev-post-start.ftl.sh @@ -4,25 +4,32 @@ set -o errexit -o nounset # No pipefail in plain shell (!= bash) #-o pipefail - # Write executed commands for easier debugging set -x - # Parameters (via env vars) # USERNAME, PASSWORD -> Human user account created in vault # ARGOCD -> Allows read access for service accounts within example-apps-staging and -production namespaces used by external secrets operator +# OIDC_ENABLED -> Enable OIDC auth method +# OIDC_DISCOVERY_URL -> OIDC discovery URL (e.g. http://keycloak.localhost/realms/gop) +# OIDC_CLIENT_ID -> OIDC client ID +# OIDC_CLIENT_SECRET -> OIDC client secret +# OIDC_ADMIN_GROUP -> OIDC group that receives admin policy +# VAULT_EXTERNAL_URL -> External URL of vault (for OIDC redirect URIs) main() { - waitForVault - + createUserAccount "$USERNAME" "$PASSWORD" - + enableKubernetesAuth - + if [ "$ARGOCD" = 'true' ]; then authorizeServiceAccountsFromArgoCDExamples fi + + if [ "${OIDC_ENABLED}" = 'true' ]; then + enableOidc + fi } waitForVault() { @@ -34,25 +41,23 @@ waitForVault() { createUserAccount() { USERNAME=$1 PASSWORD=$2 - # Create policy vault policy write secret-editor - </dev/null || true # Create and authorize user via policy vault write auth/userpass/users/$USERNAME password="$PASSWORD" policies=secret-editor } enableKubernetesAuth() { # Enable access for kubernetes service accounts - + # https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-external-vault - vault auth enable kubernetes - + vault auth enable kubernetes 2>/dev/null || true + # https://developer.hashicorp.com/vault/docs/auth/kubernetes#use-local-service-account-token-as-the-reviewer-jwt # Makes vault access the k8s API to find a service account matching an access token # Vault then checks if the k8s-SA is authorized via a vault-role. A vault-role brings together a k8s-SA + vault-policy. @@ -63,18 +68,17 @@ enableKubernetesAuth() { } authorizeServiceAccountsFromArgoCDExamples() { - # Authorize service accounts within argoCD-deployed namespaces used by external secrets operator configured via SecretStore - + # Authorize service accounts within argoCD-deployed namespaces used by external secrets operator configured via SecretStore + for STAGE in staging production do - POLICY=${STAGE}-read - + POLICY=${STAGE}-read + vault policy write ${POLICY} - < @@ -85,5 +89,61 @@ EOF done } +enableOidc() { + echo "=== OIDC-Konfiguration ===" + echo "OIDC_DISCOVERY_URL = $OIDC_DISCOVERY_URL" + echo "OIDC_CLIENT_ID = $OIDC_CLIENT_ID" + echo "OIDC_CLIENT_SECRET = $OIDC_CLIENT_SECRET" + echo "VAULT_EXTERNAL_URL = $VAULT_EXTERNAL_URL" + echo "redirect_uri (ui) = $VAULT_EXTERNAL_URL/ui/vault/auth/oidc/oidc/callback" + echo "redirect_uri (cli) = $VAULT_EXTERNAL_URL/oidc/callback" + echo "==========================" + + timeout 60s sh -c "until wget -O/dev/null -q $OIDC_DISCOVERY_URL/.well-known/openid-configuration; do sleep 2; done" + + vault auth enable oidc 2>/dev/null || true + + vault write auth/oidc/config \ + oidc_discovery_url="$OIDC_DISCOVERY_URL" \ + oidc_client_id="$OIDC_CLIENT_ID" \ + oidc_client_secret="$OIDC_CLIENT_SECRET" \ + default_role="default" + + vault write auth/oidc/role/default \ + role_type="oidc" \ + bound_audiences="$OIDC_CLIENT_ID" \ + allowed_redirect_uris="$VAULT_EXTERNAL_URL/ui/vault/auth/oidc/oidc/callback" \ + allowed_redirect_uris="$VAULT_EXTERNAL_URL/oidc/callback" \ + user_claim="sub" \ + groups_claim="groups" \ + policies="default" \ + ttl="1h" + + if [ -n "${OIDC_ADMIN_GROUP:-}" ]; then + vault policy write admin - </dev/null || true + fi + + echo "OIDC configured" +} + main "$@" - + \ No newline at end of file diff --git a/argocd/cluster-resources/apps/vault/templates/values.ftl.yaml b/argocd/cluster-resources/apps/vault/templates/values.ftl.yaml index 33b73e2aa..921c530e3 100644 --- a/argocd/cluster-resources/apps/vault/templates/values.ftl.yaml +++ b/argocd/cluster-resources/apps/vault/templates/values.ftl.yaml @@ -1,4 +1,5 @@ <#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> + <#assign vaultOidc = (config.features.secrets.vault.oidc)!{}> ui: enabled: true externalPort: 80 @@ -22,6 +23,7 @@ server: repository: ${imageObject.registryAndRepositoryAsString} tag: ${imageObject.tag} + <#if host?has_content> ingress: enabled: true @@ -49,10 +51,29 @@ server: - mountPath: /var/opt/scripts name: ${dev.vaultPostStartVolume} readOnly : true + extraSecretEnvironmentVars: + - envName: USERNAME + secretName: ${dev.userCredentialsSecret} + secretKey: username + - envName: PASSWORD + secretName: ${dev.userCredentialsSecret} + secretKey: password postStart: - - /bin/sh - - -c - - USERNAME=${config.application.username} PASSWORD=${config.application.password} ARGOCD=${config.features.argocd.active?c} /var/opt/scripts/${dev.postStartScriptName} 2>&1 ${"|"} tee /tmp/dev-post-start.log + - /bin/sh + - -c + - | + ARGOCD=${config.features.argocd.active?c} \ +<#if vaultOidc?has_content && vaultOidc.enabled> + OIDC_ENABLED=true \ + OIDC_CLIENT_ID=${vaultOidc.clientId} \ + OIDC_CLIENT_SECRET=${vaultOidc.clientSecret} \ + OIDC_DISCOVERY_URL=${vaultOidc.issuerUrl} \ + OIDC_ADMIN_GROUP=${vaultOidc.adminGroupName} \ + VAULT_EXTERNAL_URL=http://${host} \ +<#else> + OIDC_ENABLED=false \ + + /var/opt/scripts/${dev.postStartScriptName} 2>&1 | tee /tmp/dev-post-start.log <#if config.application.podResources == true> @@ -63,4 +84,4 @@ server: requests: memory: 100Mi cpu: 50m - \ No newline at end of file + diff --git a/compiler.groovy b/compiler.groovy deleted file mode 100644 index 4791e21e9..000000000 --- a/compiler.groovy +++ /dev/null @@ -1,4 +0,0 @@ -withConfig(configuration) { - ast(groovy.transform.CompileStatic) - ast(groovy.transform.TypeChecked) -} \ No newline at end of file diff --git a/docs/Configuration.md b/docs/Configuration.md index 36a79ed4b..31e4ac865 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -1,6 +1,7 @@ # Overview of all CLI and config options -All options can be set via a [config file](./configuration.schema.json). Most options are also available as CLI parameters. +All options can be set via a [config file](./configuration.schema.json). Most options are also available as CLI +parameters. ## Table of Contents @@ -11,136 +12,196 @@ All options can be set via a [config file](./configuration.schema.json). Most op - [Application](#application) - [Content](#content) - [Tools](#tools) - - [Argocd](#tools-argocd) - - [Mail](#tools-mail) - - [Monitoring](#tools-monitoring) - - [Secrets](#tools-secrets) - - [Ingress](#tools-ingress) - - [Cert Manager](#tools-cert-manager) + - [Argocd](#tools-argocd) + - [Mail](#tools-mail) + - [Monitoring](#tools-monitoring) + - [Secrets](#tools-secrets) + - [Ingress](#tools-ingress) + - [Cert Manager](#tools-cert-manager) ## Registry -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--registry` | `registry.active` | Boolean | `false` | Installs a simple cluster-local registry for demonstration purposes. Warning: Registry does not provide authentication! | -| `--internal-registry-port` | `registry.internalPort` | Integer | `30000` | Port of registry registry. Ignored when a registry*url params are set | -| `--registry-url` | `registry.url` | String | `` | The url of your external registry, used for pushing images | -| `--registry-path` | `registry.path` | String | `` | Optional when registry-url is set | -| `--registry-username` | `registry.username` | String | `` | Optional when registry-url is set | -| `--registry-password` | `registry.password` | String | `` | Optional when registry-url is set | -| `--registry-proxy-url` | `registry.proxyUrl` | String | `` | The url of your proxy-registry. Used in pipelines to authorize pull base images. Use in conjunction with petclinic base image. Used in helm charts when create-image-pull-secrets is set. Use in conjunction with helm.*image fields. | -| `--registry-proxy-path` | `registry.proxyPath` | String | `` | Optional when registry-proxy-url is set and the registry is running on a non root web path. | -| `--registry-proxy-username` | `registry.proxyUsername` | String | `` | Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set. | -| `--registry-proxy-password` | `registry.proxyPassword` | String | `` | Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set. | -| `--registry-username-read-only` | `registry.readOnlyUsername` | String | `` | Optional alternative username for registry-url with read-only permissions that is used when create-image-pull-secrets is set. | -| `--registry-password-read-only` | `registry.readOnlyPassword` | String | `` | Optional alternative password for registry-url with read-only permissions that is used when create-image-pull-secrets is set. | -| `--create-image-pull-secrets` | `registry.createImagePullSecrets` | Boolean | `false` | Create image pull secrets for registry and proxy-registry for all GOP namespaces and helm charts. Uses proxy-username, read-only-username or registry-username (in this order). Use this if your cluster is not auto-provisioned with credentials for your private registries or if you configure individual helm images to be pulled from the proxy-registry that requires authentication. | -| `--registry-namespace` | `registry.namespace` | String | `registry` | Optional defines the kubernetes namespace for registry. | -| - | `registry.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `registry.helm.chart` | String | `docker-registry` | Name of the Helm chart | -| - | `registry.helm.repoURL` | String | `https://twuni.github.io/docker-registry.helm` | Repository url from which the Helm chart should be obtained | -| - | `registry.helm.version` | String | `3.0.0` | The version of the Helm chart to be installed | +| CLI | Config key | Type | Default | Description | +|:--------------------------------|:-----------------------------------------------|:--------|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--registry` | `registry.active` | Boolean | `false` | Installs a simple cluster-local registry for demonstration purposes. Warning: Registry does not provide authentication! | +| `--internal-registry-port` | `registry.internalPort` | Integer | `30000` | Port of registry registry. Ignored when a registry*url params are set | +| `--registry-url` | `registry.url` | String | `` | The url of your external registry, used for pushing images | +| `--registry-path` | `registry.path` | String | `` | Optional when registry-url is set | +| `--registry-username` | `registry.username` | String | `` | Optional when registry-url is set | +| `--registry-password` | `registry.password` | String | `` | Optional when registry-url is set | +| - | `registry.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--registry-proxy-url` | `registry.proxyUrl` | String | `` | The url of your proxy-registry. Used in pipelines to authorize pull base images. Use in conjunction with petclinic base image. Used in helm charts when create-image-pull-secrets is set. Use in conjunction with helm.*image fields. | +| `--registry-proxy-path` | `registry.proxyPath` | String | `` | Optional when registry-proxy-url is set and the registry is running on a non root web path. | +| `--registry-proxy-username` | `registry.proxyUsername` | String | `` | Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set. | +| `--registry-proxy-password` | `registry.proxyPassword` | String | `` | Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set. | +| - | `registry.proxyCredentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.proxyCredentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.proxyCredentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.proxyCredentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.proxyCredentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--registry-username-read-only` | `registry.readOnlyUsername` | String | `` | Optional alternative username for registry-url with read-only permissions that is used when create-image-pull-secrets is set. | +| `--registry-password-read-only` | `registry.readOnlyPassword` | String | `` | Optional alternative password for registry-url with read-only permissions that is used when create-image-pull-secrets is set. | +| - | `registry.readOnlyCredentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.readOnlyCredentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.readOnlyCredentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.readOnlyCredentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `registry.readOnlyCredentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--create-image-pull-secrets` | `registry.createImagePullSecrets` | Boolean | `false` | Create image pull secrets for registry and proxy-registry for all GOP namespaces and helm charts. Uses proxy-username, read-only-username or registry-username (in this order). Use this if your cluster is not auto-provisioned with credentials for your private registries or if you configure individual helm images to be pulled from the proxy-registry that requires authentication. | +| `--registry-namespace` | `registry.namespace` | String | `registry` | Optional defines the kubernetes namespace for registry. | +| - | `registry.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `registry.helm.chart` | String | `docker-registry` | Name of the Helm chart | +| - | `registry.helm.repoURL` | String | `https://twuni.github.io/docker-registry.helm` | Repository url from which the Helm chart should be obtained | +| - | `registry.helm.version` | String | `3.0.0` | The version of the Helm chart to be installed | ## Jenkins -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--jenkins` | `jenkins.active` | Boolean | `false` | Installs Jenkins as CI server | -| `--jenkins-skip-restart` | `jenkins.skipRestart` | Boolean | `false` | Skips restarting Jenkins after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades. | -| `--jenkins-skip-plugins` | `jenkins.skipPlugins` | Boolean | `false` | Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades. | -| `--jenkins-url` | `jenkins.url` | String | `` | The url of your external jenkins | -| `--jenkins-username` | `jenkins.username` | String | `admin` | Mandatory when jenkins-url is set | -| `--jenkins-password` | `jenkins.password` | String | `` | Mandatory when jenkins-url is set | -| `--jenkins-metrics-username` | `jenkins.metricsUsername` | String | `metrics` | Mandatory when jenkins-url is set and monitoring enabled | -| `--jenkins-metrics-password` | `jenkins.metricsPassword` | String | `metrics` | Mandatory when jenkins-url is set and monitoring enabled | -| `--maven-central-mirror` | `jenkins.mavenCentralMirror` | String | `` | URL for maven mirror, used by applications built in Jenkins | -| `--jenkins-additional-envs` | `jenkins.additionalEnvs` | Map | `[:]` | Set additional environments to Jenkins | -| - | `jenkins.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `jenkins.helm.chart` | String | `jenkins` | Name of the Helm chart | -| - | `jenkins.helm.repoURL` | String | `https://charts.jenkins.io` | Repository url from which the Helm chart should be obtained | -| - | `jenkins.helm.version` | String | `5.9.18` | The version of the Helm chart to be installed | -| `--jenkins-namespace` | `jenkins.namespace` | String | `jenkins` | Optional defines the kubernetes namespace for Jenkins. | +| CLI | Config key | Type | Default | Description | +|:-----------------------------|:---------------------------------------------|:-------------------|:----------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--jenkins` | `jenkins.active` | Boolean | `false` | Installs Jenkins as CI server | +| `--jenkins-skip-restart` | `jenkins.skipRestart` | Boolean | `false` | Skips restarting Jenkins after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades. | +| `--jenkins-skip-plugins` | `jenkins.skipPlugins` | Boolean | `false` | Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades. | +| `--jenkins-url` | `jenkins.url` | String | `` | The url of your external jenkins | +| `--jenkins-username` | `jenkins.username` | String | `admin` | Mandatory when jenkins-url is set | +| `--jenkins-password` | `jenkins.password` | String | `2RkLYwaLy!P2` | Mandatory when jenkins-url is set | +| - | `jenkins.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--jenkins-metrics-username` | `jenkins.metricsUsername` | String | `metrics` | Mandatory when jenkins-url is set and monitoring enabled | +| `--jenkins-metrics-password` | `jenkins.metricsPassword` | String | `metrics` | Mandatory when jenkins-url is set and monitoring enabled | +| - | `jenkins.metricsCredentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.metricsCredentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.metricsCredentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.metricsCredentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `jenkins.metricsCredentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--jenkins-image` | `jenkins.jenkinsImage` | String | `` | Sets image for Jenkins | +| `--maven-central-mirror` | `jenkins.mavenCentralMirror` | String | `` | URL for maven mirror, used by applications built in Jenkins | +| - | `jenkins.oidc.providerName` | String | `Keycloak` | Name of the OIDC provider displayed in tool login screens | +| - | `jenkins.oidc.issuerUrl` | String | `` | OIDC issuer URL, for example http://keycloak.local.gd/realms/gop | +| - | `jenkins.oidc.clientId` | String | `jenkins` | OIDC client ID | +| - | `jenkins.oidc.clientSecret` | String | `` | OIDC client secret | +| - | `jenkins.oidc.scopes` | List<String> | `[openid, profile, email]` | OIDC scopes requested by the tool | +| - | `jenkins.oidc.adminGroupName` | String | `` | OIDC group that receives full admin permissions in all OIDC-enabled tools | +| `--jenkins-additional-envs` | `jenkins.additionalEnvs` | Map | `{}` | Set additional environments to Jenkins | +| - | `jenkins.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `jenkins.helm.chart` | String | `jenkins` | Name of the Helm chart | +| - | `jenkins.helm.repoURL` | String | `https://charts.jenkins.io` | Repository url from which the Helm chart should be obtained | +| - | `jenkins.helm.version` | String | `5.9.18` | The version of the Helm chart to be installed | +| `--jenkins-namespace` | `jenkins.namespace` | String | `jenkins` | Optional defines the kubernetes namespace for Jenkins. | ## Multi Tenant -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--central-gitlab-url` | `multiTenant.gitlab.url` | String | `-` | URL for external Gitlab | -| `--central-gitlab-username` | `multiTenant.gitlab.username` | String | `-` | GitLab username for API access. Must be 'oauth2' when using Personal Access Token (PAT) authentication | -| `--central-gitlab-token` | `multiTenant.gitlab.password` | String | `-` | Password for SCM Manager authentication | -| `--central-gitlab-group-id` | `multiTenant.gitlab.parentGroupId` | String | `-` | Main Group for Gitlab where the GOP creates it's groups/repos | -| `--central-scmm-internal` | `multiTenant.scmManager.internal` | Boolean | `-` | SCM for Central Management is running on the same cluster, so k8s internal URLs can be used for access | -| `--central-scmm-url` | `multiTenant.scmManager.url` | String | `-` | URL for the centralized Management Repo | -| `--central-scmm-username` | `multiTenant.scmManager.username` | String | `-` | CENTRAL SCMM username | -| `--central-scmm-password` | `multiTenant.scmManager.password` | String | `-` | CENTRAL SCMM password | -| `--central-scmm-namespace` | `multiTenant.scmManager.namespace` | String | `-` | Namespace where to find the Central SCMM | -| `--central-argocd-namespace` | `multiTenant.centralArgocdNamespace` | String | `argocd` | Namespace for the centralized Argocd | -| `--dedicated-instance` | `multiTenant.useDedicatedInstance` | Boolean | `false` | Toggles the Dedicated Instances Mode. See docs for more info | +| CLI | Config key | Type | Default | Description | +|:-----------------------------|:-----------------------------------------------------|:----------------|:--------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--central-scm-provider` | `multiTenant.scmProviderType` | ScmProviderType | `SCM_MANAGER` | The SCM provider type. Possible values: SCM_MANAGER, GITLAB | +| `--central-gitlab-url` | `multiTenant.gitlab.url` | String | `-` | URL for external Gitlab | +| `--central-gitlab-username` | `multiTenant.gitlab.username` | String | `-` | GitLab username for API access. Must be 'oauth2' when using Personal Access Token (PAT) authentication | +| `--central-gitlab-token` | `multiTenant.gitlab.password` | String | `-` | Password for SCM Manager authentication | +| - | `multiTenant.gitlab.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.gitlab.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.gitlab.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.gitlab.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.gitlab.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--central-gitlab-group-id` | `multiTenant.gitlab.parentGroupId` | String | `-` | Main Group for Gitlab where the GOP creates it's groups/repos | +| `--central-scmm-internal` | `multiTenant.scmManager.internal` | Boolean | `-` | SCM for Central Management is running on the same cluster, so k8s internal URLs can be used for access | +| `--central-scmm-url` | `multiTenant.scmManager.url` | String | `-` | URL for the centralized Management Repo | +| `--central-scmm-username` | `multiTenant.scmManager.username` | String | `-` | CENTRAL SCMM username | +| `--central-scmm-password` | `multiTenant.scmManager.password` | String | `-` | CENTRAL SCMM password | +| - | `multiTenant.scmManager.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.scmManager.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.scmManager.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.scmManager.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `multiTenant.scmManager.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--central-scmm-namespace` | `multiTenant.scmManager.namespace` | String | `-` | Namespace where to find the Central SCMM | +| `--central-argocd-namespace` | `multiTenant.centralArgocdNamespace` | String | `argocd` | Namespace for the centralized Argocd | +| `--dedicated-instance` | `multiTenant.useDedicatedInstance` | Boolean | `false` | Toggles the Dedicated Instances Mode. See docs for more info | ## Scm -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| - | `scm.gitlab.internal` | Boolean | `-` | True if Gitlab is running in the same K8s cluster. For now we only support access by external URL | -| `--gitlab-url` | `scm.gitlab.url` | String | `-` | Base URL for the Gitlab instance | -| `--gitlab-username` | `scm.gitlab.username` | String | `-` | Defaults to: oauth2.0 when PAT token is given. | -| `--gitlab-token` | `scm.gitlab.password` | String | `-` | PAT Token for the account. Needs read/write repo permissions. See docs for mor information | -| `--gitlab-group-id` | `scm.gitlab.parentGroupId` | String | `-` | Number for the Gitlab Group where the repos and subgroups should be created | -| - | `scm.gitlab.gitOpsUsername` | String | `-` | Username for the Gitops User | -| `--scmm-url` | `scm.scmManager.url` | String | `-` | The host of your external scm-manager | -| `--scmm-namespace` | `scm.scmManager.namespace` | String | `-` | Namespace where SCM-Manager should run | -| `--scmm-username` | `scm.scmManager.username` | String | `-` | Mandatory when scmm-url is set | -| `--scmm-password` | `scm.scmManager.password` | String | `-` | Mandatory when scmm-url is set | -| - | `scm.scmManager.helm.values` | Map | `-` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `scm.scmManager.helm.chart` | String | `-` | Name of the Helm chart | -| - | `scm.scmManager.helm.repoURL` | String | `-` | Repository url from which the Helm chart should be obtained | -| - | `scm.scmManager.helm.version` | String | `-` | The version of the Helm chart to be installed | -| `--scmm-skip-restart` | `scm.scmManager.skipRestart` | Boolean | `-` | Skips restarting SCM-Manager after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.' | -| `--scmm-skip-plugins` | `scm.scmManager.skipPlugins` | Boolean | `-` | Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades. | -| - | `scm.scmManager.gitOpsUsername` | String | `-` | Username for the Gitops User | +| CLI | Config key | Type | Default | Description | +|:----------------------|:---------------------------------------------|:----------------|:--------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--scm-provider` | `scm.scmProviderType` | ScmProviderType | `SCM_MANAGER` | The SCM provider type. Possible values: SCM_MANAGER, GITLAB | +| - | `scm.gitlab.internal` | Boolean | `-` | True if Gitlab is running in the same K8s cluster. For now we only support access by external URL | +| `--gitlab-url` | `scm.gitlab.url` | String | `-` | Base URL for the Gitlab instance | +| `--gitlab-username` | `scm.gitlab.username` | String | `-` | Defaults to: oauth2.0 when PAT token is given. | +| `--gitlab-token` | `scm.gitlab.password` | String | `-` | PAT Token for the account. Needs read/write repo permissions. See docs for mor information | +| - | `scm.gitlab.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.gitlab.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.gitlab.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.gitlab.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.gitlab.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `--gitlab-group-id` | `scm.gitlab.parentGroupId` | String | `-` | Number for the Gitlab Group where the repos and subgroups should be created | +| - | `scm.gitlab.gitOpsUsername` | String | `-` | Username for the Gitops User | +| `--scmm-url` | `scm.scmManager.url` | String | `-` | The host of your external scm-manager | +| `--scmm-namespace` | `scm.scmManager.namespace` | String | `-` | Namespace where SCM-Manager should run | +| `--scmm-username` | `scm.scmManager.username` | String | `-` | Mandatory when scmm-url is set | +| `--scmm-password` | `scm.scmManager.password` | String | `-` | Mandatory when scmm-url is set | +| - | `scm.scmManager.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.scmManager.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.scmManager.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.scmManager.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.scmManager.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `scm.scmManager.helm.values` | Map | `-` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `scm.scmManager.helm.chart` | String | `-` | Name of the Helm chart | +| - | `scm.scmManager.helm.repoURL` | String | `-` | Repository url from which the Helm chart should be obtained | +| - | `scm.scmManager.helm.version` | String | `-` | The version of the Helm chart to be installed | +| `--scmm-image` | `scm.scmManager.scmmImage` | String | `-` | Sets image for SCM-Manager | +| `--scmm-skip-restart` | `scm.scmManager.skipRestart` | Boolean | `-` | Skips restarting SCM-Manager after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.' | +| `--scmm-skip-plugins` | `scm.scmManager.skipPlugins` | Boolean | `-` | Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades. | +| - | `scm.scmManager.gitOpsUsername` | String | `-` | Username for the Gitops User | ## Application -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--config-file` | `application.configFiles` | List<String> | `[]` | - | -| `--config-map` | `application.configMaps` | List<String> | `[]` | - | -| `-d`, `--debug` | `application.debug` | Boolean | `-` | - | -| `-x`, `--trace` | `application.trace` | Boolean | `-` | - | -| `--output-config-file` | `application.outputConfigFile` | Boolean | `false` | - | -| `-v`, `--version` | `application.versionInfoRequested` | Boolean | `false` | - | -| `-h`, `--help` | `application.usageHelpRequested` | Boolean | `false` | - | -| `--insecure` | `application.insecure` | Boolean | `false` | Sets insecure-mode in cURL which skips cert validation | -| `--openshift` | `application.openshift` | Boolean | `false` | When set, openshift specific resources and configurations are applied | -| `--username` | `application.username` | String | `admin` | Set initial admin username | -| `--password` | `application.password` | String | `` | Set initial admin passwords | -| `-y`, `--yes` | `application.yes` | Boolean | `false` | Skip confirmation | -| `--name-prefix` | `application.namePrefix` | String | `` | Set name-prefix for repos, jobs, namespaces | -| `--destroy` | `application.destroy` | Boolean | `false` | Unroll playground | -| `--pod-resources` | `application.podResources` | Boolean | `false` | Write kubernetes resource requests and limits on each pod | -| `--git-name` | `application.gitName` | String | `Cloudogu` | Sets git author and committer name used for initial commits | -| `--git-email` | `application.gitEmail` | String | `hello@cloudogu.com` | Sets git author and committer email used for initial commits | -| `--base-url` | `application.baseUrl` | String | `` | the external base url (TLD) for all tools, e.g. https://example.com or http://localhost:8080. The individual -url params for argocd, grafana and vault take precedence. | -| `--url-separator-hyphen` | `application.urlSeparatorHyphen` | Boolean | `false` | Use hyphens instead of dots to separate application name from base-url | -| `--mirror-repos` | `application.mirrorRepos` | Boolean | `false` | Changes the sources of deployed tools so they are not pulled from the internet, but are pulled from git and work in air-gapped environments. | -| `--skip-crds` | `application.skipCrds` | Boolean | `false` | Skip installation of CRDs. This requires prior installation of CRDs | -| `--namespace-isolation` | `application.namespaceIsolation` | Boolean | `false` | Configure tools to explicitly work with the given namespaces only, and not cluster-wide. This way GOP can be installed without having cluster-admin permissions. | -| `--netpols` | `application.netpols` | Boolean | `false` | Sets Network Policies | -| `--cluster-admin` | `application.clusterAdmin` | Boolean | `false` | Binds ArgoCD controllers to cluster-admin ClusterRole | -| `-p`, `--profile` | `application.profile` | String | `-` | Use predefined profile (full, only-argocd, operator-mandants aso.) | -| `--gop-namespace` | `application.gopNamespace` | String | `` | If set, GOP stores specific information in this namespace. | -| `-n`, `--namespace` | `application.namespace` | String | `` | If set, GOP uses the same Kubernetes namespace for all tools and examples. Attention! Only use for test purposes. | +| CLI | Config key | Type | Default | Description | +|:-------------------------|:------------------------------------------|:-------------------|:---------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--config-file` | `application.configFiles` | List<String> | `[]` | - | +| `--config-map` | `application.configMaps` | List<String> | `[]` | - | +| `-d`, `--debug` | `application.debug` | Boolean | `false` | - | +| `-x`, `--trace` | `application.trace` | Boolean | `false` | - | +| `--output-config-file` | `application.outputConfigFile` | Boolean | `false` | - | +| `-v`, `--version` | `application.versionInfoRequested` | Boolean | `false` | - | +| `-h`, `--help` | `application.usageHelpRequested` | Boolean | `false` | - | +| `--insecure` | `application.insecure` | Boolean | `false` | Sets insecure-mode in cURL which skips cert validation | +| `--openshift` | `application.openshift` | Boolean | `false` | When set, openshift specific resources and configurations are applied | +| `--username` | `application.username` | String | `admin` | Set initial admin username | +| `--password` | `application.password` | String | `2RkLYwaLy!P2` | Set initial admin passwords | +| - | `application.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `application.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `application.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `application.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `application.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| `-y`, `--yes` | `application.yes` | Boolean | `false` | Skip confirmation | +| `--name-prefix` | `application.namePrefix` | String | `` | Set name-prefix for repos, jobs, namespaces | +| `--destroy` | `application.destroy` | Boolean | `false` | Unroll playground | +| `--pod-resources` | `application.podResources` | Boolean | `false` | Write kubernetes resource requests and limits on each pod | +| `--git-name` | `application.gitName` | String | `Cloudogu` | Sets git author and committer name used for initial commits | +| `--git-email` | `application.gitEmail` | String | `hello@cloudogu.com` | Sets git author and committer email used for initial commits | +| `--base-url` | `application.baseUrl` | String | `` | the external base url (TLD) for all tools, e.g. https://example.com or http://localhost:8080. The individual -url params for argocd, grafana and vault take precedence. | +| `--url-separator-hyphen` | `application.urlSeparatorHyphen` | Boolean | `false` | Use hyphens instead of dots to separate application name from base-url | +| `--mirror-repos` | `application.mirrorRepos` | Boolean | `false` | Changes the sources of deployed tools so they are not pulled from the internet, but are pulled from git and work in air-gapped environments. | +| `--skip-crds` | `application.skipCrds` | Boolean | `false` | Skip installation of CRDs. This requires prior installation of CRDs | +| `--namespace-isolation` | `application.namespaceIsolation` | Boolean | `false` | Configure tools to explicitly work with the given namespaces only, and not cluster-wide. This way GOP can be installed without having cluster-admin permissions. | +| `--netpols` | `application.netpols` | Boolean | `false` | Sets Network Policies | +| `--cluster-admin` | `application.clusterAdmin` | Boolean | `false` | Binds ArgoCD controllers to cluster-admin ClusterRole | +| `-p`, `--profile` | `application.profile` | String | `-` | Use predefined profile (full, only-argocd, operator-mandants aso.) | +| `--gop-namespace` | `application.gopNamespace` | String | `` | If set, GOP stores specific information in this namespace. | +| `-n`, `--namespace` | `application.namespace` | String | `` | If set, GOP uses the same Kubernetes namespace for all tools and examples. Attention! Only use for test purposes. | ## Content -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| - | `content.namespaces` | List<String> | `[]` | Additional kubernetes namespaces. These are authorized to Argo CD, supplied with image pull secrets, monitored by prometheus, etc. Namespaces can be templates, e.g. ${config.application.namePrefix}staging | -| - | `content.repos` | List<ContentRepositorySchema> | `[]` | ContentLoader repos to push into target environment | -| - | `content.variables` | Map | `[:]` | Additional variables to use in custom templates. | -| - | `content.helmReleases` | List<HelmReleaseSchema> | `[]` | - | -| `--content-whitelist` | `content.useWhitelist` | Boolean | `false` | Enables the whitelist for statics in content templating | -| - | `content.allowedStaticsWhitelist` | Set<String> | `[]` | Whitelist for Statics freemarker is allowing in user templates | +| CLI | Config key | Type | Default | Description | +|:----------------------|:----------------------------------|:------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| - | `content.namespaces` | List<String> | `[]` | Additional kubernetes namespaces. These are authorized to Argo CD, supplied with image pull secrets, monitored by prometheus, etc. Namespaces can be templates, e.g. ${config.application.namePrefix}staging | +| - | `content.repos` | List<ContentRepositorySchema> | `[]` | ContentLoader repos to push into target environment | +| - | `content.variables` | Map | `{}` | Additional variables to use in custom templates. | +| - | `content.helmReleases` | List<HelmReleaseSchema> | `[]` | Additional Helm releases to deploy through Argo CD without requiring a content Git repository. | +| `--content-whitelist` | `content.useWhitelist` | Boolean | `false` | Enables the whitelist for statics in content templating | +| - | `content.allowedStaticsWhitelist` | Set<String> | `[com.cloudogu.gitops.utils.DockerImageParser, java.lang.Float, java.lang.Long, java.lang.Double, java.lang.Boolean, java.lang.Math, java.lang.String, java.lang.Integer]` | Whitelist for Statics freemarker is allowing in user templates | ## Tools @@ -148,92 +209,116 @@ Configuration of optional tools supported by gitops-playground. ### Tool: Argocd -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--argocd` | `features.argocd.active` | Boolean | `false` | Install ArgoCD | -| `--argocd-operator` | `features.argocd.operator` | Boolean | `false` | Install ArgoCD via an already running ArgoCD Operator | -| `--argocd-url` | `features.argocd.url` | String | `` | The URL where argocd is accessible. It has to be the full URL with http:// or https:// | -| - | `features.argocd.env` | List<java.util.Map> | `-` | Pass a list of env vars to Argo CD components. Currently only works with operator | -| `--argocd-email-from` | `features.argocd.emailFrom` | String | `argocd@example.org` | Notifications, define Argo CD sender email address | -| `--argocd-email-to-user` | `features.argocd.emailToUser` | String | `app-team@example.org` | Notifications, define Argo CD user / app-team recipient email address | -| `--argocd-email-to-admin` | `features.argocd.emailToAdmin` | String | `infra@example.org` | Notifications, define Argo CD admin recipient email address | -| `--argocd-resource-inclusions-cluster` | `features.argocd.resourceInclusionsCluster` | String | `` | Internal Kubernetes API Server URL https://IP:PORT (kubernetes.default.svc). Needed in argocd-operator resourceInclusions. Use this parameter if argocd.operator=true and NOT running inside a Pod (remote mode). Full URL needed, for example: https://100.125.0.1:443 | -| `--argocd-namespace` | `features.argocd.namespace` | String | `argocd` | Defines the kubernetes namespace for ArgoCD | -| - | `features.argocd.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| CLI | Config key | Type | Default | Description | +|:---------------------------------------|:--------------------------------------------|:--------------------------------------------------------------|:---------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--argocd` | `features.argocd.active` | Boolean | `false` | Install ArgoCD | +| `--argocd-operator` | `features.argocd.operator` | Boolean | `false` | Install ArgoCD via an already running ArgoCD Operator | +| `--argocd-url` | `features.argocd.url` | String | `` | The URL where argocd is accessible. It has to be the full URL with http:// or https:// | +| - | `features.argocd.env` | List<java.util.Map> | `-` | Pass a list of env vars to Argo CD components. Currently only works with operator | +| `--argocd-email-from` | `features.argocd.emailFrom` | String | `argocd@example.org` | Notifications, define Argo CD sender email address | +| `--argocd-email-to-user` | `features.argocd.emailToUser` | String | `app-team@example.org` | Notifications, define Argo CD user / app-team recipient email address | +| `--argocd-email-to-admin` | `features.argocd.emailToAdmin` | String | `infra@example.org` | Notifications, define Argo CD admin recipient email address | +| `--argocd-resource-inclusions-cluster` | `features.argocd.resourceInclusionsCluster` | String | `` | Internal Kubernetes API Server URL https://IP:PORT (kubernetes.default.svc). Needed in argocd-operator resourceInclusions. Use this parameter if argocd.operator=true and NOT running inside a Pod (remote mode). Full URL needed, for example: https://100.125.0.1:443 | +| `--argocd-namespace` | `features.argocd.namespace` | String | `argocd` | Defines the kubernetes namespace for ArgoCD | +| - | `features.argocd.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `features.argocd.oidc.providerName` | String | `Keycloak` | Name of the OIDC provider displayed in tool login screens | +| - | `features.argocd.oidc.issuerUrl` | String | `` | OIDC issuer URL, for example http://keycloak.local.gd/realms/gop | +| - | `features.argocd.oidc.clientId` | String | `argocd` | OIDC client ID | +| - | `features.argocd.oidc.clientSecret` | String | `` | OIDC client secret | +| - | `features.argocd.oidc.scopes` | List<String> | `[openid, profile, email]` | OIDC scopes requested by the tool | +| - | `features.argocd.oidc.adminGroupName` | String | `` | OIDC group that receives full admin permissions in all OIDC-enabled tools | ### Tool: Mail -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--smtp-address` | `features.mail.smtpAddress` | String | `` | Sets smtp port of external Mailserver | -| `--smtp-port` | `features.mail.smtpPort` | Integer | `-` | Sets smtp port of external Mailserver | -| `--smtp-user` | `features.mail.smtpUser` | String | `` | Sets smtp username for external Mailserver | -| `--smtp-password` | `features.mail.smtpPassword` | String | `` | Sets smtp password of external Mailserver | +| CLI | Config key | Type | Default | Description | +|:------------------|:--------------------------------------------|:--------|:--------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--smtp-address` | `features.mail.smtpAddress` | String | `` | Sets smtp port of external Mailserver | +| `--smtp-port` | `features.mail.smtpPort` | Integer | `-` | Sets smtp port of external Mailserver | +| `--smtp-user` | `features.mail.smtpUser` | String | `` | Sets smtp username for external Mailserver | +| `--smtp-password` | `features.mail.smtpPassword` | String | `` | Sets smtp password of external Mailserver | +| - | `features.mail.credentials.username` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `features.mail.credentials.secretNamespace` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `features.mail.credentials.secretName` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `features.mail.credentials.usernameKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | +| - | `features.mail.credentials.passwordKey` | String | `-` | Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys. | ### Tool: Monitoring -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--metrics`, `--monitoring` | `features.monitoring.active` | Boolean | `false` | Installs the Kube-Prometheus-Stack. This includes Prometheus, the Prometheus operator, Grafana and some extra resources | -| `--grafana-url` | `features.monitoring.grafanaUrl` | String | `` | Sets url for grafana | -| `--grafana-email-from` | `features.monitoring.grafanaEmailFrom` | String | `grafana@example.org` | Notifications, define grafana alerts sender email address | -| `--grafana-email-to` | `features.monitoring.grafanaEmailTo` | String | `infra@example.org` | Notifications, define grafana alerts recipient email address | -| `--grafana-image` | `features.monitoring.helm.grafanaImage` | String | `` | Sets image for grafana | -| `--grafana-sidecar-image` | `features.monitoring.helm.grafanaSidecarImage` | String | `` | Sets image for grafana's sidecar | -| `--prometheus-image` | `features.monitoring.helm.prometheusImage` | String | `` | Sets image for prometheus | -| `--prometheus-operator-image` | `features.monitoring.helm.prometheusOperatorImage` | String | `` | Sets image for prometheus-operator | -| `--prometheus-config-reloader-image` | `features.monitoring.helm.prometheusConfigReloaderImage` | String | `` | Sets image for prometheus-operator's config-reloader | -| - | `features.monitoring.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `features.monitoring.helm.chart` | String | `kube-prometheus-stack` | Name of the Helm chart | -| - | `features.monitoring.helm.repoURL` | String | `https://prometheus-community.github.io/helm-charts` | Repository url from which the Helm chart should be obtained | -| - | `features.monitoring.helm.version` | String | `80.2.2` | The version of the Helm chart to be installed | -| `--monitoring-namespace` | `features.monitoring.namespace` | String | `monitoring` | Optional defines the kubernetes namespace for monitoring. | +| CLI | Config key | Type | Default | Description | +|:-------------------------------------|:---------------------------------------------------------|:-------------------|:-----------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------| +| `--metrics`, `--monitoring` | `features.monitoring.active` | Boolean | `false` | Installs the Kube-Prometheus-Stack. This includes Prometheus, the Prometheus operator, Grafana and some extra resources | +| `--grafana-url` | `features.monitoring.grafanaUrl` | String | `` | Sets url for grafana | +| `--grafana-email-from` | `features.monitoring.grafanaEmailFrom` | String | `grafana@example.org` | Notifications, define grafana alerts sender email address | +| `--grafana-email-to` | `features.monitoring.grafanaEmailTo` | String | `infra@example.org` | Notifications, define grafana alerts recipient email address | +| - | `features.monitoring.oidc.providerName` | String | `Keycloak` | Name of the OIDC provider displayed in tool login screens | +| - | `features.monitoring.oidc.issuerUrl` | String | `` | OIDC issuer URL, for example http://keycloak.local.gd/realms/gop | +| - | `features.monitoring.oidc.clientId` | String | `grafana` | OIDC client ID | +| - | `features.monitoring.oidc.clientSecret` | String | `` | OIDC client secret | +| - | `features.monitoring.oidc.scopes` | List<String> | `[openid, profile, email]` | OIDC scopes requested by the tool | +| - | `features.monitoring.oidc.adminGroupName` | String | `` | OIDC group that receives full admin permissions in all OIDC-enabled tools | +| `--grafana-image` | `features.monitoring.helm.grafanaImage` | String | `` | Sets image for grafana | +| `--grafana-sidecar-image` | `features.monitoring.helm.grafanaSidecarImage` | String | `` | Sets image for grafana's sidecar | +| `--prometheus-image` | `features.monitoring.helm.prometheusImage` | String | `` | Sets image for prometheus | +| `--prometheus-operator-image` | `features.monitoring.helm.prometheusOperatorImage` | String | `` | Sets image for prometheus-operator | +| `--prometheus-config-reloader-image` | `features.monitoring.helm.prometheusConfigReloaderImage` | String | `` | Sets image for prometheus-operator's config-reloader | +| - | `features.monitoring.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `features.monitoring.helm.chart` | String | `kube-prometheus-stack` | Name of the Helm chart | +| - | `features.monitoring.helm.repoURL` | String | `https://prometheus-community.github.io/helm-charts` | Repository url from which the Helm chart should be obtained | +| - | `features.monitoring.helm.version` | String | `80.2.2` | The version of the Helm chart to be installed | +| `--monitoring-namespace` | `features.monitoring.namespace` | String | `monitoring` | Optional defines the kubernetes namespace for monitoring. | ### Tool: Secrets -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--external-secrets-image` | `features.secrets.externalSecrets.helm.image` | String | `` | Sets image for external secrets operator | -| `--external-secrets-certcontroller-image` | `features.secrets.externalSecrets.helm.certControllerImage` | String | `` | Sets image for external secrets operator's controller | -| `--external-secrets-webhook-image` | `features.secrets.externalSecrets.helm.webhookImage` | String | `` | Sets image for external secrets operator's webhook | -| - | `features.secrets.externalSecrets.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `features.secrets.externalSecrets.helm.chart` | String | `external-secrets` | Name of the Helm chart | -| - | `features.secrets.externalSecrets.helm.repoURL` | String | `https://charts.external-secrets.io` | Repository url from which the Helm chart should be obtained | -| - | `features.secrets.externalSecrets.helm.version` | String | `0.9.16` | The version of the Helm chart to be installed | -| `--vault-url` | `features.secrets.vault.url` | String | `` | Sets url for vault ui | -| `--vault-image` | `features.secrets.vault.helm.image` | String | `` | Sets image for vault | -| - | `features.secrets.vault.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `features.secrets.vault.helm.chart` | String | `vault` | Name of the Helm chart | -| - | `features.secrets.vault.helm.repoURL` | String | `https://helm.releases.hashicorp.com` | Repository url from which the Helm chart should be obtained | -| - | `features.secrets.vault.helm.version` | String | `0.25.0` | The version of the Helm chart to be installed | -| `--secrets-namespace` | `features.secrets.namespace` | String | `secrets` | Optional defines the kubernetes namespace for secrets. | +| CLI | Config key | Type | Default | Description | +|:------------------------------------------|:------------------------------------------------------------|:-------------------|:--------------------------------------|:-----------------------------------------------------------------------------------------------------------------------| +| `--external-secrets-image` | `features.secrets.externalSecrets.helm.image` | String | `` | Sets image for external secrets operator | +| `--external-secrets-certcontroller-image` | `features.secrets.externalSecrets.helm.certControllerImage` | String | `` | Sets image for external secrets operator's controller | +| `--external-secrets-webhook-image` | `features.secrets.externalSecrets.helm.webhookImage` | String | `` | Sets image for external secrets operator's webhook | +| - | `features.secrets.externalSecrets.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `features.secrets.externalSecrets.helm.chart` | String | `external-secrets` | Name of the Helm chart | +| - | `features.secrets.externalSecrets.helm.repoURL` | String | `https://charts.external-secrets.io` | Repository url from which the Helm chart should be obtained | +| - | `features.secrets.externalSecrets.helm.version` | String | `0.9.16` | The version of the Helm chart to be installed | +| `--vault` | `features.secrets.vault.mode` | VaultMode | `-` | Installs Hashicorp vault and the external secrets operator. Possible values: dev, prod. | +| `--vault-url` | `features.secrets.vault.url` | String | `` | Sets url for vault ui | +| - | `features.secrets.vault.oidc.providerName` | String | `Keycloak` | Name of the OIDC provider displayed in tool login screens | +| - | `features.secrets.vault.oidc.issuerUrl` | String | `` | OIDC issuer URL, for example http://keycloak.local.gd/realms/gop | +| - | `features.secrets.vault.oidc.clientId` | String | `vault` | OIDC client ID | +| - | `features.secrets.vault.oidc.clientSecret` | String | `` | OIDC client secret | +| - | `features.secrets.vault.oidc.scopes` | List<String> | `[openid, profile, email]` | OIDC scopes requested by the tool | +| - | `features.secrets.vault.oidc.adminGroupName` | String | `` | OIDC group that receives full admin permissions in all OIDC-enabled tools | +| `--vault-image` | `features.secrets.vault.helm.image` | String | `` | Sets image for vault | +| - | `features.secrets.vault.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `features.secrets.vault.helm.chart` | String | `vault` | Name of the Helm chart | +| - | `features.secrets.vault.helm.repoURL` | String | `https://helm.releases.hashicorp.com` | Repository url from which the Helm chart should be obtained | +| - | `features.secrets.vault.helm.version` | String | `0.34.1` | The version of the Helm chart to be installed | +| `--secrets-namespace` | `features.secrets.namespace` | String | `secrets` | Optional defines the kubernetes namespace for secrets. | ### Tool: Ingress -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--ingress` | `features.ingress.active` | Boolean | `false` | Sets and enables Ingress Controller | -| `--ingress-image` | `features.ingress.helm.image` | String | `` | The image of the Helm chart to be installed | -| - | `features.ingress.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `features.ingress.helm.chart` | String | `traefik` | Name of the Helm chart | -| - | `features.ingress.helm.repoURL` | String | `https://traefik.github.io/charts` | Repository url from which the Helm chart should be obtained | -| - | `features.ingress.helm.version` | String | `39.0.0` | The version of the Helm chart to be installed | -| `--ingress-namespace` | `features.ingress.ingressNamespace` | String | `ingress` | Optional defines the kubernetes namespace for Ingress Controller | +| CLI | Config key | Type | Default | Description | +|:----------------------|:------------------------------------|:--------|:-----------------------------------|:-----------------------------------------------------------------------------------------------------------------------| +| `--ingress` | `features.ingress.active` | Boolean | `false` | Sets and enables Ingress Controller | +| `--ingress-image` | `features.ingress.helm.image` | String | `` | The image of the Helm chart to be installed | +| - | `features.ingress.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `features.ingress.helm.chart` | String | `traefik` | Name of the Helm chart | +| - | `features.ingress.helm.repoURL` | String | `https://traefik.github.io/charts` | Repository url from which the Helm chart should be obtained | +| - | `features.ingress.helm.version` | String | `39.0.9` | The version of the Helm chart to be installed | +| `--ingress-namespace` | `features.ingress.ingressNamespace` | String | `ingress` | Optional defines the kubernetes namespace for Ingress Controller | ### Tool: Cert Manager -| CLI | Config key | Type | Default | Description | -| :--- | :--- | :--- | :--- | :--- | -| `--cert-manager` | `features.certManager.active` | Boolean | `false` | Sets and enables Cert Manager | -| `--cert-manager-issuer` | `features.certManager.issuer` | String | `cluster-selfsigned` | Sets and enables Cert Manager | -| `--cert-manager-namespace` | `features.certManager.namespace` | String | `cert-manager` | Optional defines the kubernetes namespace for Cert Manager | -| `--cert-manager-image` | `features.certManager.helm.image` | String | `` | Sets image for Cert Manager | -| `--cert-manager-webhook-image` | `features.certManager.helm.webhookImage` | String | `` | Sets webhook Image for Cert Manager | -| `--cert-manager-cainjector-image` | `features.certManager.helm.cainjectorImage` | String | `` | Sets cainjector Image for Cert Manager | -| `--cert-manager-acme-solver-image` | `features.certManager.helm.acmeSolverImage` | String | `` | Sets acmeSolver Image for Cert Manager | -| `--cert-manager-startup-api-check-image` | `features.certManager.helm.startupAPICheckImage` | String | `` | Sets startupAPICheck Image for Cert Manager | -| - | `features.certManager.helm.values` | Map | `[:]` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | -| - | `features.certManager.helm.chart` | String | `cert-manager` | Name of the Helm chart | -| - | `features.certManager.helm.repoURL` | String | `https://charts.jetstack.io` | Repository url from which the Helm chart should be obtained | -| - | `features.certManager.helm.version` | String | `1.19.4` | The version of the Helm chart to be installed | +| CLI | Config key | Type | Default | Description | +|:-----------------------------------------|:-------------------------------------------------|:--------|:-----------------------------|:-----------------------------------------------------------------------------------------------------------------------| +| `--cert-manager` | `features.certManager.active` | Boolean | `false` | Sets and enables Cert Manager | +| `--cert-manager-issuer` | `features.certManager.issuer` | String | `cluster-selfsigned` | Sets and enables Cert Manager | +| `--cert-manager-namespace` | `features.certManager.namespace` | String | `cert-manager` | Optional defines the kubernetes namespace for Cert Manager | +| `--cert-manager-image` | `features.certManager.helm.image` | String | `` | Sets image for Cert Manager | +| `--cert-manager-webhook-image` | `features.certManager.helm.webhookImage` | String | `` | Sets webhook Image for Cert Manager | +| `--cert-manager-cainjector-image` | `features.certManager.helm.cainjectorImage` | String | `` | Sets cainjector Image for Cert Manager | +| `--cert-manager-acme-solver-image` | `features.certManager.helm.acmeSolverImage` | String | `` | Sets acmeSolver Image for Cert Manager | +| `--cert-manager-startup-api-check-image` | `features.certManager.helm.startupAPICheckImage` | String | `` | Sets startupAPICheck Image for Cert Manager | +| - | `features.certManager.helm.values` | Map | `{}` | Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration | +| - | `features.certManager.helm.chart` | String | `cert-manager` | Name of the Helm chart | +| - | `features.certManager.helm.repoURL` | String | `https://charts.jetstack.io` | Repository url from which the Helm chart should be obtained | +| - | `features.certManager.helm.version` | String | `1.19.4` | The version of the Helm chart to be installed | diff --git a/docs/Deploy-Ingress-Controller.md b/docs/Deploy-Ingress-Controller.md index 1cefe47f4..c52eb9002 100644 --- a/docs/Deploy-Ingress-Controller.md +++ b/docs/Deploy-Ingress-Controller.md @@ -24,7 +24,7 @@ features: In this Example we override the default `controller.replicaCount` (GOP's default is 2). This config file is merged with precedence over the defaults set by -* [the GOP](../argocd/cluster-resources/apps/ingress/templates/ingress-helm-values.ftl.yaml) and +* [the GOP](../argocd/cluster-resources/apps/traefik/templates/ingress-helm-values.ftl.yaml) and * [the charts itself](https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml). # Deploy Ingresses diff --git a/docs/Developers.md b/docs/Developers.md index eecf22435..e2f5d4dc3 100644 --- a/docs/Developers.md +++ b/docs/Developers.md @@ -6,50 +6,51 @@ It provides workarounds or solutions for the given issues. ## Disclaimer -The versions listed in this README may not always reflect the most current release. -Please be aware that newer versions may exist. -The versions are also specified in the `Config.groovy` file, so it is recommended to consult that file for the latest version information. - +The versions listed in this README may not always reflect the most current release. +Please be aware that newer versions may exist. +The versions are also specified in the `Config.java` file, so it is recommended to consult that file for the latest +version information. ## Table of contents - + - [Prerequisites](#prerequisites) - [Testing](#testing) - - [Unit-Tests](#unit-tests) - - [Integration-Tests](#integration-tests) + - [Unit-Tests](#unit-tests) + - [Integration-Tests](#integration-tests) - [Jenkins plugin installation issues](#jenkins-plugin-installation-issues) - - [Solution](#solution) - - [Updating all plugins](#updating-all-plugins) + - [Solution](#solution) + - [Updating all plugins](#updating-all-plugins) - [Local development](#local-development) +- [Testing OIDC locally](#testing-oidc-locally) + - [External OIDC providers](#external-oidc-providers) - [Testing URL separator hyphens](#testing-url-separator-hyphens) - [External registry for development](#external-registry-for-development) - [Testing two registries](#testing-two-registries) - - [Basic test](#basic-test) - - [Proper test](#proper-test) + - [Basic test](#basic-test) + - [Proper test](#proper-test) - [Testing Network Policies locally](#testing-network-policies-locally) - [Emulate an airgapped environment](#emulate-an-airgapped-environment) - - [Setup cluster](#setup-cluster) - - [Install the playground](#install-the-playground) + - [Setup cluster](#setup-cluster) + - [Install the playground](#install-the-playground) - [Notifications / E-Mail](#notifications--e-mail) - [Troubleshooting](#troubleshooting) - - [Using ingresses locally](#using-ingresses-locally) + - [Using ingresses locally](#using-ingresses-locally) - [Generate schema.json](#generate-schemajson) - [Releasing](#releasing) - [Installing ArgoCD Operator](#installing-argocd-operator) - - [Prerequisites:](#prerequisites) - - [Installation Script](#installation-script) - - [Install ingress manually](#install-ingress-manually) + - [Prerequisites:](#prerequisites) + - [Installation Script](#installation-script) + - [Install ingress manually](#install-ingress-manually) ## Prerequisites -- Java 17 -- Groovy +- Java 25 - Maven - Docker - [k3d](https://k3d.io/) @@ -59,12 +60,13 @@ The versions are also specified in the `Config.groovy` file, so it is recommende - [Golang](https://go.dev/doc/install) (only if you plan to use argo-cd operator) - [yq](https://mikefarah.gitbook.io/yq/) (useful for debugging purposes) -To check if you have all necessary tools installed, run the following command. If you don't see any error messages, you are good to go: +To check if you have all necessary tools installed, run the following command. If you don't see any error messages, you +are good to go: + ```bash java -version && mvn -version && docker version && k3d version && kubectl version && helm version ``` - ## Testing 1. There are integration tests implemented by Junit. Classes marked with 'IT' and the end. @@ -94,6 +96,7 @@ mvn clean test ``` where can be one of: + - full - full-prefix @@ -103,8 +106,9 @@ where can be one of: Note: 'operator-*' profiles requires you to install the argo-cd operator in a fresh cluster _before_ deploying the gop. This can be done by running: + ```bash -./scripts/local/install-argocd-operator.sh +make install-operator ``` ## Jenkins plugin installation issues @@ -114,39 +118,43 @@ Trying to overcome this issue we pinned all plugins within `scripts/jenkins/plug These pinned plugins get downloaded within the docker build and saved into a folder as `.hpi` files. Later on when configuring jenkins, we upload all the plugin files with the given version. -Turns out it does not completely circumvent this issue. In some cases jenkins updates these plugins automagically (as it seems) when installing the pinned version fails at first or being installed when resolving dependencies. -This again may lead to a broken jenkins, where some of the automatically updated plugins have changes within their dependencies. These dependencies than again are not updated but pinned and may cause issues. +Turns out it does not completely circumvent this issue. In some cases jenkins updates these plugins automagically (as it +seems) when installing the pinned version fails at first or being installed when resolving dependencies. +This again may lead to a broken jenkins, where some of the automatically updated plugins have changes within their +dependencies. These dependencies than again are not updated but pinned and may cause issues. -Since solving this issue may require some additional deep dive into bash scripts we like to get rid of in the future, we decided to give some hints how to easily solve the issue (and keep the plugins list up to date :]) instead of fixing it with tremendous effort. +Since solving this issue may require some additional deep dive into bash scripts we like to get rid of in the future, we +decided to give some hints how to easily solve the issue (and keep the plugins list up to date :]) instead of fixing it +with tremendous effort. ### Solution * Determine the plugins that cause the issue - * inspecting the logs of the jenkins-pod - * jenkins-ui (http://localhost:9090/manage) + * inspecting the logs of the jenkins-pod + * jenkins-ui (http://localhost:9090/manage) ![Jenkins-UI with broken plugins](images/example-plugin-install-fail.png) * Fix conflicts by updating the plugins with compatible versions - * Update all plugin versions via jenkins-ui (http://localhost:9090/pluginManager/) and restart + * Update all plugin versions via jenkins-ui (http://localhost:9090/pluginManager/) and restart ![Jenkins-UI update plugins](images/update-all-plugins.png) * Verify the plugin installation - * Check if jenkins starts up correctly and builds all example pipelines successfully - * verify installation of all plugins via jenkins-ui (http://localhost:9090/script) executing the following command + * Check if jenkins starts up correctly and builds all example pipelines successfully + * verify installation of all plugins via jenkins-ui (http://localhost:9090/script) executing the following command ![Jenkins-UI plugin list](images/get-plugin-list.png) ```groovy Jenkins.instance.pluginManager.activePlugins.sort().each { - println "${it.shortName}:${it.version}" + println "${it.shortName}:${it.version}" } ``` * Share and publish your plugin updates - * Make sure you have updated `plugins.txt` with working versions of the plugins - * commit and push changes to your feature-branch and submit a pr + * Make sure you have updated `plugins.txt` with working versions of the plugins + * commit and push changes to your feature-branch and submit a pr Note that `plugins.txt` contains the whole dependency tree, including transitive plugin dependencies. The bare minimum of plugins that are needed is this: @@ -162,21 +170,24 @@ scm-manager # Used in example builds workflow-aggregator # Pipelines plugin, used in example builds ``` -Note that, when running locally we also need `kubernetes` and `configuration-as-code` but these are contained in [our -jenkins helm image](https://github.com/cloudogu/jenkins-helm-image/blob/5.8.1-1/Dockerfile) (extracted from the +Note that, when running locally we also need `kubernetes` and `configuration-as-code` but these are contained in [our +jenkins helm image](https://github.com/cloudogu/jenkins-helm-image/blob/5.8.1-1/Dockerfile) (extracted from the [corresponding helm chart version](https://github.com/jenkinsci/helm-charts/blob/jenkins-5.8.1/charts/jenkins/values.yaml)). +### Updating all plugins -### Updating all plugins -To get a minimal list of plugins, start an empty jenkins that uses [the base image of our image](https://github.com/cloudogu/jenkins-helm-image/blob/main/Dockerfile): +To get a minimal list of plugins, start an empty jenkins that +uses [the base image of our image](https://github.com/cloudogu/jenkins-helm-image/blob/main/Dockerfile): ```shell docker run --rm -v $RANDOM-tmp-jenkins:/var/jenkins_home jenkins/jenkins:2.479.2-jdk17 ``` + We need a volume to persist the plugins when jenkins restarts. (These can be cleaned up afterwards like so: `docker volume ls -q | grep jenkins | xargs -I {} docker volume rm {}`). Then + * manually install the bare minimum of plugins mentioned above * extract the plugins using the groovy console as mentioned above * Write the output into `plugins.txt` @@ -186,33 +197,92 @@ We should automate this! ## Local development * Run locally - * Run from IDE (allows for easy debugging), works e.g. with IntelliJ IDEA - Note: If you encounter `error=2, No such file or directory`, - it might be necessary to explicitly set your `PATH` in Run Configuration's Environment Section. - * From shell: - Run + * Run from IDE (allows for easy debugging), works e.g. with IntelliJ IDEA + Note: If you encounter `error=2, No such file or directory`, + it might be necessary to explicitly set your `PATH` in Run Configuration's Environment Section. + * From shell: + Run + ```shell + ./mvnw package -DskipTests + ./mvnw exec:java -Dexec.arguments="" + ``` +* Running inside the container: + * Build and run dev Container: ```shell - ./mvnw package -DskipTests - ./mvnw exec:java -Dexec.arguments="" + docker build -t gitops-playground:dev --build-arg ENV=dev --progress=plain --pull . + docker run --rm -it -u $(id -u) -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \ + --net=host gitops-playground:dev #params ``` -* Running inside the container: - * Build and run dev Container: - ```shell - docker build -t gitops-playground:dev --build-arg ENV=dev --progress=plain --pull . - docker run --rm -it -u $(id -u) -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \ - --net=host gitops-playground:dev #params - ``` - * Hint: You can speed up the process by installing the Jenkins plugins from your filesystem, instead of from the internet. - To do so, download the plugins into a folder, then set this folder vie env var: - `JENKINS_PLUGIN_FOLDER=$(pwd) java -classpath .. # See above`. - A working combination of plugins be extracted from the image: - ```bash - id=$(docker create --pull=always ghcr.io/cloudogu/gitops-playground:main) - docker cp $id:/gitops/jenkins-plugins . - docker rm -v $id - ``` + * Hint: You can speed up the process by installing the Jenkins plugins from your filesystem, instead of from the + internet. + To do so, download the plugins into a folder, then set this folder vie env var: + `JENKINS_PLUGIN_FOLDER=$(pwd) java -classpath .. # See above`. + A working combination of plugins be extracted from the image: + ```bash + id=$(docker create --pull=always ghcr.io/cloudogu/gitops-playground:main) + docker cp $id:/gitops/jenkins-plugins . + docker rm -v $id + ``` + +## Testing OIDC locally + +The GOP can be tested with a local Keycloak realm. SCM-Manager is excluded because it currently has no OIDC support in +GOP. + +Create or reuse a local k3d cluster, install Keycloak and apply the OIDC-enabled GOP profile: + +```bash +make cluster +make keycloak +make image +docker run --rm -t \ + -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \ + --net=host \ + local/gop --profile=keycloak +``` + +`make keycloak` installs the local Keycloak realm from [`docs/oidc/realm-export.json`](oidc/realm-export.json) and +configures CoreDNS so pods can resolve `keycloak.local.gd`. The `keycloak` profile uses the matching typed OIDC config +from [`src/main/resources/application-keycloak.yaml`](../src/main/resources/application-keycloak.yaml). + +Local test users: + +| Username | Password | Group | Expected access | +|:---------|:---------|:-------------|:---------------------------------------------------------| +| `admin` | `admin` | `gop-admins` | Full admin access in Argo CD, Jenkins, Grafana and Vault | +| `user` | `user` | - | No GOP admin permissions | + +The relevant GOP OIDC config fields are `issuerUrl`, `clientId`, `clientSecret`, `scopes` and `adminGroupName`. +`adminGroupName` is intentionally the only authorization mapping GOP configures. New users must not receive admin +permissions unless the identity provider includes them in that group claim. + +Jenkins uses the OIDC security realm and an explicit `escapeHatch` with the configured local Jenkins admin user and +password. Opening Jenkins normally starts the OIDC login flow. Use `http://jenkins.localhost/login` with the configured +local Jenkins admin credentials for the fallback login. The form posts to Jenkins' internal `securityRealm/escapeHatch` +endpoint; that endpoint is not a standalone browser page. If the browser has already started an OIDC login flow and gets +redirected between Jenkins and Keycloak, use a private browser window or clear the Jenkins and Keycloak cookies before +testing the fallback login. This keeps the local fallback login deterministic instead of depending on manually supplied +JCasC snippets. + +### External OIDC providers + +For external providers, create one client per tool and configure the same fields under: + +* `features.argocd.oidc` +* `features.monitoring.oidc` +* `features.secrets.vault.oidc` +* `jenkins.oidc` + +The provider must expose a `groups` claim containing the configured `adminGroupName`. Configure redirect URIs for the +tool URLs that GOP exposes, for example: + +* Argo CD: `/auth/callback` +* Jenkins: `/securityRealm/finishLogin` +* Grafana: `/login/generic_oauth` +* Vault: `/ui/vault/auth/oidc/oidc/callback` ## Testing URL separator hyphens + ```bash docker run --rm -t -u $(id -u) \ -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \ @@ -230,24 +300,28 @@ kubectl get --all-namespaces ingress -o json 2> /dev/null | jq -r '.items[] | .s ## External registry for development If you need to emulate an "external", private registry with credentials, then install it like so: + ```bash helm repo add harbor https://helm.goharbor.io helm upgrade -i my-harbor harbor/harbor -f ./scripts/dev/external-registry-values.yaml --version 1.14.2 --namespace harbor --create-namespace ``` Once it's up and running either create your own private project or just set the existing `library` to private: + ```bash curl -X PUT -u admin:Harbor12345 'http://localhost:30002/api/v2.0/projects/1' -H 'Content-Type: application/json' \ --data-raw '{"metadata":{"public":"false", "id":1,"project_id":1}}' ``` Then either import external images like so (requires `skopeo` but no prior pulling or insecure config necessary): + ```bash skopeo copy docker://alpine/kubectl:1.35.4 --dest-creds admin:Harbor12345 --dest-tls-verify=false docker://localhost:30002/library/kubectl:1.35.4 ``` Alternatively, you could push existing images from your docker daemon. -However, this takes longer (pull first) and you'll have to make sure to add `localhost:30002` to `insecure-registries` in `/etc/docker/daemon.json` and restart your docker daemon first. +However, this takes longer (pull first) and you'll have to make sure to add `localhost:30002` to `insecure-registries` +in `/etc/docker/daemon.json` and restart your docker daemon first. ```bash docker login localhost:30002 -u admin -p Harbor12345 @@ -271,6 +345,7 @@ That is, for most helm charts, you'll need to set an individual value. ## Testing two registries ### Basic test + * Start playground once, * then again with these parameters: `--registry-url=localhost:30000 --registry-proxy-url=localhost:30000 --registry-proxy-username=Proxy --registry-proxy-password=Proxy12345` @@ -284,16 +359,18 @@ That is, for most helm charts, you'll need to set an individual value. * Important: Harbor has to be set up after initializing the cluster, but before installing GOP. Otherwise GOP deploys its own registry, leading to port conflicts: `Service "harbor" is invalid: spec.ports[0].nodePort: Invalid value: 30000: provided port is already allocated` -* By default, `docker run` relies on the `gitops-playground:dev` image. +* By default, `docker run` relies on the `gitops-playground:dev` image. **Setup** To set-up harbor with two projects, you can use the target "prepare-two-registries". + ```shell make prepare-two-registries ``` Afer that, deploy GOP with the generated config file: + ```bash # Create a docker container or use an available image from a registry # docker build -t gop:dev . @@ -312,11 +389,14 @@ docker run --rm -t -u $(id -u) \ ## Testing Network Policies locally -The first increment of our `--netpols` feature is intended to be used on openshift and with an external Cloudogu Ecosystem. +The first increment of our `--netpols` feature is intended to be used on openshift and with an external Cloudogu +Ecosystem. That's why we need to initialize our local cluster with some netpols for everything to work. -* The `-jenkins` , `-scm-manager` and `-registry` namespace needs to be accesible from outside the cluster (so GOP apply via `docker run` has access) -* Emulate OpenShift default netPols: allow network communication inside namespaces and access by ingress controller + +* The `-jenkins` , `-scm-manager` and `-registry` namespace needs to be accesible from outside + the cluster (so GOP apply via `docker run` has access) +* Emulate OpenShift default netPols: allow network communication inside namespaces and access by ingress controller After the cluster is initialized and before GOP is applied, do the following: @@ -381,7 +461,8 @@ done Let's set up our local playground to emulate an airgapped env, as some of our customers have. -Note that with approach bellow, the whole k3d cluster is airgapped with one exception: the Jenkins agents can work around this. +Note that with approach bellow, the whole k3d cluster is airgapped with one exception: the Jenkins agents can work +around this. To be able to run the `docker` plugin in Jenkins (in a k3d cluster that only provides containerd) we mount the host's docker socket into the agents. From there it can start containers which are not airgapped. @@ -394,6 +475,7 @@ like images or helm charts. ### Setup cluster You can prepare the airgapped cluster, by calling make with the "prepare-airgappe-cluster" target: + ```bash make prepare-airgapped-cluster ``` @@ -409,10 +491,12 @@ Don't disconnect from the internet yet, because In this case when the first PVC gets provisioned. * SCMM needs to download the plugins from the internet * Helm repo updates need access to the internet -* But also because we would have to replace the images for registry, scmm, jenkins (several images!) and argocd in the - source code, as there are no parameters to do so. +* Argo CD images are not configurable yet and may still be pulled on demand. +* Jenkins and SCM-Manager images can be pointed at the prepared registry via `jenkins.jenkinsImage` and + `scm.scmManager.scmmImage`; see `scripts/dev/gop_airgapped_config.yaml`. So, start the installation and once Argo CD is running, go offline. + ```bash docker run -it -u $(id -u) \ -v ~/.config/k3d/kubeconfig-airgapped-playground.yaml:/home/.kube/config \ @@ -420,7 +504,6 @@ docker run -it -u $(id -u) \ --net=host gitops-playground:latest --config-file=/gop.yaml -x ``` - ## Notifications / E-Mail Notifications are implemented via Mail. @@ -433,7 +516,8 @@ To test with an external mail server, set up the configuration as follows: ``` For testing, an email can be sent via the Grafana UI. -Go to Alerting > Notifications, here at contact Points click on the right side at provisioned email contact on "View contact point" +Go to Alerting > Notifications, here at contact Points click on the right side at provisioned email contact on "View +contact point" Here you can check if the configuration is implemented correctly and fire up a Testmail. For testing Argo CD, just uncomment some of the defaultTriggers in it's values.yaml and it will send a lot of emails. @@ -441,6 +525,7 @@ For testing Argo CD, just uncomment some of the defaultTriggers in it's values.y ## Troubleshooting When stuck in `Pending` this might be due to volumes not being provisioned + ```bash k get pod -n kube-system NAME READY STATUS RESTARTS AGE @@ -473,29 +558,30 @@ argocd argocd-server traefik argocd.local Where opening for example http://argocd.localhost in your browser should work. The `base-domain` parameters lead to URLs in the following schema: -`..`, e.g. - +`..`, e.g. ## Generate schema.json -Run `GenerateJsonSchema.groovy` from your IDE. +Run `GenerateJsonSchema.java` from your IDE. -Or run build and run via maven and java: +Or build the application and run the generator directly: ````shell -mvn package -DskipTests -java -classpath target/gitops-playground-cli-0.1.jar org.codehaus.groovy.tools.GroovyStarter --main groovy.ui.GroovyMain \ - --classpath src/main/groovy src/main/groovy/com/cloudogu/gitops/cli/GenerateJsonSchema.groovy +./mvnw package -DskipTests +java -classpath target/gitops-playground-cli-0.1.jar com.cloudogu.gitops.cli.GenerateJsonSchema ```` -Or build and run the via docker: +Or build and run it via Docker while mounting the local `docs` directory: ```shell -docker build -t gitops-playground:dev --build-arg ENV=dev --progress=plain . -docker run --rm --entrypoint java gitops-playground:dev -classpath /app/gitops-playground.jar \ - org.codehaus.groovy.tools.GroovyStarter --main groovy.ui.GroovyMain \ - --classpath /app/src/main/groovy /app/src/main/groovy/com/cloudogu/gitops/cli/GenerateJsonSchema.groovy - \ - > docs/configuration.schema.json +docker build -t gitops-playground:dev --build-arg ENV=dev --progress=plain . +docker run --rm \ + -v "$PWD/docs:/work/docs" \ + -w /work \ + --entrypoint java \ + gitops-playground:dev \ + -classpath /app/gitops-playground.jar \ + com.cloudogu.gitops.cli.GenerateJsonSchema ``` ## Releasing @@ -515,7 +601,8 @@ git checkout main \ For now, please start a Jenkins Build of `main` manually. We might introduce tag builds in our Jenkins organization at a later stage. -A GitHub release containing all merged PRs since the last release is create automatically via a [GitHub action](../.github/workflows/create-release.yml) +A GitHub release containing all merged PRs since the last release is create automatically via +a [GitHub action](../.github/workflows/create-release.yml) ## Installing ArgoCD Operator @@ -525,7 +612,7 @@ This guide provides instructions for developers to install the ArgoCD Operator l Ensure you have the following installed on your system: -- Git: For cloning the repository. +- Git: For cloning the repository. - golang: Version >= 1.24 ### Installation Script @@ -541,11 +628,11 @@ make deploy IMG=quay.io/argoprojlabs/argocd-operator:v0.15.0 ### Install ingress manually -The ArgoCD installed via Operator is namespace isolated and therefor can not deploy an ingress-controller, because of global scoped configurations. +The ArgoCD installed via Operator is namespace isolated and therefor can not deploy an ingress-controller, because of +global scoped configurations. GOP has to be startet with ``` --insecure ``` because of we do not use https locally. We have to install the ingress-controller manually: - ```shell helm upgrade --install traefik traefik/traefik --version 4.12.1 --namespace traefik --create-namespace ``` @@ -555,5 +642,5 @@ If the helm repos are not present or up-to-date: ```shell helm repo add traefik https://traefik.github.io/charts helm repo update -helm install traefik traefik/traefik --version 39.0.0 +helm install traefik traefik/traefik --version 39.0.9 ``` diff --git a/docs/code-format/exclude-yamls.png b/docs/code-format/exclude-yamls.png new file mode 100644 index 000000000..bb0a68d4d Binary files /dev/null and b/docs/code-format/exclude-yamls.png differ diff --git a/docs/configuration.schema.json b/docs/configuration.schema.json index 042210e69..91209fe3d 100644 --- a/docs/configuration.schema.json +++ b/docs/configuration.schema.json @@ -1,6 +1,32 @@ { "$schema" : "https://json-schema.org/draft/2020-12/schema", "$defs" : { + "Credentials-nullable" : { + "type" : [ "object", "null" ], + "properties" : { + "passwordKey" : { + "type" : [ "string", "null" ], + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, + "secretName" : { + "type" : [ "string", "null" ], + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, + "secretNamespace" : { + "type" : [ "string", "null" ], + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, + "username" : { + "type" : [ "string", "null" ], + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, + "usernameKey" : { + "type" : [ "string", "null" ], + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + } + }, + "additionalProperties" : false + }, "HelmConfigWithValues-nullable" : { "type" : [ "object", "null" ], "properties" : { @@ -32,6 +58,39 @@ "type" : "string" } }, + "OidcSchema-nullable" : { + "type" : [ "object", "null" ], + "properties" : { + "adminGroupName" : { + "type" : [ "string", "null" ], + "description" : "OIDC group that receives full admin permissions in all OIDC-enabled tools" + }, + "clientId" : { + "type" : [ "string", "null" ], + "description" : "OIDC client ID" + }, + "clientSecret" : { + "type" : [ "string", "null" ], + "description" : "OIDC client secret" + }, + "issuerUrl" : { + "type" : [ "string", "null" ], + "description" : "OIDC issuer URL, for example http://keycloak.local.gd/realms/gop" + }, + "providerName" : { + "type" : [ "string", "null" ], + "description" : "Name of the OIDC provider displayed in tool login screens" + }, + "scopes" : { + "description" : "OIDC scopes requested by the tool", + "type" : [ "array", "null" ], + "items" : { + "type" : "string" + } + } + }, + "additionalProperties" : false + }, "ScmProviderType-nullable" : { "anyOf" : [ { "type" : "null" @@ -54,6 +113,10 @@ "type" : [ "boolean", "null" ], "description" : "Binds ArgoCD controllers to cluster-admin ClusterRole" }, + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "destroy" : { "type" : [ "boolean", "null" ], "description" : "Unroll playground" @@ -141,7 +204,7 @@ } }, "helmReleases" : { - "description" : "", + "description" : "Additional Helm releases to deploy through Argo CD without requiring a content Git repository.", "type" : [ "array", "null" ], "items" : { "type" : "object", @@ -200,30 +263,7 @@ "description" : "If true, creates a Jenkins job, if jenkinsfile exists in one of the content repo's branches." }, "credentials" : { - "type" : [ "object", "null" ], - "properties" : { - "passwordKey" : { - "type" : [ "string", "null" ], - "description" : "Credentials Object to authenticate against content repo. Allows using a K8s Secret" - }, - "secretName" : { - "type" : [ "string", "null" ], - "description" : "Credentials Object to authenticate against content repo. Allows using a K8s Secret" - }, - "secretNamespace" : { - "type" : [ "string", "null" ], - "description" : "Credentials Object to authenticate against content repo. Allows using a K8s Secret" - }, - "username" : { - "type" : [ "string", "null" ], - "description" : "Credentials Object to authenticate against content repo. Allows using a K8s Secret" - }, - "usernameKey" : { - "type" : [ "string", "null" ], - "description" : "Credentials Object to authenticate against content repo. Allows using a K8s Secret" - } - }, - "additionalProperties" : false, + "$ref" : "#/$defs/Credentials-nullable", "description" : "Credentials Object to authenticate against content repo. Allows using a K8s Secret" }, "overwriteMode" : { @@ -320,6 +360,10 @@ "type" : [ "string", "null" ], "description" : "Defines the kubernetes namespace for ArgoCD" }, + "oidc" : { + "$ref" : "#/$defs/OidcSchema-nullable", + "description" : "OIDC Config for this tool. See docs for more infos" + }, "operator" : { "type" : [ "boolean", "null" ], "description" : "Install ArgoCD via an already running ArgoCD Operator" @@ -447,6 +491,10 @@ "mail" : { "type" : [ "object", "null" ], "properties" : { + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "smtpAddress" : { "type" : [ "string", "null" ], "description" : "Sets smtp port of external Mailserver" @@ -532,6 +580,10 @@ "namespace" : { "type" : [ "string", "null" ], "description" : "Optional defines the kubernetes namespace for monitoring." + }, + "oidc" : { + "$ref" : "#/$defs/OidcSchema-nullable", + "description" : "OIDC Config for this tool. See docs for more infos" } }, "additionalProperties" : false, @@ -625,6 +677,10 @@ } ], "description" : "Installs Hashicorp vault and the external secrets operator. Possible values: dev, prod." }, + "oidc" : { + "$ref" : "#/$defs/OidcSchema-nullable", + "description" : "OIDC Config for this tool. See docs for more infos" + }, "url" : { "type" : [ "string", "null" ], "description" : "Sets url for vault ui" @@ -659,14 +715,26 @@ "type" : "string" } }, + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "helm" : { "$ref" : "#/$defs/HelmConfigWithValues-nullable", "description" : "Common Config parameters for the Helm package manager: Name of Chart (chart), URl of Helm-Repository (repoURL) and Chart Version (version). Note: These config is intended to obtain the chart from a different source (e.g. in air-gapped envs), not to use a different version of a helm chart. Using a different helm chart or version to the one used in the GOP version will likely cause errors." }, + "jenkinsImage" : { + "type" : [ "string", "null" ], + "description" : "Sets image for Jenkins" + }, "mavenCentralMirror" : { "type" : [ "string", "null" ], "description" : "URL for maven mirror, used by applications built in Jenkins" }, + "metricsCredentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "metricsPassword" : { "type" : [ "string", "null" ], "description" : "Mandatory when jenkins-url is set and monitoring enabled" @@ -679,6 +747,10 @@ "type" : [ "string", "null" ], "description" : "Optional defines the kubernetes namespace for Jenkins." }, + "oidc" : { + "$ref" : "#/$defs/OidcSchema-nullable", + "description" : "OIDC Config for this tool. See docs for more infos" + }, "password" : { "type" : [ "string", "null" ], "description" : "Mandatory when jenkins-url is set" @@ -713,6 +785,10 @@ "gitlab" : { "type" : [ "object", "null" ], "properties" : { + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "parentGroupId" : { "type" : [ "string", "null" ], "description" : "Main Group for Gitlab where the GOP creates it's groups/repos" @@ -736,6 +812,10 @@ "scmManager" : { "type" : [ "object", "null" ], "properties" : { + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "internal" : { "type" : [ "boolean", "null" ], "description" : "SCM for Central Management is running on the same cluster, so k8s internal URLs can be used for access" @@ -758,7 +838,7 @@ } }, "additionalProperties" : false, - "description" : "Config for GITLAB" + "description" : "Config for SCM-Manager" }, "scmProviderType" : { "$ref" : "#/$defs/ScmProviderType-nullable", @@ -783,6 +863,10 @@ "type" : [ "boolean", "null" ], "description" : "Create image pull secrets for registry and proxy-registry for all GOP namespaces and helm charts. Uses proxy-username, read-only-username or registry-username (in this order). Use this if your cluster is not auto-provisioned with credentials for your private registries or if you configure individual helm images to be pulled from the proxy-registry that requires authentication." }, + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "helm" : { "$ref" : "#/$defs/HelmConfigWithValues-nullable", "description" : "Common Config parameters for the Helm package manager: Name of Chart (chart), URl of Helm-Repository (repoURL) and Chart Version (version). Note: These config is intended to obtain the chart from a different source (e.g. in air-gapped envs), not to use a different version of a helm chart. Using a different helm chart or version to the one used in the GOP version will likely cause errors." @@ -803,6 +887,10 @@ "type" : [ "string", "null" ], "description" : "Optional when registry-url is set" }, + "proxyCredentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "proxyPassword" : { "type" : [ "string", "null" ], "description" : "Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set." @@ -819,6 +907,10 @@ "type" : [ "string", "null" ], "description" : "Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set." }, + "readOnlyCredentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "readOnlyPassword" : { "type" : [ "string", "null" ], "description" : "Optional alternative password for registry-url with read-only permissions that is used when create-image-pull-secrets is set." @@ -845,6 +937,10 @@ "gitlab" : { "type" : [ "object", "null" ], "properties" : { + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "gitOpsUsername" : { "type" : [ "string", "null" ], "description" : "Username for the Gitops User" @@ -876,6 +972,10 @@ "scmManager" : { "type" : [ "object", "null" ], "properties" : { + "credentials" : { + "$ref" : "#/$defs/Credentials-nullable", + "description" : "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys." + }, "gitOpsUsername" : { "type" : [ "string", "null" ], "description" : "Username for the Gitops User" @@ -892,6 +992,10 @@ "type" : [ "string", "null" ], "description" : "Mandatory when scmm-url is set" }, + "scmmImage" : { + "type" : [ "string", "null" ], + "description" : "Sets image for SCM-Manager" + }, "skipPlugins" : { "type" : [ "boolean", "null" ], "description" : "Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades." @@ -910,7 +1014,7 @@ } }, "additionalProperties" : false, - "description" : "Config for GITLAB" + "description" : "Config for SCM-Manager" }, "scmProviderType" : { "$ref" : "#/$defs/ScmProviderType-nullable", diff --git a/docs/deploy-local-openshift.md b/docs/deploy-local-openshift.md new file mode 100644 index 000000000..e074f3dba --- /dev/null +++ b/docs/deploy-local-openshift.md @@ -0,0 +1,276 @@ +# GOP Deployment from Local Docker Image to Local OpenShift (CRC) + +This guide provides a step-by-step walkthrough for deploying the local Docker image `local/gop:latest` to a local +OpenShift environment (CodeReady Containers / OpenShift Local) using the internal OpenShift Image Registry. + +--- + +## 1. Prerequisites Check + +- **OpenShift Local (CRC):** Running (`crc status`) +- **OpenShift CLI (`oc`):** Installed and operational +- **Docker / Podman:** Running with the local image `local/gop:latest` + +### Recommend + +Start openshift with more cpu and memory! + +### CRC should use different default ports, because K3d has to use 80/443. + +```bash + crc config set ingress-http-port 8880 + crc config set ingress-https-port 8843 +``` + +### Start with more resources, because CRC Argocd needs more cpu and memory. + +```bash +crc start --cpus 6 --memory 16384 --disk-size 80 +``` + +Verify local image: + +```bash +docker images | grep gop +``` + +--- + +## 2. Step 1: Enable the Internal OpenShift Image Registry & Expose Route + +In OpenShift Local (CRC), the internal Image Registry is often not exposed via an external route by default. This must +be enabled once with administrator privileges. + +### 2.1 Log in as `kubeadmin` + +Retrieve the password via CRC (if not already known): + +```bash +crc console --credentials +``` + +Log in with admin rights: + +```bash +oc login -u kubeadmin -p https://api.crc.testing:6443 +``` + +### 2.2 Configure Image Registry Operator + +For local test clusters (CRC), set the storage to `emptyDir` and the operator to `Managed`: + +```bash +oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"managementState":"Managed","storage":{"emptyDir":{}}}}' +``` + +### 2.3 Expose the Default Route for External Access + +```bash +oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"defaultRoute":true}}' +``` + +### 2.4 Verify Registry Route + +```bash +oc get route default-route -n openshift-image-registry +``` + +The registry host URL typically resolves to: `default-route-openshift-image-registry.apps-crc.testing`. + +--- + +## 3. Step 2: Create Project / Namespace for GOP + +Create a new project in the OpenShift cluster (can be executed as `developer` or `kubeadmin`): + +```bash +# Optional: Switch to developer user +oc login -u developer -p developer https://api.crc.testing:6443 + +# Create new project +oc new-project gop +``` + +--- + +## 4. Step 3: Authenticate Docker with OpenShift Registry + +To allow Docker to push the local image into the cluster, authenticate using the current OpenShift session token: + +```bash +docker login -u $(oc whoami) -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing +``` + +> **Note on SSL/TLS Certificate Errors (x509: certificate signed by unknown authority):** +> Add the registry domain to `insecure-registries` in Docker Desktop under **Settings** → **Docker Engine**: +> ```json +> { +> "insecure-registries": [ +> "default-route-openshift-image-registry.apps-crc.testing" +> ] +> } +> ``` +> Then click *Apply & restart*. + +--- + +## 5. Step 4: Tag & Push Local Image to OpenShift + +Tag the local image with the registry URL and target project (`gop`), then push: + +```bash +# Tag image +docker tag local/gop:latest default-route-openshift-image-registry.apps-crc.testing/gop/gop:latest + +# Push image to OpenShift Registry +docker push default-route-openshift-image-registry.apps-crc.testing/gop/gop:latest +``` + +### Verify ImageStream in Cluster + +After pushing, OpenShift automatically creates an `ImageStream`: + +```bash +oc get is -n gop +oc describe is gop -n gop +``` + +--- + +## 6. Step 5: Configure ServiceAccount, RBAC & OpenShift SCCs + +GOP operates as an orchestrator job that provisions tools (SCM-Manager, Argo CD, Vault, etc.) across various namespaces. +By default, OpenShift blocks containers using root groups (`fsGroup: 0`) under the `restricted-v2` SCC. We therefore set +up the permissions and pre-create the required tool namespaces with the `anyuid` SCC. + +### 5.1 Create ServiceAccount & ClusterRoleBinding for GOP + +Manifest definition is located +in [scripts/local-openshift/manifest/gop-rbac.yaml](../scripts/local-openshift/manifest/gop-rbac.yaml): + +```bash +# Apply manifest (as kubeadmin) +oc apply -f scripts/local-openshift/manifest/gop-rbac.yaml +``` + +### 5.2 Assign OpenShift SCC `anyuid` to GOP ServiceAccount + +Allows the GOP installer pod itself to start: + +```bash +oc adm policy add-scc-to-user anyuid -z gop-sa -n gop +``` + +### 5.3 Pre-create Tool Namespaces & Assign `anyuid` SCC + +Ensures that tools deployed by GOP (e.g. SCM-Manager with `fsGroup: 0`) can start without security constraint errors: + +```bash +# 1. Pre-create namespaces for initial tools +oc create namespace scm-manager || true +oc create namespace argocd || true +oc create namespace vault || true + +# 2. Grant anyuid SCC to all ServiceAccounts in these namespaces +oc adm policy add-scc-to-group anyuid system:serviceaccounts:scm-manager +oc adm policy add-scc-to-group anyuid system:serviceaccounts:argocd +oc adm policy add-scc-to-group anyuid system:serviceaccounts:vault +``` + +> **Note for additional tools (e.g. Monitoring / Prometheus):** +> When activating additional components later, simply execute the same commands for the new namespace: +> `oc create namespace monitoring || true` +> `oc adm policy add-scc-to-group anyuid system:serviceaccounts:monitoring` + +### 5.4 Clean Up Previous Failed Jobs (if any) + +```bash +oc delete job -l app.kubernetes.io/name=gop-helm -n gop || true +oc delete job gop-installer-job -n gop || true +``` + +--- + +## 7. Step 6: Run GOP in OpenShift Cluster + +Two execution options are available: + +### Option A: Installation via Helm (Path A with `gop-values.yaml`) + +Configuration file is located +at [scripts/local-openshift/helm/gop-values.yaml](../scripts/local-openshift/helm/gop-values.yaml): + +```bash +helm upgrade -i gop oci://ghcr.io/cloudogu/gop-helm -n gop -f scripts/local-openshift/helm/gop-values.yaml +``` + +Stream live installer logs: + +```bash +oc logs -f -l app.kubernetes.io/name=gop-helm -n gop +``` + +--- + +### Option B: Direct Manifest via OpenShift Job (Path B with `gop-job.yaml`) + +Job manifest is located +at [scripts/local-openshift/manifest/gop-job.yaml](../scripts/local-openshift/manifest/gop-job.yaml): + +```bash +# Clean up prior installer job (if present) +oc delete job gop-installer-job -n gop || true + +# Apply manifest and start job +oc apply -f scripts/local-openshift/manifest/gop-job.yaml +``` + +Stream live installer logs: + +```bash +oc logs -f job/gop-installer-job -n gop +``` + +--- + +## 8. Step 7: Access Installed Tools & Routes + +Once the GOP installer job finishes with status `Completed`, the deployed tools are accessible via OpenShift Routes. + +### 8.1 List All Created Routes + +```bash +oc get routes -A +``` + +### 8.2 Default URLs with `baseUrl: http://apps-crc.testing` + +* **SCM-Manager:** `http://scmm.apps-crc.testing` +* **Argo CD:** `http://argocd.apps-crc.testing` +* **Vault:** `http://vault.apps-crc.testing` +* **Grafana / Metrics:** `http://grafana.apps-crc.testing` + +**Default Credentials:** + +* **Username:** `admin` +* **Password:** `admin` (or the configured value in `gop-values.yaml`) + +--- + +## 9. Troubleshooting & Common Commands + +- **Rerun GOP Job (Helm):** + ```bash + oc delete job -l app.kubernetes.io/name=gop-helm -n gop + helm upgrade -i gop oci://ghcr.io/cloudogu/gop-helm -n gop -f scripts/local-openshift/helm/gop-values.yaml + ``` +- **Rerun GOP Job (Manifest):** + ```bash + oc delete job gop-installer-job -n gop + oc apply -f scripts/local-openshift/manifest/gop-job.yaml + ``` +- **Update Image after Local Code Changes:** + ```bash + docker tag local/gop:latest default-route-openshift-image-registry.apps-crc.testing/gop/gop:latest + docker push default-route-openshift-image-registry.apps-crc.testing/gop/gop:latest + ``` diff --git a/docs/oidc/credentials.yaml b/docs/oidc/credentials.yaml new file mode 100644 index 000000000..32afd970a --- /dev/null +++ b/docs/oidc/credentials.yaml @@ -0,0 +1,13 @@ +# $schema: https://raw.githubusercontent.com/cloudogu/gitops-playground/main/docs/configuration.schema.json +application: + baseUrl: http://localhost + password: "admin" +jenkins: + password: "admin" + metricsUsername: "admin" + metricsPassword: "admin" +registry: + password: "admin" +scm: + scmManager: + password: "admin" diff --git a/docs/oidc/oidc-local.yaml b/docs/oidc/oidc-local.yaml new file mode 100644 index 000000000..622a2695c --- /dev/null +++ b/docs/oidc/oidc-local.yaml @@ -0,0 +1,32 @@ +features: + argocd: + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: argocd + clientSecret: "Cq3U2Dvx6WR5ep2ZLH8rZXNKFNcIX5Zx" + adminGroupName: gop-admins + + secrets: + vault: + oidc: + providerName: Keycloak + issuerUrl: "http://keycloak.local.gd/realms/gop" + clientId: "vault" + clientSecret: "XySg7UyAzVkcaU4Visqfe9EChDvARYA1" + adminGroupName: gop-admins + + monitoring: + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: grafana + clientSecret: "46uALR2HeqLPtJxzkkn6tK6FVC9vMmRc" + adminGroupName: gop-admins +jenkins: + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: jenkins + clientSecret: "mtuIbUdggI2ZSy7jSsQW0WkwK0CwMbvO" + adminGroupName: gop-admins \ No newline at end of file diff --git a/docs/oidc/oidc.md b/docs/oidc/oidc.md new file mode 100644 index 000000000..8dce58942 --- /dev/null +++ b/docs/oidc/oidc.md @@ -0,0 +1,199 @@ +# Deploy a local Keycloak as OIDC provider + +This setup installs Keycloak into the `keycloak` namespace and imports the local GOP realm from +[`realm-export.json`](./realm-export.json). The realm is meant for local demos only. It contains the clients for Argo +CD, +Jenkins, Vault and Grafana. SCM-Manager does not support OIDC yet. + +The imported realm and [`oidc-local.yaml`](./oidc-local.yaml) use the same checked-in demo client secrets. Replace them +before using this outside of a local throwaway cluster. + +## Prerequisites + +- A GOP cluster with the current `kubectl` context pointing to it. +- Helm 3 installed locally. +- The GOP local ingress setup, usually with `http://localhost` as base URL. This gives the GOP application URLs such as + `http://argocd.localhost`, `http://jenkins.localhost`, `http://grafana.localhost` and `http://vault.localhost`. +- A Traefik ingress controller for the `keycloak.local.gd` ingress. In a fresh GOP k3d cluster this controller is + created + when GOP is applied with `--ingress` or a profile that enables ingress, for example `--profile=full`. +- Keycloak intentionally uses `keycloak.local.gd` instead of `keycloak.localhost`. Pods often resolve any + `*.localhost` name to their own loopback address before asking CoreDNS, so a CoreDNS rewrite for + `keycloak.localhost` is not reliable. + +Run the following commands from the repository root. + +For the usual local developer setup, prefer the make target and the matching GOP profile: + +```bash +make keycloak +docker run --rm -t \ + -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \ + --net=host \ + local/gop --profile=keycloak +``` + +The manual steps below are useful when you want to inspect or adapt individual parts of the Keycloak setup. + +## Reapply GOP from a clean local k3d cluster + +If you already have a local GOP instance and want to reapply it from scratch, delete the current k3d cluster first. +This removes GOP, Keycloak, persistent volumes and the generated kubeconfig for that cluster: + +```bash +k3d cluster delete gitops-playground +``` + +Then recreate the cluster with the GOP cluster bootstrap script: + +```bash +bash scripts/init-cluster.sh +``` + +If you do not have this repository checked out or want to use the published script, use: + +```bash +bash <(curl -s https://raw.githubusercontent.com/cloudogu/gitops-playground/main/scripts/init-cluster.sh) +``` + +After the cluster exists again, continue with the steps below: create the Keycloak realm ConfigMap, install Keycloak, +configure the CoreDNS rewrite and then apply GOP with `oidc-local.yaml`. + +## 1. Create the realm ConfigMap + +Keycloak imports files from `data/import` during startup when started with `--import-realm`. Store the realm export as a +ConfigMap first: + +```bash +kubectl create namespace keycloak --dry-run=client -o yaml | kubectl apply -f - + +kubectl -n keycloak create configmap keycloak-realm \ + --from-file=realm-export.json=docs/oidc/realm-export.json \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +## 2. Install Keycloak and import the realm + +The realm export is mounted into Keycloak's import directory and imported by Keycloak itself. Do not use the chart's +`keycloakConfigCli` job for this export. The job can lag behind Keycloak's realm-export format and fail on newer fields. + +```bash +helm upgrade --install keycloak oci://registry-1.docker.io/bitnamicharts/keycloak \ + --namespace keycloak \ + --reset-values \ + --set global.security.allowInsecureImages=true \ + --set image.registry=docker.io \ + --set image.repository=bitnamilegacy/keycloak \ + --set postgresql.image.registry=docker.io \ + --set postgresql.image.repository=bitnamilegacy/postgresql \ + --set auth.adminUser=admin \ + --set auth.adminPassword=admin \ + --set production=false \ + --set tls.enabled=false \ + --set proxyHeaders=xforwarded \ + --set hostnameStrict=false \ + --set httpEnabled=true \ + --set extraEnvVars[0].name=KC_HOSTNAME \ + --set extraEnvVars[0].value=keycloak.local.gd \ + --set ingress.enabled=true \ + --set ingress.ingressClassName=traefik \ + --set ingress.hostname=keycloak.local.gd \ + --set ingress.tls=false \ + --set keycloakConfigCli.enabled=false \ + --set extraStartupArgs=--import-realm \ + --set extraVolumes[0].name=realm-import \ + --set extraVolumes[0].configMap.name=keycloak-realm \ + --set extraVolumeMounts[0].name=realm-import \ + --set extraVolumeMounts[0].mountPath=/opt/bitnami/keycloak/data/import/realm-export.json \ + --set extraVolumeMounts[0].subPath=realm-export.json \ + --set extraVolumeMounts[0].readOnly=true +``` + +Wait until Keycloak is running: + +```bash +kubectl -n keycloak rollout status statefulset/keycloak --timeout=10m +kubectl -n keycloak logs statefulset/keycloak --tail=100 +``` + +After the ingress controller is available, Keycloak is reachable at `http://keycloak.local.gd`. The admin console is +available at `http://keycloak.local.gd/admin/` with user `admin` and password `admin`. The imported realm is `gop`. + +## 3. Make `keycloak.local.gd` resolvable from pods + +The browser and the applications must use the same issuer URL: `http://keycloak.local.gd/realms/gop`. Pods inside the +cluster therefore also need to resolve `keycloak.local.gd`. Prefer a CoreDNS rewrite over fixed `hostAliases`, because +the service IP can change. + +Add this line to the CoreDNS `Corefile`, before the `kubernetes` or `forward` plugin: + +```text +rewrite name keycloak.local.gd keycloak.keycloak.svc.cluster.local +``` + +Then restart CoreDNS: + +```bash +kubectl -n kube-system edit configmap coredns +kubectl -n kube-system rollout restart deployment/coredns +``` + +You can verify the issuer from any pod that has `curl`: + +```bash +kubectl run oidc-check --rm -it --restart=Never --image=curlimages/curl -- \ + curl -s http://keycloak.local.gd/realms/gop/.well-known/openid-configuration +``` + +## 4. Apply GOP with the OIDC configuration + +When applying or re-applying GOP, include [`credentials.yaml`](./credentials.yaml) and +[`oidc-local.yaml`](./oidc-local.yaml). The credentials file pins the local demo passwords and configures Jenkins' +Prometheus scrape to use the OIDC escape hatch account. With the published container image this means mounting both +files +into the container: + +```bash +export CLUSTER_NAME=gitops-playground + +docker run --rm -t --pull=always \ + -v ~/.config/k3d/kubeconfig-${CLUSTER_NAME}.yaml:/home/.kube/config \ + -v "$PWD/docs/oidc/credentials.yaml:/tmp/credentials.yaml:ro" \ + -v "$PWD/docs/oidc/oidc-local.yaml:/tmp/oidc-local.yaml:ro" \ + --net=host \ + ghcr.io/cloudogu/gitops-playground \ + --profile=full \ + --config-file=/tmp/credentials.yaml \ + --config-file=/tmp/oidc-local.yaml +``` + +For local development from this repository, use the same config file directly: + +```bash +./mvnw exec:java -Dexec.arguments="--profile=full --config-file=docs/oidc/credentials.yaml --config-file=docs/oidc/oidc-local.yaml" +``` + +## Troubleshooting + +- `Substituted images detected`: this warning is expected for the `bitnamilegacy` images used by the local setup. It is + not fatal by itself. Check the actual pod state and events with `kubectl -n keycloak describe pod keycloak-0`. + During the first start Keycloak can take around two minutes until the Quarkus augmentation, database initialization and + realm import are complete. Temporary readiness probe failures with `connection refused` are expected during that time. +- `Script upload is disabled`: the export still contains Keycloak Authorization Services JavaScript policies. The + checked-in export intentionally removes Authorization Services from the demo OIDC clients because Argo CD, Jenkins, + Vault and Grafana only need normal OIDC clients. +- `http://keycloak.local.gd` does not open: check that the Traefik ingress controller is installed and that your machine + resolves `*.localhost`. If your OS does not resolve `*.localhost`, use the GOP local ingress alternatives described in + [Deploy Ingress Controller](../Deploy-Ingress-Controller.md#local-ingresses) and adjust the URLs in + [`realm-export.json`](./realm-export.json) and [`oidc-local.yaml`](./oidc-local.yaml). +- Apps fail OIDC discovery from inside the cluster: check the CoreDNS rewrite and verify the well-known endpoint from a + pod. +- Client authentication fails: make sure the client secrets in Keycloak match [`oidc-local.yaml`](./oidc-local.yaml). + The demo export in this repository already matches the file. +- Jenkins fails during startup with + `No hudson.security.SecurityRealm implementation found for oic`: Jenkins is reading the OIDC JCasC file before the + OIDC plugin is available. Install the Jenkins OIDC boot plugins through the Jenkins Helm values so they are present + during controller boot; installing them later through GOP's post-start plugin upload is too late for JCasC. +- Jenkins redirects between `http://jenkins.localhost/login` and Keycloak during fallback login: the fallback form at + `/login` posts to the configured `escapeHatch`, but stale OIDC browser state can continue an already started Keycloak + login flow. Use a private browser window or clear the Jenkins and Keycloak cookies before testing the fallback login. diff --git a/docs/oidc/realm-export.json b/docs/oidc/realm-export.json new file mode 100644 index 000000000..e4fa2a26e --- /dev/null +++ b/docs/oidc/realm-export.json @@ -0,0 +1,2923 @@ +{ + "id": "cf8b9dc7-b106-4178-b8d6-9e2ad51ae8c7", + "realm": "gop", + "notBefore": 0, + "defaultSignatureAlgorithm": "RS256", + "revokeRefreshToken": false, + "refreshTokenMaxReuse": 0, + "accessTokenLifespan": 300, + "accessTokenLifespanForImplicitFlow": 900, + "ssoSessionIdleTimeout": 1800, + "ssoSessionMaxLifespan": 36000, + "ssoSessionIdleTimeoutRememberMe": 0, + "ssoSessionMaxLifespanRememberMe": 0, + "offlineSessionIdleTimeout": 2592000, + "offlineSessionMaxLifespanEnabled": false, + "offlineSessionMaxLifespan": 5184000, + "clientSessionIdleTimeout": 0, + "clientSessionMaxLifespan": 0, + "clientOfflineSessionIdleTimeout": 0, + "clientOfflineSessionMaxLifespan": 0, + "accessCodeLifespan": 60, + "accessCodeLifespanUserAction": 300, + "accessCodeLifespanLogin": 1800, + "actionTokenGeneratedByAdminLifespan": 43200, + "actionTokenGeneratedByUserLifespan": 300, + "oauth2DeviceCodeLifespan": 600, + "oauth2DevicePollingInterval": 5, + "enabled": true, + "sslRequired": "external", + "registrationAllowed": false, + "registrationEmailAsUsername": false, + "rememberMe": false, + "verifyEmail": false, + "loginWithEmailAllowed": true, + "duplicateEmailsAllowed": false, + "resetPasswordAllowed": false, + "editUsernameAllowed": false, + "bruteForceProtected": false, + "permanentLockout": false, + "maxTemporaryLockouts": 0, + "bruteForceStrategy": "MULTIPLE", + "maxFailureWaitSeconds": 900, + "minimumQuickLoginWaitSeconds": 60, + "waitIncrementSeconds": 60, + "quickLoginCheckMilliSeconds": 1000, + "maxDeltaTimeSeconds": 43200, + "failureFactor": 30, + "roles": { + "realm": [ + { + "id": "88720d9f-5a37-419f-835e-e27335df274e", + "name": "uma_authorization", + "description": "${role_uma_authorization}", + "composite": false, + "clientRole": false, + "containerId": "cf8b9dc7-b106-4178-b8d6-9e2ad51ae8c7", + "attributes": {} + }, + { + "id": "3d52a0ed-0710-4726-b0a6-8391a0974ade", + "name": "offline_access", + "description": "${role_offline-access}", + "composite": false, + "clientRole": false, + "containerId": "cf8b9dc7-b106-4178-b8d6-9e2ad51ae8c7", + "attributes": {} + }, + { + "id": "5c67838d-09d4-498e-b237-06439f12e298", + "name": "default-roles-gop", + "description": "${role_default-roles}", + "composite": true, + "composites": { + "realm": [ + "offline_access", + "uma_authorization" + ], + "client": { + "account": [ + "manage-account", + "view-profile" + ] + } + }, + "clientRole": false, + "containerId": "cf8b9dc7-b106-4178-b8d6-9e2ad51ae8c7", + "attributes": {} + } + ], + "client": { + "realm-management": [ + { + "id": "b979701c-725b-4d3f-a463-a6985dadb484", + "name": "view-authorization", + "description": "${role_view-authorization}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "3346c597-4938-4176-b7a8-caf4c81ea6fd", + "name": "view-realm", + "description": "${role_view-realm}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "d6371c1e-84fc-4240-bb5b-c1ba70039308", + "name": "realm-admin", + "description": "${role_realm-admin}", + "composite": true, + "composites": { + "client": { + "realm-management": [ + "view-realm", + "view-authorization", + "view-events", + "query-groups", + "manage-authorization", + "view-identity-providers", + "manage-events", + "manage-identity-providers", + "manage-clients", + "create-client", + "view-clients", + "manage-users", + "query-realms", + "view-users", + "impersonation", + "query-clients", + "query-users", + "manage-realm" + ] + } + }, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "f923e58f-a094-4c2f-bd17-5bac688c750e", + "name": "view-events", + "description": "${role_view-events}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "fea9b5e8-6d4d-46a5-bb9f-65b221ff6578", + "name": "query-groups", + "description": "${role_query-groups}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "ecbb9f2b-0364-43a9-93ec-d4331c49aa0b", + "name": "manage-authorization", + "description": "${role_manage-authorization}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "3df187a3-3d7e-4d31-8020-84895246b727", + "name": "manage-events", + "description": "${role_manage-events}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "b0b7bf5c-f07a-42ae-85f6-19465b48a255", + "name": "manage-identity-providers", + "description": "${role_manage-identity-providers}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "40c5a460-46d9-4c19-87e5-699dd1196935", + "name": "view-identity-providers", + "description": "${role_view-identity-providers}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "68b9b73a-01ca-4484-980d-6156c2604414", + "name": "manage-clients", + "description": "${role_manage-clients}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "b711020c-9bf9-48e8-90b0-255dd05008ec", + "name": "create-client", + "description": "${role_create-client}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "67bf97c7-3096-4f8a-a34c-4801349d8c41", + "name": "manage-users", + "description": "${role_manage-users}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "815867bd-a0f8-4cf1-b3ae-10f0bc11881a", + "name": "view-clients", + "description": "${role_view-clients}", + "composite": true, + "composites": { + "client": { + "realm-management": [ + "query-clients" + ] + } + }, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "12b78178-05f7-42cc-8286-164300e97ef7", + "name": "query-realms", + "description": "${role_query-realms}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "c6699d8e-2f21-49c1-ae95-90ff04a6efe8", + "name": "view-users", + "description": "${role_view-users}", + "composite": true, + "composites": { + "client": { + "realm-management": [ + "query-groups", + "query-users" + ] + } + }, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "723323fb-0c5e-4817-acc7-84ecd14bdb59", + "name": "impersonation", + "description": "${role_impersonation}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "84a543a7-1450-4213-bedf-95427b0cb46b", + "name": "query-clients", + "description": "${role_query-clients}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "646b3929-85fd-4df3-b931-3861b9576f91", + "name": "query-users", + "description": "${role_query-users}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + }, + { + "id": "5fb51589-215d-4b97-99eb-08fcfd736b13", + "name": "manage-realm", + "description": "${role_manage-realm}", + "composite": false, + "clientRole": true, + "containerId": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "attributes": {} + } + ], + "grafana": [ + { + "id": "b6f5569c-eda4-4942-81a7-413de086be8f", + "name": "uma_protection", + "composite": false, + "clientRole": true, + "containerId": "2ed0d5a0-6efa-43c5-bc9c-07f51f153459", + "attributes": {} + } + ], + "security-admin-console": [], + "argocd": [ + { + "id": "74553f7c-6cf2-4ef3-928a-7103f4910498", + "name": "uma_protection", + "composite": false, + "clientRole": true, + "containerId": "b552400b-2da1-4c2e-ac5d-d66023bd762f", + "attributes": {} + } + ], + "jenkins": [ + { + "id": "b17fba3e-79dd-4fc7-9895-31d64e077186", + "name": "uma_protection", + "composite": false, + "clientRole": true, + "containerId": "662df4d9-5df0-4954-8c8f-7cb954b257e3", + "attributes": {} + } + ], + "admin-cli": [], + "account-console": [], + "broker": [ + { + "id": "aa78439e-81b7-492b-9bbf-d5096b0b332b", + "name": "read-token", + "description": "${role_read-token}", + "composite": false, + "clientRole": true, + "containerId": "5f194498-254e-4759-881f-f900588630b1", + "attributes": {} + } + ], + "account": [ + { + "id": "f4a08f84-da99-41d9-863c-b02694e4096d", + "name": "view-groups", + "description": "${role_view-groups}", + "composite": false, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "2ca36430-002c-4525-8dfd-8e3cb03eaeb7", + "name": "manage-account", + "description": "${role_manage-account}", + "composite": true, + "composites": { + "client": { + "account": [ + "manage-account-links" + ] + } + }, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "58f52674-e325-4554-9496-9e8ed71ffc63", + "name": "delete-account", + "description": "${role_delete-account}", + "composite": false, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "f93ee4d4-c380-4c33-9cc2-2aef09e04043", + "name": "manage-consent", + "description": "${role_manage-consent}", + "composite": true, + "composites": { + "client": { + "account": [ + "view-consent" + ] + } + }, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "f7db8a9d-1caa-404e-85cc-e8cf5df8744d", + "name": "view-profile", + "description": "${role_view-profile}", + "composite": false, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "96956ffd-5135-4aaf-8d93-c308cb7740a6", + "name": "view-consent", + "description": "${role_view-consent}", + "composite": false, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "f47bd50c-4a92-4527-b7cb-47cffd1a5b2b", + "name": "manage-account-links", + "description": "${role_manage-account-links}", + "composite": false, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + }, + { + "id": "c43b7f84-9f2f-452d-af55-04eed9b62861", + "name": "view-applications", + "description": "${role_view-applications}", + "composite": false, + "clientRole": true, + "containerId": "dad19cc6-6518-469c-bea3-de245650e4b5", + "attributes": {} + } + ], + "vault": [ + { + "id": "c5ca7240-33e6-4225-89e4-69e5d1f00de7", + "name": "uma_protection", + "composite": false, + "clientRole": true, + "containerId": "544c7f5d-9cc9-4a1e-949a-895aa7ab147e", + "attributes": {} + } + ] + } + }, + "groups": [ + { + "id": "c53f9f52-70fd-4a44-8434-b2e0afde2ed9", + "name": "gop-admins", + "path": "/gop-admins", + "attributes": {}, + "realmRoles": [], + "clientRoles": {}, + "subGroups": [] + } + ], + "defaultRole": { + "id": "5c67838d-09d4-498e-b237-06439f12e298", + "name": "default-roles-gop", + "description": "${role_default-roles}", + "composite": true, + "clientRole": false, + "containerId": "cf8b9dc7-b106-4178-b8d6-9e2ad51ae8c7" + }, + "requiredCredentials": [ + "password" + ], + "otpPolicyType": "totp", + "otpPolicyAlgorithm": "HmacSHA1", + "otpPolicyInitialCounter": 0, + "otpPolicyDigits": 6, + "otpPolicyLookAheadWindow": 1, + "otpPolicyPeriod": 30, + "otpPolicyCodeReusable": false, + "otpSupportedApplications": [ + "totpAppFreeOTPName", + "totpAppGoogleName", + "totpAppMicrosoftAuthenticatorName" + ], + "localizationTexts": {}, + "webAuthnPolicyRpEntityName": "keycloak", + "webAuthnPolicySignatureAlgorithms": [ + "ES256", + "RS256" + ], + "webAuthnPolicyRpId": "", + "webAuthnPolicyAttestationConveyancePreference": "not specified", + "webAuthnPolicyAuthenticatorAttachment": "not specified", + "webAuthnPolicyRequireResidentKey": "not specified", + "webAuthnPolicyUserVerificationRequirement": "not specified", + "webAuthnPolicyCreateTimeout": 0, + "webAuthnPolicyAvoidSameAuthenticatorRegister": false, + "webAuthnPolicyAcceptableAaguids": [], + "webAuthnPolicyExtraOrigins": [], + "webAuthnPolicyPasswordlessRpEntityName": "keycloak", + "webAuthnPolicyPasswordlessSignatureAlgorithms": [ + "ES256", + "RS256" + ], + "webAuthnPolicyPasswordlessRpId": "", + "webAuthnPolicyPasswordlessAttestationConveyancePreference": "not specified", + "webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified", + "webAuthnPolicyPasswordlessRequireResidentKey": "not specified", + "webAuthnPolicyPasswordlessUserVerificationRequirement": "not specified", + "webAuthnPolicyPasswordlessCreateTimeout": 0, + "webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": false, + "webAuthnPolicyPasswordlessAcceptableAaguids": [], + "webAuthnPolicyPasswordlessExtraOrigins": [], + "users": [ + { + "id": "67462229-3abf-4da5-812c-30d27068fc5f", + "username": "admin", + "firstName": "GOP", + "lastName": "Admin", + "email": "admin@example.org", + "emailVerified": true, + "enabled": true, + "totp": false, + "credentials": [ + { + "type": "password", + "value": "admin", + "temporary": false + } + ], + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "default-roles-gop" + ], + "notBefore": 0, + "groups": [ + "/gop-admins" + ] + }, + { + "id": "348d24de-24d9-494e-8470-bc899b6e33df", + "username": "user", + "firstName": "GOP", + "lastName": "User", + "email": "user@example.org", + "emailVerified": true, + "enabled": true, + "totp": false, + "credentials": [ + { + "type": "password", + "value": "user", + "temporary": false + } + ], + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "default-roles-gop" + ], + "notBefore": 0, + "groups": [] + }, + { + "id": "f88d9807-35f6-4f31-ac26-124a9d59373e", + "username": "service-account-argocd", + "emailVerified": false, + "enabled": true, + "createdTimestamp": 1779192964239, + "totp": false, + "serviceAccountClientId": "argocd", + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "default-roles-gop" + ], + "clientRoles": { + "argocd": [ + "uma_protection" + ] + }, + "notBefore": 0, + "groups": [] + }, + { + "id": "93ee9325-38b7-43c4-9f1b-59cb2abbeac9", + "username": "service-account-grafana", + "emailVerified": false, + "enabled": true, + "createdTimestamp": 1779198324666, + "totp": false, + "serviceAccountClientId": "grafana", + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "default-roles-gop" + ], + "clientRoles": { + "grafana": [ + "uma_protection" + ] + }, + "notBefore": 0, + "groups": [] + }, + { + "id": "1c07b963-5f1a-4112-951b-aec6f118c057", + "username": "service-account-jenkins", + "emailVerified": false, + "enabled": true, + "createdTimestamp": 1779345420158, + "totp": false, + "serviceAccountClientId": "jenkins", + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "default-roles-gop" + ], + "clientRoles": { + "jenkins": [ + "uma_protection" + ] + }, + "notBefore": 0, + "groups": [] + }, + { + "id": "338d50c7-835b-4c7a-8adf-b376e9378899", + "username": "service-account-vault", + "emailVerified": false, + "enabled": true, + "createdTimestamp": 1779435557530, + "totp": false, + "serviceAccountClientId": "vault", + "disableableCredentialTypes": [], + "requiredActions": [], + "realmRoles": [ + "default-roles-gop" + ], + "clientRoles": { + "vault": [ + "uma_protection" + ] + }, + "notBefore": 0, + "groups": [] + } + ], + "scopeMappings": [ + { + "clientScope": "offline_access", + "roles": [ + "offline_access" + ] + } + ], + "clientScopeMappings": { + "account": [ + { + "client": "account-console", + "roles": [ + "manage-account", + "view-groups" + ] + } + ] + }, + "clients": [ + { + "id": "dad19cc6-6518-469c-bea3-de245650e4b5", + "clientId": "account", + "name": "${client_account}", + "rootUrl": "${authBaseUrl}", + "baseUrl": "/realms/gop/account/", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "redirectUris": [ + "/realms/gop/account/*" + ], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": false, + "publicClient": true, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "post.logout.redirect.uris": "+" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": false, + "nodeReRegistrationTimeout": 0, + "defaultClientScopes": [ + "web-origins", + "acr", + "profile", + "roles", + "basic", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "4eb83a7d-55e5-4816-8c60-e70e6894ccc0", + "clientId": "account-console", + "name": "${client_account-console}", + "rootUrl": "${authBaseUrl}", + "baseUrl": "/realms/gop/account/", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "redirectUris": [ + "/realms/gop/account/*" + ], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": false, + "publicClient": true, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "post.logout.redirect.uris": "+", + "pkce.code.challenge.method": "S256" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": false, + "nodeReRegistrationTimeout": 0, + "protocolMappers": [ + { + "id": "afae8344-80a3-47aa-a7f1-02789aff90b0", + "name": "audience resolve", + "protocol": "openid-connect", + "protocolMapper": "oidc-audience-resolve-mapper", + "consentRequired": false, + "config": {} + } + ], + "defaultClientScopes": [ + "web-origins", + "acr", + "profile", + "roles", + "basic", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "b68640a3-b6ff-444f-a437-4b14a47d1a5a", + "clientId": "admin-cli", + "name": "${client_admin-cli}", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "redirectUris": [], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": false, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": false, + "publicClient": true, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "client.use.lightweight.access.token.enabled": "true" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": 0, + "defaultClientScopes": [ + "web-origins", + "acr", + "profile", + "roles", + "basic", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "b552400b-2da1-4c2e-ac5d-d66023bd762f", + "clientId": "argocd", + "name": "", + "description": "", + "rootUrl": "http://argocd.localhost", + "adminUrl": "http://argocd.localhost", + "baseUrl": "http://argocd.localhost", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": true, + "clientAuthenticatorType": "client-secret", + "secret": "Cq3U2Dvx6WR5ep2ZLH8rZXNKFNcIX5Zx", + "redirectUris": [ + "http://argocd.localhost", + "http://argocd.localhost/auth/callback", + "http://argocd.localhost/auth/callback/", + "http://argocd.localhost/", + "https://argocd.localhost", + "https://argocd.localhost/", + "https://argocd.localhost/*" + ], + "webOrigins": [ + "http://argocd.localhost", + "https://argocd.localhost", + "*" + ], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": true, + "authorizationServicesEnabled": false, + "publicClient": false, + "frontchannelLogout": true, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "oidc.ciba.grant.enabled": "false", + "client.secret.creation.time": "1779192964", + "backchannel.logout.session.required": "true", + "standard.token.exchange.enabled": "false", + "frontchannel.logout.session.required": "true", + "display.on.consent.screen": "false", + "oauth2.device.authorization.grant.enabled": "false", + "use.jwks.url": "false", + "backchannel.logout.revoke.offline.tokens": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "defaultClientScopes": [ + "web-origins", + "service_account", + "acr", + "profile", + "roles", + "basic", + "groups", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "5f194498-254e-4759-881f-f900588630b1", + "clientId": "broker", + "name": "${client_broker}", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "redirectUris": [], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": true, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": false, + "publicClient": false, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "realm_client": "true" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": false, + "nodeReRegistrationTimeout": 0, + "defaultClientScopes": [ + "web-origins", + "acr", + "profile", + "roles", + "basic", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "2ed0d5a0-6efa-43c5-bc9c-07f51f153459", + "clientId": "grafana", + "name": "", + "description": "", + "rootUrl": "http://grafana.localhost", + "adminUrl": "http://grafana.localhost", + "baseUrl": "http://grafana.localhost", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "secret": "46uALR2HeqLPtJxzkkn6tK6FVC9vMmRc", + "redirectUris": [ + "http://grafana.localhost", + "*" + ], + "webOrigins": [ + "*" + ], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": true, + "authorizationServicesEnabled": false, + "publicClient": false, + "frontchannelLogout": true, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "oidc.ciba.grant.enabled": "false", + "client.secret.creation.time": "1779198324", + "backchannel.logout.session.required": "true", + "standard.token.exchange.enabled": "false", + "oauth2.device.authorization.grant.enabled": "false", + "backchannel.logout.revoke.offline.tokens": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "defaultClientScopes": [ + "web-origins", + "service_account", + "acr", + "profile", + "roles", + "basic", + "groups", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "662df4d9-5df0-4954-8c8f-7cb954b257e3", + "clientId": "jenkins", + "name": "", + "description": "", + "rootUrl": "http://jenkins.localhost", + "adminUrl": "http://jenkins.localhost", + "baseUrl": "http://jenkins.localhost", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "secret": "mtuIbUdggI2ZSy7jSsQW0WkwK0CwMbvO", + "redirectUris": [ + "http://jenkins.localhost", + "*" + ], + "webOrigins": [ + "*" + ], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": true, + "authorizationServicesEnabled": false, + "publicClient": false, + "frontchannelLogout": true, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "oidc.ciba.grant.enabled": "false", + "client.secret.creation.time": "1779345420", + "backchannel.logout.session.required": "true", + "standard.token.exchange.enabled": "false", + "oauth2.device.authorization.grant.enabled": "false", + "backchannel.logout.revoke.offline.tokens": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "defaultClientScopes": [ + "web-origins", + "service_account", + "acr", + "profile", + "roles", + "basic", + "groups", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "10ab9905-3584-4dd1-b436-f1dfcb434968", + "clientId": "realm-management", + "name": "${client_realm-management}", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "redirectUris": [], + "webOrigins": [], + "notBefore": 0, + "bearerOnly": true, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": false, + "publicClient": false, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "realm_client": "true" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": false, + "nodeReRegistrationTimeout": 0, + "defaultClientScopes": [ + "web-origins", + "acr", + "profile", + "roles", + "basic", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "1c23e40a-a0bf-4ee9-97b6-2b4371ea9938", + "clientId": "security-admin-console", + "name": "${client_security-admin-console}", + "rootUrl": "${authAdminUrl}", + "baseUrl": "/admin/gop/console/", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "redirectUris": [ + "/admin/gop/console/*" + ], + "webOrigins": [ + "+" + ], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": false, + "publicClient": true, + "frontchannelLogout": false, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "client.use.lightweight.access.token.enabled": "true", + "post.logout.redirect.uris": "+", + "pkce.code.challenge.method": "S256" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": 0, + "protocolMappers": [ + { + "id": "1b286acf-b21d-4b50-b54d-5dc92c511e1e", + "name": "locale", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "locale", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "locale", + "jsonType.label": "String" + } + } + ], + "defaultClientScopes": [ + "web-origins", + "acr", + "profile", + "roles", + "basic", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + }, + { + "id": "544c7f5d-9cc9-4a1e-949a-895aa7ab147e", + "clientId": "vault", + "name": "", + "description": "", + "rootUrl": "http://vault.localhost", + "adminUrl": "http://vault.localhost", + "baseUrl": "http://vault.localhost", + "surrogateAuthRequired": false, + "enabled": true, + "alwaysDisplayInConsole": false, + "clientAuthenticatorType": "client-secret", + "secret": "XySg7UyAzVkcaU4Visqfe9EChDvARYA1", + "redirectUris": [ + "*", + "http://vault.localhost" + ], + "webOrigins": [ + "*" + ], + "notBefore": 0, + "bearerOnly": false, + "consentRequired": false, + "standardFlowEnabled": true, + "implicitFlowEnabled": false, + "directAccessGrantsEnabled": false, + "serviceAccountsEnabled": true, + "authorizationServicesEnabled": false, + "publicClient": false, + "frontchannelLogout": true, + "protocol": "openid-connect", + "attributes": { + "realm_client": "false", + "oidc.ciba.grant.enabled": "false", + "client.secret.creation.time": "1779435557", + "backchannel.logout.session.required": "true", + "standard.token.exchange.enabled": "false", + "post.logout.redirect.uris": "http://vault.localhost", + "oauth2.device.authorization.grant.enabled": "false", + "backchannel.logout.revoke.offline.tokens": "false" + }, + "authenticationFlowBindingOverrides": {}, + "fullScopeAllowed": true, + "nodeReRegistrationTimeout": -1, + "defaultClientScopes": [ + "web-origins", + "service_account", + "acr", + "profile", + "roles", + "basic", + "groups", + "email" + ], + "optionalClientScopes": [ + "address", + "phone", + "organization", + "offline_access", + "microprofile-jwt" + ] + } + ], + "clientScopes": [ + { + "id": "51501633-93f2-4b93-9252-6fd8c765c9f4", + "name": "groups", + "description": "OIDC group membership claim for local GOP testing", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "a186dd76-a626-4ef2-92d7-2516b3bb7d97", + "name": "groups", + "protocol": "openid-connect", + "protocolMapper": "oidc-group-membership-mapper", + "consentRequired": false, + "config": { + "full.path": "false", + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "groups" + } + } + ] + }, + { + "id": "abf4e1b3-548b-45c5-af89-1dd6c6c04175", + "name": "address", + "description": "OpenID Connect built-in scope: address", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "consent.screen.text": "${addressScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "15ad7769-3868-478b-a041-d1e7e4a0e989", + "name": "address", + "protocol": "openid-connect", + "protocolMapper": "oidc-address-mapper", + "consentRequired": false, + "config": { + "user.attribute.formatted": "formatted", + "user.attribute.country": "country", + "introspection.token.claim": "true", + "user.attribute.postal_code": "postal_code", + "userinfo.token.claim": "true", + "user.attribute.street": "street", + "id.token.claim": "true", + "user.attribute.region": "region", + "access.token.claim": "true", + "user.attribute.locality": "locality" + } + } + ] + }, + { + "id": "b5789687-60d3-4e93-be9a-2f7d4c47a136", + "name": "basic", + "description": "OpenID Connect scope for add all basic claims to the token", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "false", + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "a38853cb-f988-482b-80ef-beb578c3d0ce", + "name": "auth_time", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "AUTH_TIME", + "id.token.claim": "true", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "auth_time", + "jsonType.label": "long" + } + }, + { + "id": "868d40ac-b36a-47aa-a335-2a5780f530d2", + "name": "sub", + "protocol": "openid-connect", + "protocolMapper": "oidc-sub-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "access.token.claim": "true" + } + } + ] + }, + { + "id": "0f784e67-0eac-401d-b464-3a69a0fd23b0", + "name": "role_list", + "description": "SAML role list", + "protocol": "saml", + "attributes": { + "consent.screen.text": "${samlRoleListScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "52e876dc-b184-4730-aa47-8a17f0a372b3", + "name": "role list", + "protocol": "saml", + "protocolMapper": "saml-role-list-mapper", + "consentRequired": false, + "config": { + "single": "false", + "attribute.nameformat": "Basic", + "attribute.name": "Role" + } + } + ] + }, + { + "id": "fe9f1383-a95c-40e8-b12b-27e85125c0f3", + "name": "saml_organization", + "description": "Organization Membership", + "protocol": "saml", + "attributes": { + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "4461e73f-f271-48d4-99cd-273ddbd409df", + "name": "organization", + "protocol": "saml", + "protocolMapper": "saml-organization-membership-mapper", + "consentRequired": false, + "config": {} + } + ] + }, + { + "id": "2298db2b-a9e2-4520-9c3d-9c66fac1769d", + "name": "acr", + "description": "OpenID Connect scope for add acr (authentication context class reference) to the token", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "false", + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "48b765df-ddd4-49f8-9e1f-e46df3703768", + "name": "acr loa level", + "protocol": "openid-connect", + "protocolMapper": "oidc-acr-mapper", + "consentRequired": false, + "config": { + "id.token.claim": "true", + "access.token.claim": "true", + "introspection.token.claim": "true" + } + } + ] + }, + { + "id": "94b6a95d-0098-4ba8-855e-318af02c0e27", + "name": "service_account", + "description": "Specific scope for a client enabled for service accounts", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "false", + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "3bef728c-4de3-4e21-a2ba-a2491f035efb", + "name": "Client IP Address", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientAddress", + "id.token.claim": "true", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientAddress", + "jsonType.label": "String" + } + }, + { + "id": "8ab78d5b-cb9f-45fd-b78e-7b5ff321c12f", + "name": "Client ID", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "client_id", + "id.token.claim": "true", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "client_id", + "jsonType.label": "String" + } + }, + { + "id": "8e82240f-7d94-4098-89f2-6caa262bdfee", + "name": "Client Host", + "protocol": "openid-connect", + "protocolMapper": "oidc-usersessionmodel-note-mapper", + "consentRequired": false, + "config": { + "user.session.note": "clientHost", + "id.token.claim": "true", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "clientHost", + "jsonType.label": "String" + } + } + ] + }, + { + "id": "04eb5797-b709-42af-9802-c5713fd00a22", + "name": "email", + "description": "OpenID Connect built-in scope: email", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "consent.screen.text": "${emailScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "76e30970-4699-4b44-9178-0d6c4bb288ff", + "name": "email", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "email", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "email", + "jsonType.label": "String" + } + }, + { + "id": "dc8ac8d1-6b27-4b85-8935-29b99b8e5355", + "name": "email verified", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-property-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "emailVerified", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "email_verified", + "jsonType.label": "boolean" + } + } + ] + }, + { + "id": "14aba137-e032-42cc-b24a-a4c41f22558d", + "name": "phone", + "description": "OpenID Connect built-in scope: phone", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "consent.screen.text": "${phoneScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "93e24a41-2fa6-411f-87fe-dbe8dd8744c4", + "name": "phone number", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "phoneNumber", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "phone_number", + "jsonType.label": "String" + } + }, + { + "id": "5964f360-757c-4374-8bbd-ab4ac53ed6d3", + "name": "phone number verified", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "phoneNumberVerified", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "phone_number_verified", + "jsonType.label": "boolean" + } + } + ] + }, + { + "id": "6a583cbd-9ab9-446a-85c4-16aaee35ad9d", + "name": "web-origins", + "description": "OpenID Connect scope for add allowed web origins to the access token", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "false", + "consent.screen.text": "", + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "299cd61c-6d37-49ec-89ac-2b57e17b39a3", + "name": "allowed web origins", + "protocol": "openid-connect", + "protocolMapper": "oidc-allowed-origins-mapper", + "consentRequired": false, + "config": { + "access.token.claim": "true", + "introspection.token.claim": "true" + } + } + ] + }, + { + "id": "a7a391d9-fe62-4511-b738-19fd123e5044", + "name": "profile", + "description": "OpenID Connect built-in scope: profile", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "consent.screen.text": "${profileScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "60aa362f-38bd-4e57-8b32-825e71ade1a6", + "name": "zoneinfo", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "zoneinfo", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "zoneinfo", + "jsonType.label": "String" + } + }, + { + "id": "b9dce93d-1058-45df-bf6a-0f45e6ede1fc", + "name": "picture", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "picture", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "picture", + "jsonType.label": "String" + } + }, + { + "id": "06ba2577-e87b-47e6-8d74-f24088e3b0d5", + "name": "full name", + "protocol": "openid-connect", + "protocolMapper": "oidc-full-name-mapper", + "consentRequired": false, + "config": { + "id.token.claim": "true", + "introspection.token.claim": "true", + "access.token.claim": "true", + "userinfo.token.claim": "true" + } + }, + { + "id": "bb18b613-e446-4944-a335-4b3ee6eaf4c7", + "name": "nickname", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "nickname", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "nickname", + "jsonType.label": "String" + } + }, + { + "id": "8cc9cace-1d77-4c58-bdc9-2d3717cca5ca", + "name": "profile", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "profile", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "profile", + "jsonType.label": "String" + } + }, + { + "id": "40c80eb8-9701-44bf-ac73-ddfdb12dcf3b", + "name": "middle name", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "middleName", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "middle_name", + "jsonType.label": "String" + } + }, + { + "id": "6b7f0889-f2b3-41f3-a5b4-991df13a3619", + "name": "website", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "website", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "website", + "jsonType.label": "String" + } + }, + { + "id": "602d8a79-d9c4-437c-a9b2-4c4e0e8d07f7", + "name": "family name", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "lastName", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "family_name", + "jsonType.label": "String" + } + }, + { + "id": "f4f53bce-af1d-4332-befd-56d954a21594", + "name": "username", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "username", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "preferred_username", + "jsonType.label": "String" + } + }, + { + "id": "98e7a039-6f26-4b56-908a-690476fbf5d3", + "name": "updated at", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "updatedAt", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "updated_at", + "jsonType.label": "long" + } + }, + { + "id": "6f470356-0b1f-4951-affe-bd0b2db3012f", + "name": "gender", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "gender", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "gender", + "jsonType.label": "String" + } + }, + { + "id": "9492ecfa-f375-4ec8-accb-a2c0a1dbf57b", + "name": "locale", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "locale", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "locale", + "jsonType.label": "String" + } + }, + { + "id": "6602cd93-80c8-489d-93a7-16a294e2b9fc", + "name": "given name", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "firstName", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "given_name", + "jsonType.label": "String" + } + }, + { + "id": "34acb2eb-3a98-444e-a605-ce08d5e1b1d5", + "name": "birthdate", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "birthdate", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "birthdate", + "jsonType.label": "String" + } + } + ] + }, + { + "id": "44da44c9-43f2-4e4c-a9a7-634488168d5d", + "name": "microprofile-jwt", + "description": "Microprofile - JWT built-in scope", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + }, + "protocolMappers": [ + { + "id": "b6f90a9e-538d-486a-b164-40765ec2c34c", + "name": "upn", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-attribute-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "userinfo.token.claim": "true", + "user.attribute": "username", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "upn", + "jsonType.label": "String" + } + }, + { + "id": "020796ad-936e-4926-982d-96e9bdc4273c", + "name": "groups", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-realm-role-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "multivalued": "true", + "user.attribute": "foo", + "id.token.claim": "true", + "access.token.claim": "true", + "claim.name": "groups", + "jsonType.label": "String" + } + } + ] + }, + { + "id": "486515a1-fdef-4fe5-a264-35d92fe1b0ac", + "name": "organization", + "description": "Additional claims about the organization a subject belongs to", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "consent.screen.text": "${organizationScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "798d847d-10e5-4875-9fef-d9bf619e318e", + "name": "organization", + "protocol": "openid-connect", + "protocolMapper": "oidc-organization-membership-mapper", + "consentRequired": false, + "config": { + "id.token.claim": "true", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "organization", + "jsonType.label": "String", + "multivalued": "true" + } + } + ] + }, + { + "id": "53e382de-8f38-4296-b91e-77ccf4689cf6", + "name": "offline_access", + "description": "OpenID Connect built-in scope: offline_access", + "protocol": "openid-connect", + "attributes": { + "consent.screen.text": "${offlineAccessScopeConsentText}", + "display.on.consent.screen": "true" + } + }, + { + "id": "8610d461-9dd0-4626-9fba-88462433abe8", + "name": "roles", + "description": "OpenID Connect scope for add user roles to the access token", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "false", + "consent.screen.text": "${rolesScopeConsentText}", + "display.on.consent.screen": "true" + }, + "protocolMappers": [ + { + "id": "0eca599c-f155-4998-9b70-45d6d22f76ea", + "name": "realm roles", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-realm-role-mapper", + "consentRequired": false, + "config": { + "user.attribute": "foo", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "realm_access.roles", + "jsonType.label": "String", + "multivalued": "true" + } + }, + { + "id": "493dadef-6d74-4775-aec5-4c18283e5b8d", + "name": "audience resolve", + "protocol": "openid-connect", + "protocolMapper": "oidc-audience-resolve-mapper", + "consentRequired": false, + "config": { + "introspection.token.claim": "true", + "access.token.claim": "true" + } + }, + { + "id": "d01f22ab-bbf7-4c1f-bfa6-c3fa7e3480af", + "name": "client roles", + "protocol": "openid-connect", + "protocolMapper": "oidc-usermodel-client-role-mapper", + "consentRequired": false, + "config": { + "user.attribute": "foo", + "introspection.token.claim": "true", + "access.token.claim": "true", + "claim.name": "resource_access.${client_id}.roles", + "jsonType.label": "String", + "multivalued": "true" + } + } + ] + } + ], + "defaultDefaultClientScopes": [ + "role_list", + "saml_organization", + "profile", + "email", + "roles", + "web-origins", + "acr", + "basic" + ], + "defaultOptionalClientScopes": [ + "offline_access", + "address", + "phone", + "microprofile-jwt", + "organization" + ], + "browserSecurityHeaders": { + "contentSecurityPolicyReportOnly": "", + "xContentTypeOptions": "nosniff", + "referrerPolicy": "no-referrer", + "xRobotsTag": "none", + "xFrameOptions": "SAMEORIGIN", + "contentSecurityPolicy": "frame-src 'self'; frame-ancestors 'self'; object-src 'none';", + "strictTransportSecurity": "max-age=31536000; includeSubDomains" + }, + "smtpServer": {}, + "eventsEnabled": false, + "eventsListeners": [ + "jboss-logging" + ], + "enabledEventTypes": [], + "adminEventsEnabled": false, + "adminEventsDetailsEnabled": false, + "identityProviders": [], + "identityProviderMappers": [], + "components": { + "org.keycloak.services.clientregistration.policy.ClientRegistrationPolicy": [ + { + "id": "b748fb6c-e439-4e86-840c-887159ef14ed", + "name": "Allowed Protocol Mapper Types", + "providerId": "allowed-protocol-mappers", + "subType": "authenticated", + "subComponents": {}, + "config": { + "allowed-protocol-mapper-types": [ + "oidc-usermodel-attribute-mapper", + "oidc-address-mapper", + "oidc-full-name-mapper", + "saml-user-attribute-mapper", + "saml-role-list-mapper", + "oidc-sha256-pairwise-sub-mapper", + "oidc-usermodel-property-mapper", + "saml-user-property-mapper" + ] + } + }, + { + "id": "5b944467-cdfd-408a-96fc-16eefa7c5247", + "name": "Allowed Client Scopes", + "providerId": "allowed-client-templates", + "subType": "anonymous", + "subComponents": {}, + "config": { + "allow-default-scopes": [ + "true" + ] + } + }, + { + "id": "89b2bd2d-00c2-4aaa-b3d9-2eb52ec6045a", + "name": "Allowed Protocol Mapper Types", + "providerId": "allowed-protocol-mappers", + "subType": "anonymous", + "subComponents": {}, + "config": { + "allowed-protocol-mapper-types": [ + "oidc-usermodel-property-mapper", + "saml-role-list-mapper", + "saml-user-property-mapper", + "oidc-usermodel-attribute-mapper", + "oidc-full-name-mapper", + "oidc-sha256-pairwise-sub-mapper", + "saml-user-attribute-mapper", + "oidc-address-mapper" + ] + } + }, + { + "id": "c29ce7a0-0826-4a50-95d1-ca8d7602e3d5", + "name": "Allowed Client Scopes", + "providerId": "allowed-client-templates", + "subType": "authenticated", + "subComponents": {}, + "config": { + "allow-default-scopes": [ + "true" + ] + } + }, + { + "id": "3d25335d-171a-4154-92d6-b0090cb2dcb5", + "name": "Trusted Hosts", + "providerId": "trusted-hosts", + "subType": "anonymous", + "subComponents": {}, + "config": { + "host-sending-registration-request-must-match": [ + "true" + ], + "client-uris-must-match": [ + "true" + ] + } + }, + { + "id": "34176546-da43-48cd-aaa6-e8cf12652ec8", + "name": "Max Clients Limit", + "providerId": "max-clients", + "subType": "anonymous", + "subComponents": {}, + "config": { + "max-clients": [ + "200" + ] + } + }, + { + "id": "44bae2cb-dbea-4d93-b4fc-e6d10e57f69d", + "name": "Consent Required", + "providerId": "consent-required", + "subType": "anonymous", + "subComponents": {}, + "config": {} + }, + { + "id": "5c453875-7aca-47d6-9499-253c029b09ef", + "name": "Full Scope Disabled", + "providerId": "scope", + "subType": "anonymous", + "subComponents": {}, + "config": {} + } + ], + "org.keycloak.keys.KeyProvider": [ + { + "id": "172042bb-05fb-4970-b56c-2b6cd8b0f84a", + "name": "rsa-generated", + "providerId": "rsa-generated", + "subComponents": {}, + "config": { + "priority": [ + "100" + ] + } + }, + { + "id": "af6b81cb-bccd-45fb-b2f4-388f803f8697", + "name": "rsa-enc-generated", + "providerId": "rsa-enc-generated", + "subComponents": {}, + "config": { + "priority": [ + "100" + ], + "algorithm": [ + "RSA-OAEP" + ] + } + }, + { + "id": "2ac1963b-3bd0-49e9-bbd0-052ed634055a", + "name": "hmac-generated-hs512", + "providerId": "hmac-generated", + "subComponents": {}, + "config": { + "priority": [ + "100" + ], + "algorithm": [ + "HS512" + ] + } + }, + { + "id": "4cfc4415-06d4-454b-9a7f-4f0bcda2994d", + "name": "aes-generated", + "providerId": "aes-generated", + "subComponents": {}, + "config": { + "priority": [ + "100" + ] + } + } + ] + }, + "internationalizationEnabled": false, + "authenticationFlows": [ + { + "id": "066625a5-ac33-4457-a574-4f81276c543f", + "alias": "Account verification options", + "description": "Method with which to verity the existing account", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "idp-email-verification", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "ALTERNATIVE", + "priority": 20, + "autheticatorFlow": true, + "flowAlias": "Verify Existing Account by Re-authentication", + "userSetupAllowed": false + } + ] + }, + { + "id": "43ad1bcf-dd48-48c4-9e12-c51ceafb665e", + "alias": "Browser - Conditional 2FA", + "description": "Flow to determine if any 2FA is required for the authentication", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "auth-otp-form", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "webauthn-authenticator", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 30, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "auth-recovery-authn-code-form", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 40, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "6bd4957c-fd5c-4a89-85cc-43ff296c65d7", + "alias": "Browser - Conditional Organization", + "description": "Flow to determine if the organization identity-first login is to be used", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "organization", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "93f6e2b7-e8e5-465e-90c8-bc3b9cfd3b64", + "alias": "Direct Grant - Conditional OTP", + "description": "Flow to determine if the OTP is required for the authentication", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "direct-grant-validate-otp", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "dca88a2c-b345-4af2-8b12-1c6a1f141bac", + "alias": "First Broker Login - Conditional Organization", + "description": "Flow to determine if the authenticator that adds organization members is to be used", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "idp-add-organization-member", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "ba0d4f5b-6655-4793-b1f7-e0da7a6648d3", + "alias": "First broker login - Conditional 2FA", + "description": "Flow to determine if any 2FA is required for the authentication", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "auth-otp-form", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "webauthn-authenticator", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 30, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "auth-recovery-authn-code-form", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 40, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "f1fc6ecf-9bb8-4a30-beb0-73ff5d1cb219", + "alias": "Handle Existing Account", + "description": "Handle what to do if there is existing account with same email/username like authenticated identity provider", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "idp-confirm-link", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": true, + "flowAlias": "Account verification options", + "userSetupAllowed": false + } + ] + }, + { + "id": "89a68bcc-d833-4c38-aa81-a4db1b8f5852", + "alias": "Organization", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticatorFlow": true, + "requirement": "CONDITIONAL", + "priority": 10, + "autheticatorFlow": true, + "flowAlias": "Browser - Conditional Organization", + "userSetupAllowed": false + } + ] + }, + { + "id": "d20b86de-0c22-4cea-b6b3-d5e8471b5131", + "alias": "Reset - Conditional OTP", + "description": "Flow to determine if the OTP should be reset or not. Set to REQUIRED to force.", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "conditional-user-configured", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "reset-otp", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "d07f02b4-7f41-429b-b305-2a7e62f7b4b7", + "alias": "User creation or linking", + "description": "Flow for the existing/non-existing user alternatives", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticatorConfig": "create unique user config", + "authenticator": "idp-create-user-if-unique", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "ALTERNATIVE", + "priority": 20, + "autheticatorFlow": true, + "flowAlias": "Handle Existing Account", + "userSetupAllowed": false + } + ] + }, + { + "id": "efaad87d-981c-4f5f-bcc3-82fc8f451421", + "alias": "Verify Existing Account by Re-authentication", + "description": "Reauthentication of existing account", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "idp-username-password-form", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "CONDITIONAL", + "priority": 20, + "autheticatorFlow": true, + "flowAlias": "First broker login - Conditional 2FA", + "userSetupAllowed": false + } + ] + }, + { + "id": "cd4cc485-12f6-4ce8-93c4-83014624e973", + "alias": "browser", + "description": "Browser based authentication", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "auth-cookie", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "auth-spnego", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "identity-provider-redirector", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 25, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "ALTERNATIVE", + "priority": 26, + "autheticatorFlow": true, + "flowAlias": "Organization", + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "ALTERNATIVE", + "priority": 30, + "autheticatorFlow": true, + "flowAlias": "forms", + "userSetupAllowed": false + } + ] + }, + { + "id": "3a7a7017-c67a-40ae-9fe8-040f7f9a65bd", + "alias": "clients", + "description": "Base authentication for clients", + "providerId": "client-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "client-secret", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "client-jwt", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "client-secret-jwt", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 30, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "client-x509", + "authenticatorFlow": false, + "requirement": "ALTERNATIVE", + "priority": 40, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "5bede317-1225-4e8b-b4e8-bc55bdd744cd", + "alias": "direct grant", + "description": "OpenID Connect Resource Owner Grant", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "direct-grant-validate-username", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "direct-grant-validate-password", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "CONDITIONAL", + "priority": 30, + "autheticatorFlow": true, + "flowAlias": "Direct Grant - Conditional OTP", + "userSetupAllowed": false + } + ] + }, + { + "id": "b9295a56-84e5-420c-8943-b9ac4cf60691", + "alias": "docker auth", + "description": "Used by Docker clients to authenticate against the IDP", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "docker-http-basic-authenticator", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "e34e0960-bce9-46b3-8ea0-e37928f76fab", + "alias": "first broker login", + "description": "Actions taken after first broker login with identity provider account, which is not yet linked to any Keycloak account", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticatorConfig": "review profile config", + "authenticator": "idp-review-profile", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": true, + "flowAlias": "User creation or linking", + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "CONDITIONAL", + "priority": 50, + "autheticatorFlow": true, + "flowAlias": "First Broker Login - Conditional Organization", + "userSetupAllowed": false + } + ] + }, + { + "id": "8a4cba72-950d-46a8-9f72-4ff284ebae74", + "alias": "forms", + "description": "Username, password, otp and other auth forms.", + "providerId": "basic-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "auth-username-password-form", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "CONDITIONAL", + "priority": 20, + "autheticatorFlow": true, + "flowAlias": "Browser - Conditional 2FA", + "userSetupAllowed": false + } + ] + }, + { + "id": "94b4c16e-1e1e-43e4-86e3-02159ebe1590", + "alias": "registration", + "description": "Registration flow", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "registration-page-form", + "authenticatorFlow": true, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": true, + "flowAlias": "registration form", + "userSetupAllowed": false + } + ] + }, + { + "id": "85485e81-b0f7-4e83-8a91-fb946217290c", + "alias": "registration form", + "description": "Registration form", + "providerId": "form-flow", + "topLevel": false, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "registration-user-creation", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "registration-password-action", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 50, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "registration-recaptcha-action", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 60, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "registration-terms-and-conditions", + "authenticatorFlow": false, + "requirement": "DISABLED", + "priority": 70, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + }, + { + "id": "f6b55a3b-8047-4310-84d7-89c051e1417c", + "alias": "reset credentials", + "description": "Reset credentials for a user if they forgot their password or something", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "reset-credentials-choose-user", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "reset-credential-email", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 20, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticator": "reset-password", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 30, + "autheticatorFlow": false, + "userSetupAllowed": false + }, + { + "authenticatorFlow": true, + "requirement": "CONDITIONAL", + "priority": 40, + "autheticatorFlow": true, + "flowAlias": "Reset - Conditional OTP", + "userSetupAllowed": false + } + ] + }, + { + "id": "6883dd85-f047-439f-8a6a-c9e691018ad4", + "alias": "saml ecp", + "description": "SAML ECP Profile Authentication Flow", + "providerId": "basic-flow", + "topLevel": true, + "builtIn": true, + "authenticationExecutions": [ + { + "authenticator": "http-basic-authenticator", + "authenticatorFlow": false, + "requirement": "REQUIRED", + "priority": 10, + "autheticatorFlow": false, + "userSetupAllowed": false + } + ] + } + ], + "authenticatorConfig": [ + { + "id": "1defbd92-0417-419b-b6cc-92dac59970fc", + "alias": "create unique user config", + "config": { + "require.password.update.after.registration": "false" + } + }, + { + "id": "ead3c3f1-d1f6-4d38-85f0-0cf57727f4b2", + "alias": "review profile config", + "config": { + "update.profile.on.first.login": "missing" + } + } + ], + "requiredActions": [ + { + "alias": "CONFIGURE_TOTP", + "name": "Configure OTP", + "providerId": "CONFIGURE_TOTP", + "enabled": true, + "defaultAction": false, + "priority": 10, + "config": {} + }, + { + "alias": "TERMS_AND_CONDITIONS", + "name": "Terms and Conditions", + "providerId": "TERMS_AND_CONDITIONS", + "enabled": false, + "defaultAction": false, + "priority": 20, + "config": {} + }, + { + "alias": "UPDATE_PASSWORD", + "name": "Update Password", + "providerId": "UPDATE_PASSWORD", + "enabled": true, + "defaultAction": false, + "priority": 30, + "config": {} + }, + { + "alias": "UPDATE_PROFILE", + "name": "Update Profile", + "providerId": "UPDATE_PROFILE", + "enabled": true, + "defaultAction": false, + "priority": 40, + "config": {} + }, + { + "alias": "VERIFY_EMAIL", + "name": "Verify Email", + "providerId": "VERIFY_EMAIL", + "enabled": true, + "defaultAction": false, + "priority": 50, + "config": {} + }, + { + "alias": "delete_account", + "name": "Delete Account", + "providerId": "delete_account", + "enabled": false, + "defaultAction": false, + "priority": 60, + "config": {} + }, + { + "alias": "webauthn-register", + "name": "Webauthn Register", + "providerId": "webauthn-register", + "enabled": true, + "defaultAction": false, + "priority": 70, + "config": {} + }, + { + "alias": "webauthn-register-passwordless", + "name": "Webauthn Register Passwordless", + "providerId": "webauthn-register-passwordless", + "enabled": true, + "defaultAction": false, + "priority": 80, + "config": {} + }, + { + "alias": "VERIFY_PROFILE", + "name": "Verify Profile", + "providerId": "VERIFY_PROFILE", + "enabled": true, + "defaultAction": false, + "priority": 90, + "config": {} + }, + { + "alias": "delete_credential", + "name": "Delete Credential", + "providerId": "delete_credential", + "enabled": true, + "defaultAction": false, + "priority": 100, + "config": {} + }, + { + "alias": "idp_link", + "name": "Linking Identity Provider", + "providerId": "idp_link", + "enabled": true, + "defaultAction": false, + "priority": 110, + "config": {} + }, + { + "alias": "CONFIGURE_RECOVERY_AUTHN_CODES", + "name": "Recovery Authentication Codes", + "providerId": "CONFIGURE_RECOVERY_AUTHN_CODES", + "enabled": true, + "defaultAction": false, + "priority": 120, + "config": {} + }, + { + "alias": "update_user_locale", + "name": "Update User Locale", + "providerId": "update_user_locale", + "enabled": true, + "defaultAction": false, + "priority": 1000, + "config": {} + } + ], + "browserFlow": "browser", + "registrationFlow": "registration", + "directGrantFlow": "direct grant", + "resetCredentialsFlow": "reset credentials", + "clientAuthenticationFlow": "clients", + "dockerAuthenticationFlow": "docker auth", + "firstBrokerLoginFlow": "first broker login", + "attributes": { + "cibaBackchannelTokenDeliveryMode": "poll", + "cibaExpiresIn": "120", + "cibaAuthRequestedUserHint": "login_hint", + "oauth2DeviceCodeLifespan": "600", + "oauth2DevicePollingInterval": "5", + "parRequestUriLifespan": "60", + "cibaInterval": "5", + "realmReusableOtpCode": "false" + }, + "keycloakVersion": "26.3.3", + "userManagedAccessAllowed": false, + "clientPolicies": { + "policies": [] + } +} diff --git a/pom.xml b/pom.xml index 7b9d6968f..b05dfbbc1 100644 --- a/pom.xml +++ b/pom.xml @@ -1,40 +1,41 @@ - 4.0.0 com.cloudogu gitops-playground-cli 0.1 - ${packaging} + jar io.micronaut.platform micronaut-parent - - 4.10.12 + 4.10.17 - jar - 17 - 17 + + 25 + 25 com.cloudogu.gitops.cli.GitopsPlaygroundCliMain + 5.7.0.6970 - + - ${git.tags} (${git.commit.id.abbrev}, + + ${git.tags} (${git.commit.id.abbrev}, ${maven.build.timestamp})\n${project.licenses[0].comments}\n${project.licenses[0].name}\n${project.licenses[0].url} yyyy-MM-dd HH:mm - 2.2.0 5.3.2 3.0.0 - 5.0.5 5.0.0 - 26.0.0 - 7.7.0 + 7.8.0 3.13.2 + 4.2.17.Final @@ -64,14 +65,55 @@ org.eclipse.jetty jetty-bom - 12.1.8 + 12.1.13 pom import org.eclipse.jetty.ee10 jetty-ee10-bom - 12.1.8 + 12.1.13 + pom + import + + + + + tools.jackson.core + jackson-databind + 3.2.1 + compile + + + + + tools.jackson.core + jackson-core + 3.2.0 + compile + + + + + com.fasterxml.jackson.core + jackson-core + 2.22.0 + compile + + + + + com.fasterxml.jackson.core + jackson-databind + 2.22.1 + compile + + + + io.netty + netty-bom + ${netty.version} pom import @@ -81,49 +123,21 @@ io.micronaut - micronaut-inject-groovy + micronaut-aop io.micronaut.reactor micronaut-reactor - 3.9.1 runtime - - org.apache.groovy - groovy-all - ${groovy.version} - pom - - - org.testng - testng - - - org.apache.groovy - groovy-testng - - - org.apache.groovy - groovy-test - - - - - - - org.apache.groovy - groovy-yaml - - - + com.fasterxml.jackson.dataformat jackson-dataformat-yaml - 2.21.2 + 2.22.0 @@ -131,11 +145,6 @@ picocli - - io.micronaut.groovy - micronaut-runtime-groovy - - io.micronaut.picocli micronaut-picocli @@ -167,18 +176,20 @@ 2.22.0 - + org.springframework.security spring-security-crypto - 7.0.5 + 7.1.1 org.eclipse.jgit org.eclipse.jgit - - 7.6.0.202603022253-r + + 7.7.1.202607240634-r @@ -196,7 +207,7 @@ com.squareup.okhttp3 logging-interceptor - 5.3.2 + ${okhttpVersion} @@ -206,7 +217,7 @@ - + com.squareup.retrofit2 converter-jackson ${retrofitVersion} @@ -214,7 +225,7 @@ org.wiremock - wiremock-jetty12 + wiremock-standalone ${wiremock.version} test @@ -230,7 +241,7 @@ org.springframework spring-jcl - 6.2.18 + 6.2.19 test @@ -238,7 +249,7 @@ org.freemarker freemarker - 2.3.34 + 2.3.35 @@ -249,20 +260,13 @@ test - + io.fabric8 - openshift-client + kubernetes-client ${kubernetes.fabric8.java.version} - - io.kubernetes - client-java - ${kubernetes.client.java.version} - test - - io.micronaut micronaut-http-client @@ -285,12 +289,25 @@ org.mockito mockito-core test + + + + net.bytebuddy + byte-buddy-agent + + org.mockito mockito-junit-jupiter test + + + net.bytebuddy + byte-buddy-agent + + @@ -300,20 +317,6 @@ test - - org.apache.groovy - groovy-test - test - - - - - com.github.stefanbirkner - system-lambda - 1.2.1 - test - - javax.xml.bind jaxb-api @@ -324,7 +327,7 @@ org.gitlab4j gitlab4j-api - 6.2.0 + 6.3.0 @@ -342,7 +345,7 @@ com.networknt json-schema-validator - 3.0.2 + 3.0.7 org.apache.commons @@ -351,28 +354,13 @@ - - com.cloudogu.versionName - processor - ${versionNameVersion} - - provided - - - - org.awaitility - awaitility - 4.3.0 - test - - io.github.classgraph classgraph - 4.8.184 + 4.8.194 - + jakarta.xml.bind jakarta.xml.bind-api @@ -381,20 +369,50 @@ org.glassfish.jaxb jaxb-runtime - 4.0.7 + 4.0.9 runtime + + org.projectlombok + lombok + 1.18.48 + provided + + + + + src/main/resources + + + src/main/version + com/cloudogu/gitops/cli + true + + + + + com.diffplug.spotless + spotless-maven-plugin + 3.10.1 + + + + + true + 4 + + + + maven-surefire-plugin - 3.5.5 + 3.6.0 - - @{argLine} --add-opens java.base/java.util=ALL-UNNAMED ROOT_LOG_LEVEL @@ -405,13 +423,12 @@ OFF - org.jacoco jacoco-maven-plugin - 0.8.14 + 0.8.15 @@ -462,7 +479,7 @@ maven-jar-plugin - 3.5.0 + 3.5.1 @@ -470,75 +487,35 @@ micronaut-maven-plugin - - org.codehaus.gmavenplus - gmavenplus-plugin - 4.3.1 - - - - execute - - - - - compiler.groovy - 17 - - - - - - org.codehaus.mojo - truezip-maven-plugin - 1.2 - - - remove-a-file-in-sub-archive - - remove - - package - - - - ${project.build.directory}/${project.artifactId}-${project.version}.jar/META-INF/ - - - *.DSA - *.RSA - *.SF - - - - - - - org.apache.maven.plugins maven-compiler-plugin - 3.15.0 + 3.16.0 - + + + + org.projectlombok + lombok + 1.18.48 + - com.cloudogu.versionName - processor - ${versionNameVersion} + io.micronaut + micronaut-inject-java + ${micronaut.version} - - - -AversionName=${versionName} - io.github.git-commit-id git-commit-id-maven-plugin - 10.0.0 + 10.0.1 get-the-git-infos @@ -558,7 +535,7 @@ maven-dependency-plugin - 3.10.0 + 3.11.0 @@ -580,32 +557,13 @@ - org.codehaus.mojo - properties-maven-plugin - 1.3.0 - - - - set-system-properties - - - - - groovy.target.directory - ${project.build.directory}/classes - - - groovy.parameters - true - - - - - + org.sonarsource.scanner.maven + sonar-maven-plugin + ${sonar-maven-plugin.version} - \ No newline at end of file + diff --git a/renovate.json b/renovate.json index 7d25d2997..0ca99aa82 100644 --- a/renovate.json +++ b/renovate.json @@ -1,32 +1,48 @@ { - "$schema": "https://docs.renovatebot.com/renovate-schema.json", - "baseBranchPatterns": [ - "develop" - ], - "assignees": [ - "avetgit", - "DerDaehne", - "mdroll", - "ThomasMichael1811" - ], - "dependencyDashboard": true, - "minimumReleaseAge": "7 days", - "extends": [ - ":automergeMinor", - ":combinePatchMinorReleases", - ":configMigration", - ":automergeDigest" - ], - "packageRules": [ - { - "matchManagers": [ - "jenkins" - ], - "automerge": false, - "registryUrls": [ - "http://updates.jenkins-ci.org/stable/update-center.json" - ], - "groupName": "Jenkins Updates" - } - ] + "$schema": "https://docs.renovatebot.com/renovate-schema.json", + "customManagers": [ + { + "customType": "regex", + "managerFilePatterns": [ + "/^src/main/java/com/cloudogu/gitops/config/Config\\.java$/", + "/^src/main/java/com/cloudogu/gitops/config/scm/ScmTenantSchema\\.java$/" + ], + "matchStrings": [ + "// renovate: depName=(?[^\\s]+) registryUrl=(?[^\\s]+)\\s+.*setVersion\\(\"(?[^\"]+)\"\\);" + ], + "datasourceTemplate": "helm" + }, + { + "customType": "regex", + "managerFilePatterns": [ + "/^Dockerfile$/" + ], + "matchStrings": [ + "# renovate: depName=(?[^\\s]+) datasource=(?[^\\s]+)\\s+.*ARG HELM_VERSION=(?[^\\s]+)" + ] + } + ], + "baseBranchPatterns": [ + "develop" + ], + "dependencyDashboard": true, + "minimumReleaseAge": "7 days", + "extends": [ + ":automergeMinor", + ":combinePatchMinorReleases", + ":configMigration", + ":automergeDigest" + ], + "packageRules": [ + { + "matchManagers": [ + "jenkins" + ], + "automerge": false, + "registryUrls": [ + "http://updates.jenkins-ci.org/stable/update-center.json" + ], + "groupName": "Jenkins Updates" + } + ] } diff --git a/scripts/dev/gop-secrets-values.yaml b/scripts/dev/gop-secrets-values.yaml new file mode 100644 index 000000000..2dae5adea --- /dev/null +++ b/scripts/dev/gop-secrets-values.yaml @@ -0,0 +1,79 @@ +# $schema: https://raw.githubusercontent.com/cloudogu/gitops-playground/main/docs/configuration.schema.json +application: + "yes": true + baseUrl: http://localhost + credentials: + secretName: argocd-credentials + secretNamespace: gop-job +scm: + scmManager: + credentials: + secretName: scm-tenant-credentials + secretNamespace: gop-job +features: + certManager: + active: true + argocd: + active: true + operator: false + ingress: + active: true + monitoring: + active: true + secrets: + vault: + mode: "dev" +jenkins: + active: true + credentials: + secretName: jenkins-credentials + secretNamespace: gop-job +registry: + active: true + credentials: + secretName: registry-credentials + secretNamespace: gop-job +content: + repos: + - url: https://github.com/cloudogu/gitops-build-lib + target: 3rd-party-dependencies/gitops-build-lib + overwriteMode: RESET + - url: https://github.com/cloudogu/ces-build-lib + target: 3rd-party-dependencies/ces-build-lib + overwriteMode: RESET + - url: https://github.com/cloudogu/spring-boot-helm-chart + target: 3rd-party-dependencies/spring-boot-helm-chart + overwriteMode: RESET + - url: https://github.com/cloudogu/spring-petclinic + target: argocd/petclinic-plain + ref: feature/gitops_ready + targetRef: main + overwriteMode: UPGRADE + createJenkinsJob: true + - url: https://github.com/cloudogu/spring-petclinic + target: argocd/petclinic-helm + ref: feature/gitops_ready + targetRef: main + overwriteMode: UPGRADE + createJenkinsJob: true + - url: https://github.com/cloudogu/gitops-examples + path: example-apps-via-content-loader/ + ref: main + templating: true + type: FOLDER_BASED + overwriteMode: UPGRADE + + namespaces: + - ${config.application.namePrefix}example-apps-production + - ${config.application.namePrefix}example-apps-staging + variables: + petclinic: + baseDomain: "petclinic" + images: + kubectl: "alpine/kubectl:latest" + helm: "ghcr.io/cloudogu/helm:latest" + kubeval: "ghcr.io/cloudogu/helm:latest" + helmKubeval: "ghcr.io/cloudogu/helm:latest" + yamllint: "cytopia/yamllint:1.25-0.7" + petclinic: "eclipse-temurin:17-jre" + maven: "" diff --git a/scripts/dev/gop-secrets.yaml b/scripts/dev/gop-secrets.yaml new file mode 100644 index 000000000..a6f5f220b --- /dev/null +++ b/scripts/dev/gop-secrets.yaml @@ -0,0 +1,39 @@ +apiVersion: v1 +kind: Secret +metadata: + name: jenkins-credentials + namespace: gop-job +type: Opaque +stringData: + username: admin + password: this_is_for_your_ads +--- +apiVersion: v1 +kind: Secret +metadata: + name: argocd-credentials + namespace: gop-job +type: Opaque +stringData: + username: admin + password: who_can_read_this +--- +apiVersion: v1 +kind: Secret +metadata: + name: registry-credentials + namespace: gop-job +type: Opaque +stringData: + username: myregistry + password: mypassword +--- +apiVersion: v1 +kind: Secret +metadata: + name: scm-tenant-credentials + namespace: gop-job +type: Opaque +stringData: + username: miniadmin + password: this_is_my_password diff --git a/scripts/dev/gop_airgapped_config.yaml b/scripts/dev/gop_airgapped_config.yaml index dde97c5af..21ad51ad5 100644 --- a/scripts/dev/gop_airgapped_config.yaml +++ b/scripts/dev/gop_airgapped_config.yaml @@ -1,6 +1,11 @@ application: baseUrl: "http://localhost" insecure: true +jenkins: + jenkinsImage: "k3d-agreg:5000/library/jenkins-helm:latest" +scm: + scmManager: + scmmImage: "k3d-agreg:5000/library/scm-manager:latest" features: argocd: active: true diff --git a/scripts/dev/gop_airgapped_config.yaml.tpl b/scripts/dev/gop_airgapped_config.yaml.tpl index bed0ee906..f554c2901 100644 --- a/scripts/dev/gop_airgapped_config.yaml.tpl +++ b/scripts/dev/gop_airgapped_config.yaml.tpl @@ -1,6 +1,11 @@ application: baseUrl: "http://localhost" insecure: true +jenkins: + jenkinsImage: "
/library/jenkins-helm:latest" +scm: + scmManager: + scmmImage: "
/library/scm-manager:latest" features: argocd: active: true diff --git a/scripts/dev/mirror_images_to_registry.sh b/scripts/dev/mirror_images_to_registry.sh index b203b9693..a4c407739 100755 --- a/scripts/dev/mirror_images_to_registry.sh +++ b/scripts/dev/mirror_images_to_registry.sh @@ -10,8 +10,8 @@ HARBOR=$2 REGISTRY_DOCKER_BASE_URL=docker:$(echo $REGISTRY_BASE_URL | cut -d: -f2-) ESO_IMAGE="docker://ghcr.io/external-secrets/external-secrets:v0.9.16" -VAULT_IMAGE="docker://hashicorp/vault:1.14.0" -TRAEFIK_IMAGE="docker://docker.io/library/traefik:v3.3.3" +VAULT_IMAGE="docker://hashicorp/vault:2.0.4" +TRAEFIK_IMAGE="docker://docker.io/library/traefik:v3.6.15" PROMETHEUS_IMAGE="docker://quay.io/prometheus/prometheus:v3.8.0" PROMETHEUS_OPERATOR_IMAGE="docker://quay.io/prometheus-operator/prometheus-operator:v0.87.1" @@ -19,12 +19,17 @@ PROMETHEUS_OPERATOR_CONFIG_RELOADER="docker://quay.io/prometheus-operator/promet GRAFANA_IMAGE="docker://docker.io/grafana/grafana:12.3.0" K8S_SIDECAR="docker://quay.io/kiwigrid/k8s-sidecar:2.1.2" +JENKINS_IMAGE_TAG="5.9.18" +SCM_MANAGER_IMAGE_TAG="3.11.6" +JENKINS_IMAGE="docker://ghcr.io/cloudogu/jenkins-helm:${JENKINS_IMAGE_TAG}" +SCM_MANAGER_IMAGE="docker://docker.io/scmmanager/scm-manager:${SCM_MANAGER_IMAGE_TAG}" + CERT_MANAGER_CONTROLLER="docker://quay.io/jetstack/cert-manager-controller:v1.16.1" CERT_MANAGER_CA_INJECTOR="docker://quay.io/jetstack/cert-manager-cainjector:v1.16.1" CERT_MANAGER_WEBHOOK="docker://quay.io/jetstack/cert-manager-webhook:v1.16.1" KUBECTL_IMAGE="docker://alpine/kubectl:latest" -TEMURIN_IMAGE="docker://eclipse-temurin:17-jre-alpine" +TEMURIN_IMAGE="docker://eclipse-temurin:17-jre" HELM_IMAGE="docker://ghcr.io/cloudogu/helm:latest" MVN_IMAGE="docker://maven:3-eclipse-temurin-17-alpine" YAMLLINT_IMAGE="docker://cytopia/yamllint:latest" @@ -38,6 +43,7 @@ if [[ -n $HARBOR ]]; then operations=("Proxy" "Registry") readOnlyUser='RegistryRead' + declare -A projectIds for operation in "${operations[@]}"; do @@ -45,19 +51,26 @@ if [[ -n $HARBOR ]]; then lower_operation=$(echo "$operation" | tr '[:upper:]' '[:lower:]') echo "creating project ${lower_operation}" - projectId=$(curl -is --fail "$REGISTRY_BASE_URL/api/v2.0/projects" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"project_name\":\"$lower_operation\",\"metadata\":{\"public\":\"false\"},\"storage_limit\":-1,\"registry_id\":null}" | grep -i 'Location:' | awk '{print $2}' | awk -F '/' '{print $NF}' | tr -d '[:space:]') + projectResponse=$(curl -is "$REGISTRY_BASE_URL/api/v2.0/projects" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"project_name\":\"$lower_operation\",\"metadata\":{\"public\":\"false\"},\"storage_limit\":-1,\"registry_id\":null}" || true) + projectId=$(echo "$projectResponse" | grep -i 'Location:' | awk '{print $2}' | awk -F '/' '{print $NF}' | tr -d '[:space:]' || true) + + if [[ -z "$projectId" ]]; then + projectId=$(curl -s --fail "$REGISTRY_BASE_URL/api/v2.0/projects/${lower_operation}" -u admin:Harbor12345 | sed -n 's/.*"project_id":\([0-9]*\).*/\1/p') + fi + + projectIds[$lower_operation]=$projectId echo creating user ${operation} with PW ${operation}12345 - curl -s --fail "$REGISTRY_BASE_URL/api/v2.0/users" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"username\":\"$operation\",\"email\":\"$operation@example.com\",\"realname\":\"$operation example\",\"password\":\"${operation}12345\",\"comment\":null}" + curl -s "$REGISTRY_BASE_URL/api/v2.0/users" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"username\":\"$operation\",\"email\":\"$operation@example.com\",\"realname\":\"$operation example\",\"password\":\"${operation}12345\",\"comment\":null}" || true echo "Adding member ${operation} to project ${lower_operation}; ID=${projectId}" - curl --fail "$REGISTRY_BASE_URL/api/v2.0/projects/${projectId}/members" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"role_id\":4,\"member_user\":{\"username\":\"$operation\"}}" + curl "$REGISTRY_BASE_URL/api/v2.0/projects/${projectId}/members" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"role_id\":4,\"member_user\":{\"username\":\"$operation\"}}" || true done echo "creating user ${readOnlyUser} with PW ${readOnlyUser}12345" - curl -s --fail "$REGISTRY_BASE_URL/api/v2.0/users" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"username\":\"$readOnlyUser\",\"email\":\"$readOnlyUser@example.com\",\"realname\":\"$readOnlyUser example\",\"password\":\"${readOnlyUser}12345\",\"comment\":null}" - echo "Adding member ${readOnlyUser} to project proxy; ID=${projectId}" - curl --fail "$REGISTRY_BASE_URL/api/v2.0/projects/${projectId}/members" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"role_id\":5,\"member_user\":{\"username\":\"${readOnlyUser}\"}}" + curl -s "$REGISTRY_BASE_URL/api/v2.0/users" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"username\":\"$readOnlyUser\",\"email\":\"$readOnlyUser@example.com\",\"realname\":\"$readOnlyUser example\",\"password\":\"${readOnlyUser}12345\",\"comment\":null}" || true + echo "Adding member ${readOnlyUser} to project proxy; ID=${projectIds[proxy]}" + curl "$REGISTRY_BASE_URL/api/v2.0/projects/${projectIds[proxy]}/members" -X POST -u admin:Harbor12345 -H 'Content-Type: application/json' --data-raw "{\"role_id\":5,\"member_user\":{\"username\":\"${readOnlyUser}\"}}" || true # sleep 5 seconds just to make sure the registry is ready sleep 5 @@ -65,7 +78,7 @@ if [[ -n $HARBOR ]]; then # When updating the container image versions note that all images of a chart are listed at artifact hub on the right hand side under "Containers Images" skopeo copy $ESO_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/external-secrets skopeo copy $VAULT_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/vault - skopeo copy $TRAEFIK_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/traefik:v3.3.3 + skopeo copy $TRAEFIK_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/traefik:v3.6.15 # Monitoring skopeo copy $PROMETHEUS_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/prometheus @@ -74,6 +87,10 @@ if [[ -n $HARBOR ]]; then skopeo copy $GRAFANA_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/grafana skopeo copy $K8S_SIDECAR --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/k8s-sidecar + # Core tools + skopeo copy $JENKINS_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/jenkins-helm + skopeo copy $SCM_MANAGER_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/scm-manager + # Cert Manager images skopeo copy $CERT_MANAGER_CONTROLLER --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/cert-manager-controller skopeo copy $CERT_MANAGER_CA_INJECTOR --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/cert-manager-cainjector @@ -81,7 +98,7 @@ if [[ -n $HARBOR ]]; then # Needed for the builds to work with proxy-registry skopeo copy $KUBECTL_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/alpine/kubectl:latest - skopeo copy $TEMURIN_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/eclipse-temurin:17-jre-alpine + skopeo copy $TEMURIN_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/eclipse-temurin:17-jre skopeo copy $HELM_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/helm:latest skopeo copy $MVN_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/maven:3-eclipse-temurin-17-alpine skopeo copy $YAMLLINT_IMAGE --dest-creds Proxy:Proxy12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/proxy/yamllint:latest @@ -92,7 +109,7 @@ fi # When updating the container image versions note that all images of a chart are listed at artifact hub on the right hand side under "Containers Images" skopeo copy $ESO_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/external-secrets skopeo copy $VAULT_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/vault -skopeo copy $TRAEFIK_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/traefik:v3.3.3 +skopeo copy $TRAEFIK_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/traefik:v3.6.15 # Monitoring skopeo copy $PROMETHEUS_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/prometheus @@ -101,6 +118,10 @@ skopeo copy $PROMETHEUS_OPERATOR_CONFIG_RELOADER --dest-creds admin:Harbor12345 skopeo copy $GRAFANA_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/grafana skopeo copy $K8S_SIDECAR --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/k8s-sidecar +# Core tools +skopeo copy $JENKINS_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/jenkins-helm +skopeo copy $SCM_MANAGER_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/scm-manager + # Cert Manager images skopeo copy $CERT_MANAGER_CONTROLLER --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/cert-manager-controller skopeo copy $CERT_MANAGER_CA_INJECTOR --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/cert-manager-cainjector @@ -108,7 +129,7 @@ skopeo copy $CERT_MANAGER_WEBHOOK --dest-creds admin:Harbor12345 --dest-tls-veri # Needed for the builds to work with proxy-registry skopeo copy $KUBECTL_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/alpine/kubectl:latest -skopeo copy $TEMURIN_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/eclipse-temurin:17-jre-alpine +skopeo copy $TEMURIN_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/eclipse-temurin:17-jre skopeo copy $HELM_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/helm:latest skopeo copy $MVN_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/maven:3-eclipse-temurin-17-alpine skopeo copy $YAMLLINT_IMAGE --dest-creds admin:Harbor12345 --dest-tls-verify=false $REGISTRY_DOCKER_BASE_URL/library/yamllint:latest diff --git a/scripts/dev/prepare_two_registries.sh b/scripts/dev/prepare_two_registries.sh index 21666059b..0908985f4 100755 --- a/scripts/dev/prepare_two_registries.sh +++ b/scripts/dev/prepare_two_registries.sh @@ -45,14 +45,14 @@ content: - \${config.application.namePrefix}example-apps-staging variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "localhost:30000/proxy/kubectl:latest" helm: "localhost:30000/proxy/helm:latest" kubeval: "localhost:30000/proxy/helm:latest" helmKubeval: "localhost:30000/proxy/helm:latest" yamllint: "localhost:30000/proxy/cytopia/yamllint:latest" - petclinic: "localhost:30000/proxy/eclipse-temurin:17-jre-alpine" + petclinic: "localhost:30000/proxy/eclipse-temurin:17-jre" maven: "localhost:30000/proxy/maven:3-eclipse-temurin-17-alpine" registry: internalPort: 30000 @@ -68,6 +68,10 @@ registry: createImagePullSecrets: true jenkins: active: true + jenkinsImage: "localhost:30000/proxy/jenkins-helm:latest" +scm: + scmManager: + scmmImage: "localhost:30000/proxy/scm-manager:latest" application: baseUrl: "http://localhost" insecure: true diff --git a/scripts/downloadHelmCharts.sh b/scripts/downloadHelmCharts.sh index 3cb297f09..f9bef53ce 100755 --- a/scripts/downloadHelmCharts.sh +++ b/scripts/downloadHelmCharts.sh @@ -2,8 +2,8 @@ #execute from root folder set -o errexit -o nounset -o pipefail charts=( 'kube-prometheus-stack' 'external-secrets' 'vault' 'traefik' 'cert-manager' 'jenkins' 'scm-manager') -CONFIG="${1:-src/main/groovy/com/cloudogu/gitops/config/Config.groovy}" -SCM_MANAGER_CONFIG="${2:-src/main/groovy/com/cloudogu/gitops/config/scm/ScmTenantSchema.groovy}" +CONFIG="${1:-src/main/java/com/cloudogu/gitops/config/Config.java}" +SCM_MANAGER_CONFIG="${2:-src/main/java/com/cloudogu/gitops/config/scm/ScmTenantSchema.java}" CONFIG_FILES=("${CONFIG}") if [[ "${CONFIG}" != "${SCM_MANAGER_CONFIG}" ]]; then @@ -18,7 +18,7 @@ function extractChartProperty() { local chartDetails="$1" local property="$2" - echo "$chartDetails" | sed -nE "s/.*${property}[[:space:]]*:[[:space:]]*'([^']+)'.*/\1/p" | head -n1 + echo "$chartDetails" | sed -nE "s/.*set${property}\(\"([^\"]+)\"\).*/\1/p" | head -n1 } for chart in "${charts[@]}"; do @@ -28,7 +28,7 @@ for chart in "${charts[@]}"; do if [[ ! -f "${configFile}" ]]; then continue fi - chartDetails=$(grep -m1 -EA5 "chart[[:space:]]*:[[:space:]]*'${chart}'" "${configFile}" || true) + chartDetails=$(grep -m1 -EA5 "setChart\(\"${chart}\"\)" "${configFile}" || true) if [[ -n "$chartDetails" ]]; then chartConfig="${configFile}" break @@ -39,9 +39,9 @@ for chart in "${charts[@]}"; do echo "Did not find chart details for chart $chart in files: ${CONFIG_FILES[*]}" >&2 exit 1 fi - repo=$(extractChartProperty "$chartDetails" "repoURL") - chart=$(extractChartProperty "$chartDetails" "chart") - version=$(extractChartProperty "$chartDetails" "version") + repo=$(extractChartProperty "$chartDetails" "RepoURL") + chart=$(extractChartProperty "$chartDetails" "Chart") + version=$(extractChartProperty "$chartDetails" "Version") if [[ -z "$repo" || -z "$chart" || -z "$version" ]]; then echo "Could not extract chart details from ${chartConfig}: repoURL='${repo}', chart='${chart}', version='${version}'" >&2 @@ -54,7 +54,7 @@ for chart in "${charts[@]}"; do helm repo add "$chart" "$repo" --repository-config="${tmpRepoFile}" helm pull --untar --untardir ./charts "$chart/$chart" --version "$version" --repository-config="${tmpRepoFile}" # Note that keeping charts as tgx would need only 1/10 of storage - # But untaring them in groovy would need additional libraries. + # But untarring them in application code would need additional libraries. # As layers of the image are compressed anyway, we'll do the untar process here, pragmatically # Do a simple verification diff --git a/scripts/init-cluster.sh b/scripts/init-cluster.sh index 6421573d3..2fc4a5d0d 100755 --- a/scripts/init-cluster.sh +++ b/scripts/init-cluster.sh @@ -2,9 +2,10 @@ # See https://github.com/rancher/k3d/releases # This variable is also read in Jenkinsfile -K3D_VERSION=5.8.3 +K3D_VERSION=5.9.0 # When updating please also adapt in Dockerfile, vars.tf and Config.groovy -K8S_VERSION=1.35.3 +K8S_VERSION=1.36.4 + K3S_VERSION="rancher/k3s:v${K8S_VERSION}-k3s1" set -o errexit @@ -13,12 +14,12 @@ set -o pipefail function main() { readParameters "$@" - + [[ $TRACE == true ]] && set -x; - + # Install k3d if necessary if ! command -v k3d >/dev/null 2>&1; then - echo The GitOps playground uses k3d, which is not found on the PATH. + echo The GitOps playground uses k3d, which is not found on the PATH. installK3d else ACTUAL_K3D_VERSION="$(k3d --version | grep k3d | sed 's/k3d version v\(.*\)/\1/')" @@ -62,13 +63,28 @@ function createCluster() { fi fi + # Ensure loopback alias exists when using a non-default loopback IP (e.g. 127.0.0.2) + # This avoids port conflicts with CRC/OpenShift which binds *:80/*:443 on the default 127.0.0.1 + if [[ "${BIND_INGRESS_HOST}" != "127.0.0.1" && "${BIND_INGRESS_HOST}" =~ ^127\. ]]; then + if ! ifconfig lo0 | grep -q "${BIND_INGRESS_HOST}"; then + echo "Adding loopback alias ${BIND_INGRESS_HOST} to lo0 (requires sudo)..." + sudo ifconfig lo0 alias "${BIND_INGRESS_HOST}" + else + echo "Loopback alias ${BIND_INGRESS_HOST} already configured." + fi + fi + HOST_PORT_RANGE='8010-65535' + DOCKER_SOCK_PATH="/var/run/docker.sock" + if [[ -S "$HOME/.orbstack/run/docker.sock" ]]; then + DOCKER_SOCK_PATH="$HOME/.orbstack/run/docker.sock" + fi K3D_ARGS=( # Allow services to bind to portBindings < 30000 > 32xxx # This makes is easier to match for example --bind-registry-port=0 on ci or use lower ports for development "--k3s-arg=--kube-apiserver-arg=service-node-port-range=${HOST_PORT_RANGE}@server:*" # Used by Jenkins Agents pods - '-v /var/run/docker.sock:/var/run/docker.sock@server:*' + "-v ${DOCKER_SOCK_PATH}:/var/run/docker.sock@server:*" # Allows for finding out the GID of the docker group in order to allow the Jenkins agents pod to access docker socket '-v /etc/group:/etc/group@server:*' # Persists the cache of Jenkins agents pods for faster builds @@ -78,7 +94,7 @@ function createCluster() { # Disable traefik (we roll our own ingress-controller) '--k3s-arg=--disable=traefik@server:*' ) - + REGISTRIES="" if [[ -n "$DOCKER_IO_REGISTRY_MIRROR" ]]; then REGISTRIES=$(cat <> Help screen" echo echo " | --cluster-name=STRING >> Set your preferred cluster name to install k3d. Defaults to 'gitops-playground'." - + echo " | --bind-localhost=BOOLEAN >> Bind the k3d container to host network. Exposes all k8s nodePorts to localhost. Defaults to false." - echo " | --bind-ingress-host=STRING >> Bind the ingress controller to this local ip. Defaults to 127.0.0.1." + echo " | --bind-ingress-host=STRING >> Bind the ingress controller to this local ip. Defaults to 127.0.0.2 (avoids port conflict with CRC/OpenShift)." echo " | --bind-ingress-port=INT >> Bind the ingress controller to this port. Defaults to 80. Set to - to disable." echo " | --bind-registry-port=INT >> Specify a custom port for the container registry to bind to localhost port. Only use this when port 30000 is blocked and --bind-localhost=true. Defaults to 30000 (default used by the playground)." echo " | --bind-ports=STRING >> A comma separated list of additional port bindings like 443:443,9090:9090. Ignored when --bind-localhost." - + echo " | --docker-io-registry-mirror=STRING >> the hostname of a registry that mirrors DockerHub. Useful when encountering rate limits" echo echo " -x | --trace >> Debug + Show each command executed (set -x)" @@ -213,7 +229,7 @@ function confirm() { # shellcheck disable=SC2145 # - the line break between args is intended here! printf "%s\n" "${@:-Are you sure? [y/N]} " - + read -r response case "$response" in [yY][eE][sS] | [yY]) @@ -233,7 +249,7 @@ get_longopt_value(){ # or # 2='--expected' # 3='value' - + # check $2 has the form --longopt=value VALUE=$(echo "$2" | sed -e 's/^[^=]*=//') if [ -z "$VALUE" ]; then @@ -249,6 +265,7 @@ get_longopt_value(){ readParameters() { CLUSTER_NAME=gitops-playground BIND_LOCALHOST=false + # Use 127.0.0.2 to avoid port conflict with CRC/OpenShift which binds *:80/*:443 BIND_INGRESS_HOST="127.0.0.1" BIND_INGRESS_PORT="80" BIND_INGRESS_HTTPS_PORT="443" @@ -264,7 +281,7 @@ readParameters() { -x | --trace ) TRACE=true; shift ;; --bind-localhost) BIND_LOCALHOST=true; shift ;; --cluster-name*) CLUSTER_NAME=$(get_longopt_value "--cluster-name" "$@") - # Allow passing portBindings with and without '=' + # Allow passing portBindings with and without '=' if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; --bind-ingress-port*) BIND_INGRESS_PORT=$(get_longopt_value "--bind-ingress-port" "$@") if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; @@ -272,11 +289,11 @@ readParameters() { if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; --bind-ingress-https-port*) BIND_INGRESS_HTTPS_PORT=$(get_longopt_value "--bind-ingress-https-port" "$@") if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; - --bind-registry-port*) BIND_REGISTRY_PORT=$(get_longopt_value "--bind-registry-port" "$@") + --bind-registry-port*) BIND_REGISTRY_PORT=$(get_longopt_value "--bind-registry-port" "$@") if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; - --bind-ports*) BIND_PORTS=$(get_longopt_value "--bind-ports" "$@"); + --bind-ports*) BIND_PORTS=$(get_longopt_value "--bind-ports" "$@"); if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; - --docker-io-registry-mirror*) DOCKER_IO_REGISTRY_MIRROR=$(get_longopt_value "--docker-io-registry-mirror" "$@"); + --docker-io-registry-mirror*) DOCKER_IO_REGISTRY_MIRROR=$(get_longopt_value "--docker-io-registry-mirror" "$@"); if [[ "$1" == *"="* ]]; then shift; else shift 2; fi ;; --) shift; break ;; *) break ;; diff --git a/scripts/jenkins/plugins/install-plugins.sh b/scripts/jenkins/plugins/install-plugins.sh index b79f66538..d51b4cd87 100755 --- a/scripts/jenkins/plugins/install-plugins.sh +++ b/scripts/jenkins/plugins/install-plugins.sh @@ -285,9 +285,7 @@ main() { fi echo "Cleaning up locks" - find "$REF_DIR" -regex ".*.lock" | while read -r filepath; do - rm -r "$filepath" - done + find "$REF_DIR" -type d -name "*.lock" -prune -exec rm -rf {} + } diff --git a/scripts/jenkins/plugins/plugins.txt b/scripts/jenkins/plugins/plugins.txt index 3b04b8838..e402f68e4 100644 --- a/scripts/jenkins/plugins/plugins.txt +++ b/scripts/jenkins/plugins/plugins.txt @@ -1,85 +1,89 @@ antisamy-markup-formatter:173.v680e3a_b_69ff3 apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83 -asm-api:9.9.1-189.vb_5ef2964da_91 +asm-api:9.10.1-216.va_9256d3b_844b_ authentication-tokens:1.144.v5ff4a_5ec5c33 -bootstrap5-api:5.3.8-1024.v127320880c60 +bootstrap5-api:5.3.8-1048.va_c299057e35c bouncycastle-api:2.30.1.84-291.v9f17b_21896e2 branch-api:2.1280.v0d4e5b_b_460ef -caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81 -checks-api:402.vca_263b_f200e3 -cloudbees-folder:6.1100.ve9eed61d16c4 -commons-compress-api:1.28.0-3 +caffeine-api:3.2.4-208.v7e2da_a_7db_82b_ +checks-api:415.vf022234a_931d +cloudbees-folder:6.1106.v3a_d9a_6d2465e +commons-compress-api:1.28.0-87.v48a_8104cb_b_25 commons-lang3-api:3.20.0-109.ve43756e2d2b_4 commons-text-api:1.15.0-218.va_61573470393 -configuration-as-code:2077.v41f1011a_5110 -credentials:1502.v5c95e620ddfe -credentials-binding:719.v80e905ef14eb_ +configuration-as-code:2117.vc05a_0b_e6b_f4e +credentials:1511.v2e3cb_0008ef0 +credentials-binding:728.v902a_273b_8947 display-url-api:2.217.va_6b_de84cc74b_ -docker-commons:472.vee120e23d3a_c -docker-workflow:634.vedc7242b_eda_7 -durable-task:664.v2b_e7a_dfff66c -echarts-api:6.0.0-1281.vd3d21a_1ca_cb_4 +docker-commons:477.v289085a_b_6896 +docker-workflow:653.v2f2c08eff0ec +durable-task:686.v80ff80875b_82 +echarts-api:6.0.0-1287.vfd24c22a_3d00 eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_ -font-awesome-api:7.2.0-983.v3f63c34eddb_9 +font-awesome-api:7.2.0-990.vf220b_2a_496f9 git:5.10.1 -git-client:6.6.0 +git-client:6.6.1 gson-api:2.14.0-201.v8eefe5515533 instance-identity:203.v15e81a_1b_7a_38 ionicons-api:94.vcc3065403257 -jackson-annotations2-api:2.21-7.v4777a_f3a_a_d47 -jackson2-api:2.21.2-436.v29efdb_7418ff -jackson3-api:3.1.2-73.v3e5485d8b_148 -jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3 +jackson-annotations2-api:2.22-19.v10a_a_582ea_26e +jackson2-api:2.22.1-443.vc91f592333c4 +jackson3-api:3.2.2-96.v599957900a_1a_ jakarta-activation-api:2.1.4-1 jakarta-mail-api:2.1.5-1 +jakarta-xml-bind-api:4.0.9-19.v2b_a_5b_44d9a_1c javax-activation-api:1.2.0-8 jaxb:2.3.9-143.v5979df3304e6 -joda-time-api:2.14.1-187.vdf2def02b_8a_1 -jquery3-api:3.7.1-682.vfa_cdce169929 -json-api:20250517-173.v596efb_962a_31 -junit:1403.vd9d1413fd205 -kubernetes:4423.vb_59f230b_ce53 +joda-time-api:2.14.3-200.v65623733c99f +jquery3-api:3.7.1-687.v68d468e40b_30 +json-api:20260814-226.v20f9685d642c +json-path-api:3.0.0-218.vcd4dd1355de2 +junit:1421.v99cb_b_2577709 +kubernetes:4547.v52f3080db_8cd kubernetes-client-api:7.3.1-256.v788a_0b_787114 kubernetes-credentials:207.v492f58828b_ed mailer:534.v1b_36f5864073 +matrix-auth:3.3 metrics:4.2.37-494.v06f9a_939d33a_ -mina-sshd-api-common:2.16.0-184.v1e0e8b_e8e813 -mina-sshd-api-core:2.16.0-184.v1e0e8b_e8e813 +mina-sshd-api-common:2.19.0-192.v2b_a_7b_2c1dc71 +mina-sshd-api-core:2.19.0-192.v2b_a_7b_2c1dc71 +nimbus-jose-jwt-api:10.9.1-4.v58e0353801ec +oic-auth:4.718.ve731df6ca_88a_ okhttp-api:5.3.2-200.vedb_720a_cf1f8 -pipeline-build-step:584.vdb_a_2cc3a_d07a_ +pipeline-build-step:599.v4b_67ea_11b_152 pipeline-graph-analysis:254.v0f63a_a_447dca_ -pipeline-groovy-lib:797.v90ea_a_9b_e45a_0 -pipeline-input-step:551.vdff487c5998c +pipeline-groovy-lib:798.v5cc688825312 +pipeline-input-step:560.v56198a_642157 pipeline-milestone-step:152.v6e22b_8cfc66c -pipeline-model-api:2.2277.v00573e73ddf1 -pipeline-model-definition:2.2277.v00573e73ddf1 -pipeline-model-extensions:2.2277.v00573e73ddf1 +pipeline-model-api:2.2293.v6e7193cec599 +pipeline-model-definition:2.2293.v6e7193cec599 +pipeline-model-extensions:2.2293.v6e7193cec599 pipeline-rest-api:2.41 pipeline-stage-step:345.va_96187909426 -pipeline-stage-tags-metadata:2.2277.v00573e73ddf1 +pipeline-stage-tags-metadata:2.2293.v6e7193cec599 pipeline-stage-view:2.41 -pipeline-utility-steps:2.20.0 +pipeline-utility-steps:3.810.va_7672d206740 plain-credentials:199.v9f8e1f741799 -plugin-util-api:7.1330.v47b_46ee2047a_ -prism-api:1.30.0-720.v1eb_7496954b_3 -prometheus:852.v317db_5d17a_b_0 +plugin-util-api:7.1341.v039f146993d9 +prism-api:1.30.0-741.v034eb_0b_0a_a_fa_ +prometheus:860.v532442b_44e9a_ scm-api:728.vc30dcf7a_0df5 -scm-manager:1.11.1 -script-security:1399.ve6a_66547f6e1 +scm-manager:1.12.1 +script-security:1412.v7737b_3405f86 snakeyaml-api:2.5-149.v72471e9c6371 -snakeyaml-engine-api:3.0.1-5.vd98ea_ff3b_92e +snakeyaml-engine-api:3.1.1-12.v4320c7d6f89c ssh-credentials:372.va_250881b_08cd structs:362.va_b_695ef4fdf9 trilead-api:2.284.v1974ea_324382 variant:70.va_d9f17f859e0 -woodstox-core-api:7.1.1-1.v4d297985f397 +woodstox-core-api:7.2.2-10.vcb_629759b_2c2 workflow-aggregator:608.v67378e9d3db_1 workflow-api:1413.v2ff1a_5e720fa_ workflow-basic-steps:1098.v808b_fd7f8cf4 -workflow-cps:4285.v8df38f05c3c5 -workflow-durable-task-step:1475.ved562f6ec8b_3 -workflow-job:1571.vb_423c255d6d9 -workflow-multibranch:821.vc3b_4ea_780798 +workflow-cps:4370.v49a_6937566b_6 +workflow-durable-task-step:1479.v56e587f413a_7 +workflow-job:1571.1580.v18e46842c125 +workflow-multibranch:841.vec5b_9e1806ec workflow-scm-step:466.va_d69e602552b_ workflow-step-api:724.v538c2362b_dfb_ workflow-support:1015.v785e5a_b_b_8b_22 \ No newline at end of file diff --git a/scripts/keycloak/install-keycloak.sh b/scripts/keycloak/install-keycloak.sh new file mode 100644 index 000000000..13daa390a --- /dev/null +++ b/scripts/keycloak/install-keycloak.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +set -o errexit -o nounset -o pipefail + +KEYCLOAK_NAMESPACE="${KEYCLOAK_NAMESPACE:-keycloak}" +KEYCLOAK_HOST="${KEYCLOAK_HOST:-keycloak.local.gd}" +REALM_EXPORT="${REALM_EXPORT:-docs/oidc/realm-export.json}" + +kubectl create namespace "${KEYCLOAK_NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f - + +kubectl -n "${KEYCLOAK_NAMESPACE}" create configmap keycloak-realm \ + --from-file=realm-export.json="${REALM_EXPORT}" \ + --dry-run=client -o yaml | kubectl apply -f - + +helm upgrade --install keycloak oci://registry-1.docker.io/bitnamicharts/keycloak \ + --namespace "${KEYCLOAK_NAMESPACE}" \ + --reset-values \ + --set global.security.allowInsecureImages=true \ + --set image.registry=docker.io \ + --set image.repository=bitnamilegacy/keycloak \ + --set postgresql.image.registry=docker.io \ + --set postgresql.image.repository=bitnamilegacy/postgresql \ + --set auth.adminUser=admin \ + --set auth.adminPassword=admin \ + --set production=false \ + --set tls.enabled=false \ + --set proxyHeaders=xforwarded \ + --set hostnameStrict=false \ + --set httpEnabled=true \ + --set extraEnvVars[0].name=KC_HOSTNAME \ + --set extraEnvVars[0].value="${KEYCLOAK_HOST}" \ + --set ingress.enabled=true \ + --set ingress.ingressClassName=traefik \ + --set ingress.hostname="${KEYCLOAK_HOST}" \ + --set ingress.tls=false \ + --set keycloakConfigCli.enabled=false \ + --set extraStartupArgs=--import-realm \ + --set extraVolumes[0].name=realm-import \ + --set extraVolumes[0].configMap.name=keycloak-realm \ + --set extraVolumeMounts[0].name=realm-import \ + --set extraVolumeMounts[0].mountPath=/opt/bitnami/keycloak/data/import/realm-export.json \ + --set extraVolumeMounts[0].subPath=realm-export.json \ + --set extraVolumeMounts[0].readOnly=true + +tmp_override="$(mktemp)" +cat > "${tmp_override}" </dev/null 2>&1; then + echo "WARNING: kube-system/coredns-custom already exists; this script will overwrite it." >&2 +fi +kubectl -n kube-system create configmap coredns-custom \ + --from-file=keycloak.override="${tmp_override}" \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl -n kube-system rollout restart deployment/coredns +rm -f "${tmp_override}" + +kubectl -n "${KEYCLOAK_NAMESPACE}" rollout status statefulset/keycloak --timeout=10m \ No newline at end of file diff --git a/scripts/local-openshift/helm/gop-rbac.yaml b/scripts/local-openshift/helm/gop-rbac.yaml new file mode 100644 index 000000000..14124115d --- /dev/null +++ b/scripts/local-openshift/helm/gop-rbac.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: gop-sa + namespace: gop +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: gop-sa-cluster-admin-binding +subjects: + - kind: ServiceAccount + name: gop-sa + namespace: gop +roleRef: + kind: ClusterRole + name: cluster-admin + apiGroup: rbac.authorization.k8s.io diff --git a/scripts/local-openshift/helm/gop-values.yaml b/scripts/local-openshift/helm/gop-values.yaml new file mode 100644 index 000000000..595e6aeea --- /dev/null +++ b/scripts/local-openshift/helm/gop-values.yaml @@ -0,0 +1,29 @@ +# gop-values.yaml - Configuration for GOP Helm Chart on OpenShift Local (CRC) + +# Container image from the internal OpenShift Image Registry +image: + repository: image-registry.openshift-image-registry.svc:5000/gop/gop + tag: latest + pullPolicy: Always + +# Use pre-created ServiceAccount with assigned permissions (cluster-admin + SCC) +serviceAccount: + create: false + name: gop-sa + +logLevel: trace + +# Direct CLI flags passed to the GOP container (takes highest precedence) +extraArgs: + - "--profile=full" + - "--openshift" + - "--base-url=http://apps-crc.testing" + # Optional: Uncomment if all tools should run in a single shared namespace: + # - "--namespace=gop" + +config: + # yaml-language-server: $schema=https://raw.githubusercontent.com/cloudogu/gitops-playground/refs/heads/main/docs/configuration.schema.json + application: + "yes": false # strange, but toggle issue with picocli + password: "admin" + insecure: true diff --git a/scripts/local-openshift/manifest/gop-job.yaml b/scripts/local-openshift/manifest/gop-job.yaml new file mode 100644 index 000000000..0c8a0d8b3 --- /dev/null +++ b/scripts/local-openshift/manifest/gop-job.yaml @@ -0,0 +1,28 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: gop-installer-job + namespace: gop +spec: + # Retain job after completion to preserve logs + backoffLimit: 0 + template: + metadata: + name: gop-installer + labels: + app: gop-installer + spec: + serviceAccountName: gop-sa + containers: + - name: gop-container + image: image-registry.openshift-image-registry.svc:5000/gop/gop:latest + imagePullPolicy: Always + args: + - "--yes=true" + - "--profile=full" + - "-x" + - "--openshift" + - "--base-url=http://apps-crc.testing" + # Optional: Uncomment if all tools should run in a single shared namespace: + # - "--namespace=gop" + restartPolicy: Never diff --git a/scripts/local-openshift/manifest/gop-rbac.yaml b/scripts/local-openshift/manifest/gop-rbac.yaml new file mode 100644 index 000000000..14124115d --- /dev/null +++ b/scripts/local-openshift/manifest/gop-rbac.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: gop-sa + namespace: gop +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: gop-sa-cluster-admin-binding +subjects: + - kind: ServiceAccount + name: gop-sa + namespace: gop +roleRef: + kind: ClusterRole + name: cluster-admin + apiGroup: rbac.authorization.k8s.io diff --git a/scripts/local/install-argocd-operator.sh b/scripts/local/install-argocd-operator.sh deleted file mode 100755 index e94acd2ef..000000000 --- a/scripts/local/install-argocd-operator.sh +++ /dev/null @@ -1,6 +0,0 @@ -git clone https://github.com/argoproj-labs/argocd-operator && \ -cd argocd-operator && \ -git checkout release-0.16 && \ -make deploy IMG=quay.io/argoprojlabs/argocd-operator:v0.17.0 -rm -Rf ../argocd-operator/ -cd .. diff --git a/scripts/local/manual-ingress-deploy.sh b/scripts/local/manual-ingress-deploy.sh index 6f4e067b4..8c914b922 100755 --- a/scripts/local/manual-ingress-deploy.sh +++ b/scripts/local/manual-ingress-deploy.sh @@ -43,7 +43,7 @@ EOF helm repo add traefik https://traefik.github.io/charts helm upgrade --install traefik traefik/traefik \ - --version 39.0.0 \ + --version 39.0.9 \ --namespace ingress \ --create-namespace \ -f values.yaml && rm ./values.yaml \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/application/Application.groovy b/src/main/groovy/com/cloudogu/gitops/application/Application.groovy deleted file mode 100644 index 5e9a491b1..000000000 --- a/src/main/groovy/com/cloudogu/gitops/application/Application.groovy +++ /dev/null @@ -1,90 +0,0 @@ -package com.cloudogu.gitops.application - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.utils.TemplatingEngine -import freemarker.template.Configuration -import freemarker.template.DefaultObjectWrapperBuilder -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton - -@Slf4j -@Singleton -class Application { - - final List features - final Config config - final K8sClient k8sClient - - Application(Config config, K8sClient k8sClient, - List features) { - this.config = config - // Order is important. Enforced by @Order-Annotation on the Singletons - this.features = features - this.k8sClient = k8sClient - } - - def start() { - log.debug("Starting Application") - - setNamespaceListToConfig(config) - // if set, stores configuration in a secret. - storeGopInformationInSecret(config) - - features.forEach(feature -> { - feature.validate() - }) - features.forEach(feature -> { - feature.install() - }) - log.debug("Application finished") - } - - private void storeGopInformationInSecret(Config config) { - String namespace = "gop-job" - // Fallback, if run from IDE - if (!config.application.gopNamespace.isEmpty()) { - // if set, take namespace from configuration - namespace = "${config.application.namePrefix}${config.application.gopNamespace}" - } else if (this.k8sClient.getCurrentNamespace() != null) { - // if gop-namespace not set, take namespace from running GOP - namespace = this.k8sClient.getCurrentNamespace() - } - log.debug("Storing GOP configuration in secret 'gop-configuration' in namespace '${namespace}'") - k8sClient.createNamespace(namespace) - k8sClient.createSecret('generic', 'gop-configuration', namespace, - new Tuple2('gop-initial-password', config.application.password), - new Tuple2('gop-config', config.toYaml(true))) - } - - List getFeatures() { - return features - } - - void setNamespaceListToConfig(Config config) { - LinkedHashSet dedicatedNamespaces = new LinkedHashSet<>() - LinkedHashSet tenantNamespaces = new LinkedHashSet<>() - def engine = new TemplatingEngine() - - config.content.namespaces.each { String ns -> - tenantNamespaces.add(engine.template(ns, [config : config, - // Allow for using static classes inside the templates - statics: new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build().getStaticModels()])) - } - config.content.namespaces = tenantNamespaces.toList() - - //iterates over all FeatureWithImages and gets their namespaces - dedicatedNamespaces.addAll(this.features - .collect { it.activeNamespaceFromFeature } - .findAll { it } - .unique() - .collect { "${it}".toString() }) - - config.application.namespaces.dedicatedNamespaces = dedicatedNamespaces - config.application.namespaces.tenantNamespaces = tenantNamespaces - log.debug("Active namespaces retrieved: {}", config.application.namespaces.activeNamespaces) - } - - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/application/content/ContentLoader.groovy b/src/main/groovy/com/cloudogu/gitops/application/content/ContentLoader.groovy deleted file mode 100644 index bfdae1a71..000000000 --- a/src/main/groovy/com/cloudogu/gitops/application/content/ContentLoader.groovy +++ /dev/null @@ -1,602 +0,0 @@ -package com.cloudogu.gitops.application.content - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Config.OverwriteMode -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.tools.core.Jenkins -import com.cloudogu.gitops.utils.AllowListFreemarkerObjectWrapper -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.MapUtils -import com.cloudogu.gitops.utils.TemplatingEngine -import com.fasterxml.jackson.annotation.JsonIgnore -import freemarker.template.Configuration -import freemarker.template.DefaultObjectWrapperBuilder -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton -import org.apache.commons.io.FileUtils -import org.eclipse.jgit.api.CloneCommand -import org.eclipse.jgit.api.Git -import org.eclipse.jgit.lib.Ref -import org.eclipse.jgit.lib.Repository -import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider - -import java.nio.file.Path - -import static com.cloudogu.gitops.config.Config.ContentRepoType -import static com.cloudogu.gitops.config.Config.ContentSchema.ContentRepositorySchema - -@Slf4j -@Singleton -@Order(999) -// We want to evaluate content last, to allow for changing all other repos -class ContentLoader extends Tool { - private Config config - private K8sClient k8sClient - private GitRepoFactory repoProvider - private Jenkins jenkins - // set by lazy initialisation - private TemplatingEngine templatingEngine - // used to clone repos in validation phase - private List cachedRepoCoordinates = new ArrayList<>() - private GitHandler gitHandler - - protected File mergedReposFolder - - //For security reasons we safe the credentialsProvider for each repo here and not in config pro each repo - @JsonIgnore - UsernamePasswordCredentialsProvider credentialsProvider - - ContentLoader(Config config, - K8sClient k8sClient, - GitRepoFactory repoProvider, - Jenkins jenkins, - GitHandler gitHandler, - FileSystemUtils fileSystemUtils, - DeploymentStrategy deployer) { - this.config = config - this.k8sClient = k8sClient - this.repoProvider = repoProvider - this.jenkins = jenkins - this.gitHandler = gitHandler - this.fileSystemUtils = fileSystemUtils - this.deployer = deployer - } - - @Override - boolean isEnabled() { - return true // for now always on. Once we refactor from Argo CD class we add a param to enable - } - - @Override - void enable() { - // ensure cache is cleaned - clearCache() - // clones repo to check valid configuration and reuse result for further step. - cachedRepoCoordinates = cloneContentRepos() - createImagePullSecrets() - createContentRepos() - deployHelmReleasesFromContent() - } - - @Override - void validate() { - - } - - @Override - void preConfigInit(Config configToSet) { - config.content.repos.each { repo -> - - if (!repo.url) { - throw new RuntimeException("content.repos requires a url parameter.") - } - if (repo.target) { - if (repo.target.count('/') == 0) { - throw new RuntimeException("content.target needs / to separate namespace/group from repo name. Repo: ${repo.url}") - } - } - - switch (repo.type) { - case ContentRepoType.COPY: - if (!repo.target) { - throw new RuntimeException("content.repos.type ${ContentRepoType.COPY} requires content.repos.target to be set. Repo: ${repo.url}") - } - break - case ContentRepoType.FOLDER_BASED: - if (repo.target) { - throw new RuntimeException("content.repos.type ${ContentRepoType.FOLDER_BASED} does not support target parameter. Repo: ${repo.url}") - } - if (repo.targetRef) { - throw new RuntimeException("content.repos.type ${ContentRepoType.FOLDER_BASED} does not support targetRef parameter. Repo: ${repo.url}") - } - break - case ContentRepoType.MIRROR: - if (!repo.target) { - throw new RuntimeException("content.repos.type ${ContentRepoType.MIRROR} requires content.repos.target to be set. Repo: ${repo.url}") - } - if (repo.path != ContentRepositorySchema.DEFAULT_PATH) { - throw new RuntimeException("content.repos.type ${ContentRepoType.MIRROR} does not support path. Current path: ${repo.path}. Repo: ${repo.url}") - } - if (repo.templating) { - throw new RuntimeException("content.repos.type ${ContentRepoType.MIRROR} does not support templating. Repo: ${repo.url}") - } - break - } - } - } - - protected void deployHelmReleasesFromContent() { - if (!config.content?.helmReleases) { - log.debug("No content.helmReleases configured - skipping.") - return - } - - config.content.helmReleases.each { helmRelease -> - String version = helmRelease.version?.trim() - if (!version) { - version = "*" - } - - Config.HelmConfigWithValues helmConfig = new Config.HelmConfigWithValues(repoURL: helmRelease.repoURL, - chart: helmRelease.chart, - version: version, - values: [:] as Map // IMPORTANT: we will pass merged values via a file - ) - - Map fileValues = [:] - if (helmRelease.valuesPath?.trim()) { - // This is a plain YAML file (NOT a .ftl template) - fileValues = (fileSystemUtils.readYaml(Path.of(helmRelease.valuesPath)) ?: [:]) as Map - } - - Map inlineValues = (helmRelease.values ?: [:]) as Map - - // merge: file first, inline overrides - Map mergedValues = MapUtils.deepMerge(inlineValues, fileValues) - - // always write a temp values file and pass its path to deployHelmChart - Path mergedValuesFile = fileSystemUtils.writeTempFile(mergedValues) - - deployHelmChart(helmRelease.name, - helmRelease.releaseName ?: helmRelease.name, - helmRelease.namespace, - helmConfig, - mergedValuesFile.toString(), - config) - } - } - - void createImagePullSecrets() { - if (config.registry.createImagePullSecrets) { - String registryUsername = config.registry.readOnlyUsername ?: config.registry.username - String registryPassword = config.registry.readOnlyPassword ?: config.registry.password - - config.content.namespaces.each { String namespace -> - def registrySecretName = 'registry' - - k8sClient.createNamespace(namespace) - - k8sClient.createImagePullSecret(registrySecretName, namespace, - config.registry.url /* Only domain matters, path would be ignored */, - registryUsername, registryPassword) - - k8sClient.patch('serviceaccount', 'default', namespace, - [imagePullSecrets: [[name: registrySecretName]]]) - - if (config.registry.twoRegistries) { - k8sClient.createImagePullSecret('proxy-registry', namespace, - config.registry.proxyUrl, config.registry.proxyUsername, - config.registry.proxyPassword) - } - } - } - } - - void createContentRepos() { - if (cachedRepoCoordinates.empty) { - cachedRepoCoordinates = cloneContentRepos() - } - pushTargetRepos(cachedRepoCoordinates) - // after all, clean folders and list - clearCache() - } - - protected List cloneContentRepos() { - mergedReposFolder = File.createTempDir('gitops-playground-based-content-repos-') - List repoCoordinates = [] - - log.debug("Aggregating structure for all ${config.content.repos.size()} repos.") - config.content.repos.each { repoConfig -> createRepoCoordinates(repoConfig, mergedReposFolder, repoCoordinates) - } - return repoCoordinates - } - - private TemplatingEngine getTemplatingEngine() { - if (templatingEngine == null) { - templatingEngine = new TemplatingEngine() - } - return templatingEngine - } - - private void createRepoCoordinates(ContentRepositorySchema repoConfig, File mergedReposFolder, List repoCoordinates) { - def repoTmpDir = File.createTempDir('gitops-playground-content-repo-') - log.debug("Cloning content repo, ${repoConfig.url}, revision ${repoConfig.ref}, path ${repoConfig.path}, overwriteMode ${repoConfig.overwriteMode}") - - if (repoConfig.credentials?.username != null && repoConfig.credentials?.password != null) { - credentialsProvider = new UsernamePasswordCredentialsProvider(repoConfig.credentials.username, repoConfig.credentials.password) - } else if (repoConfig.credentials?.secretName && repoConfig.credentials?.secretNamespace) { - Credentials credentials = this.k8sClient.getCredentialsFromSecret(repoConfig.credentials) - credentialsProvider = new UsernamePasswordCredentialsProvider(credentials.username, credentials.password) - } - - cloneToLocalFolder(repoConfig, repoTmpDir) - - def contentRepoDir = new File(repoTmpDir, repoConfig.path) - applyTemplatingIfApplicable(repoConfig, contentRepoDir) - - switch (repoConfig.type) { - case ContentRepoType.FOLDER_BASED: - createRepoCoordinatesForTypeFolderBased(repoConfig, repoTmpDir, contentRepoDir, mergedReposFolder, repoCoordinates) - repoTmpDir.deleteDir() - break - case ContentRepoType.COPY: - createRepoCoordinatesForTypeCopy(repoConfig, contentRepoDir, mergedReposFolder, repoTmpDir, repoCoordinates) - repoTmpDir.deleteDir() - break - case ContentRepoType.MIRROR: - createRepoCoordinateForTypeMirror(repoConfig, repoTmpDir, repoCoordinates) - // intentionally not deleting repoTmpDir, it is contained in RepoCoordinates for MIRROR usage - break - } - log.debug("Finished cloning content repos. repoCoordinates=${repoCoordinates}") - } - - private static void createRepoCoordinatesForTypeCopy(ContentRepositorySchema repoConfig, File contentRepoDir, File mergedReposFolder, File repoTmpDir, - List repoCoordinates) { - String namespace = repoConfig.target.split('/')[0] - String repoName = repoConfig.target.split('/')[1] - - def repoCoordinate = mergeRepoDirs(contentRepoDir, namespace, repoName, mergedReposFolder, repoConfig) - repoCoordinate.refIsTag = GitRepo.isTag(repoTmpDir, repoConfig.ref) - addRepoCoordinates(repoCoordinates, repoCoordinate) - } - - private static void createRepoCoordinatesForTypeFolderBased(ContentRepositorySchema repoConfig, File repoTmpDir, File contentRepoDir, File mergedReposFolder, - List repoCoordinates) { - boolean refIsTag = GitRepo.isTag(repoTmpDir, repoConfig.ref) - findRepoDirectories(contentRepoDir) - .each { contentRepoNamespaceDir -> - findRepoDirectories(contentRepoNamespaceDir) - .each { contentRepoFolder -> - String namespace = contentRepoNamespaceDir.name - String repoName = contentRepoFolder.name - def repoCoordinate = mergeRepoDirs(contentRepoFolder, namespace, repoName, mergedReposFolder, repoConfig) - repoCoordinate.refIsTag = refIsTag - addRepoCoordinates(repoCoordinates, repoCoordinate) - } - } - } - - private static void createRepoCoordinateForTypeMirror(ContentRepositorySchema repoConfig, File repoTmpDir, List repoCoordinates) { - // Don't merge but keep these in separate dirs. - // This avoids messing up .git folders with possible confusing exceptions for the user - String namespace = repoConfig.target.split('/')[0] - String repoName = repoConfig.target.split('/')[1] - def repoCoordinate = new RepoCoordinate(namespace: namespace, - repoName: repoName, - clonedContentRepo: repoTmpDir, - repoConfig: repoConfig, - refIsTag: GitRepo.isTag(repoTmpDir, repoConfig.ref)) - addRepoCoordinates(repoCoordinates, repoCoordinate) - } - - /** - * Merges the files of src into the mergeRepoFolder/namespace/name and adds a new object to repoCoordinates. - * - * Note that existing repoCoordinate objects with different overwriteMode are overwritten. The last repo to be mentioned within config.content.repos wins!*/ - private static RepoCoordinate mergeRepoDirs(File src, String namespace, String repoName, File mergedRepoFolder, - ContentRepositorySchema repoConfig) { - File target = new File(new File(mergedRepoFolder, namespace), repoName) - log.debug("Merging content repo, namespace ${namespace}, repoName ${repoName} from ${src} to ${target}") - FileUtils.copyDirectory(src, target, new FileSystemUtils.IgnoreDotGitFolderFilter()) - - def repoCoordinate = new RepoCoordinate(namespace: namespace, - repoName: repoName, - clonedContentRepo: target, - repoConfig: repoConfig,) - return repoCoordinate - } - - private static List findRepoDirectories(File srcRepo) { - srcRepo.listFiles().findAll { - it.isDirectory() && // Exclude .git for example - !it.name.startsWith('.') - } - } - - private void applyTemplatingIfApplicable(ContentRepositorySchema repoConfig, File srcPath) { - if (repoConfig.templating) { - def engine = getTemplatingEngine() - - GitRepo repo = this.repoProvider.getRepo(repoConfig.target, this.gitHandler.tenant) - - engine.replaceTemplates(srcPath, [config : config, - scm : [baseUrl : repo.gitProvider.url, - host : repo.gitProvider.host, - protocol: repo.gitProvider.protocol, - repoUrl : repo.gitProvider.repoPrefix(),], - // Allow for using static classes inside the templates - statics: !config.content.useWhitelist ? new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build().getStaticModels() : - new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, config.content.getAllowedStaticsWhitelist()).getStaticModels()]) - } - } - - private void cloneToLocalFolder(ContentRepositorySchema repoConfig, File repoTmpDir) { - - def cloneCommand = gitClone() - .setURI(repoConfig.url) - .setDirectory(repoTmpDir) - .setNoCheckout(false) - // Checkout default branch - - if (credentialsProvider) { - cloneCommand.setCredentialsProvider(credentialsProvider) - } - - def git = cloneCommand.call() - - if (ContentRepoType.MIRROR == repoConfig.type) { - def fetch = git.fetch() - - if (credentialsProvider) { - fetch.setCredentialsProvider(credentialsProvider) - } - fetch.setRefSpecs("+refs/*:refs/*").call() // Fetch all branches and tags - } - - if (repoConfig.ref) { - def actualRef = findRef(repoConfig, git.repository) - git.checkout().setName(actualRef).call() - } - } - - private static String findRef(ContentRepositorySchema repoConfig, Repository gitRepo) { - // Check if ref exists first to avoid InvalidRefNameException - // Note that this works for commits and shortname tags but not shortname branches 🙄 - if (gitRepo.resolve(repoConfig.ref)) { - return repoConfig.ref - } - - // Check branches or tags - def remoteCommand = Git.lsRemoteRepository() - .setRemote(repoConfig.url) - .setHeads(true) - .setTags(true) - - Collection refs = remoteCommand.call() - String potentialRef = refs.find { it.name.endsWith(repoConfig.ref) }?.name - - if (!potentialRef) { - // Jgit silently ignores some missing refs and just continues with default branch. - // This might lead to unexpected surprises for our users, so better fail explicitly - throw new RuntimeException("Reference '${repoConfig.ref}' not found in content repository '${repoConfig.url}'") - } - - // Jgit only checks out remote branches when they start in origin/ 🙄 - return potentialRef.replace('refs/heads/', 'origin/') - } - - private void pushTargetRepos(List repoCoordinates) { - repoCoordinates.each { repoCoordinate -> - - GitRepo targetRepo = repoProvider.getRepo(repoCoordinate.fullRepoName, this.gitHandler.tenant) - boolean isNewRepo = targetRepo.createRepositoryAndSetPermission("", false) - - if (isValidForPush(isNewRepo, repoCoordinate)) { - targetRepo.cloneRepo() - - switch (repoCoordinate.repoConfig.type) { - case ContentRepoType.MIRROR: - handleRepoMirroring(repoCoordinate, targetRepo) - break - // COPY and FOLDER_BASED same treatment - case ContentRepoType.FOLDER_BASED: - case ContentRepoType.COPY: - handleRepoCopyingOrFolderBased(repoCoordinate, targetRepo, isNewRepo) - break - } - - createJenkinsJobIfApplicable(repoCoordinate, targetRepo) - - // cleaning tmp folders - repoCoordinate.clonedContentRepo.deleteDir() - new File(targetRepo.absoluteLocalRepoTmpDir).deleteDir() - } // no else needed - } - - } - - /** - * Copies repoCoordinate to targetRepo, commits and pushes - * Same logic for both FOLDER_BASED and COPY repo types.*/ - private static void handleRepoCopyingOrFolderBased(RepoCoordinate repoCoordinate, GitRepo targetRepo, boolean isNewRepo) { - if (!isNewRepo) { - clearTargetRepoIfApplicable(repoCoordinate, targetRepo) - } - // Avoid overwriting .git in target to avoid, because we don't need it for copying and - // git pack files are typically read-only, leading to IllegalArgumentException: - // File parameter 'destFile is not writable: .git/objects/pack/pack-123.pack - targetRepo.copyDirectoryContents(repoCoordinate.clonedContentRepo.absolutePath, new FileSystemUtils.IgnoreDotGitFolderFilter()) - - String commitMessage = "Initialize content repo ${repoCoordinate.namespace}/${repoCoordinate.repoName}" - String targetRefShort = repoCoordinate.repoConfig.targetRef.replace('refs/heads/', '').replace('refs/tags/', '') - if (targetRefShort) { - String refSpec = setRefSpec(repoCoordinate, targetRefShort) - targetRepo.commitAndPush(commitMessage, targetRefShort, refSpec) - } else { - targetRepo.commitAndPush(commitMessage) - } - - } - - private static String setRefSpec(RepoCoordinate repoCoordinate, String targetRefShort) { - String refSpec - if ((repoCoordinate.refIsTag && !repoCoordinate.repoConfig.targetRef.startsWith('refs/heads')) || repoCoordinate.repoConfig.targetRef.startsWith('refs/tags')) { - refSpec = "refs/tags/${targetRefShort}:refs/tags/${targetRefShort}" - } else { - refSpec = "HEAD:refs/heads/${targetRefShort}" - } - refSpec - } - - private static void clearTargetRepoIfApplicable(RepoCoordinate repoCoordinate, GitRepo targetRepo) { - if (OverwriteMode.INIT != repoCoordinate.repoConfig.overwriteMode) { - if (OverwriteMode.RESET == repoCoordinate.repoConfig.overwriteMode) { - log.info("OverwriteMode ${OverwriteMode.RESET} set for repo '${repoCoordinate.fullRepoName}': " + - "Deleting existing files in repo and replacing them with new content.") - targetRepo.clearRepo() - } else { - log.debug("OverwriteMode ${OverwriteMode.UPGRADE} set for repo '${repoCoordinate.fullRepoName}': " + "Merging new content into existing repo. ") - } - } - } - - /** - * Force pushes repoCoordinate.repoConfig.ref or all refs to targetRepo*/ - private static void handleRepoMirroring(RepoCoordinate repoCoordinate, GitRepo targetRepo) { - try (def targetGit = Git.open(new File(targetRepo.absoluteLocalRepoTmpDir))) { - def remoteUrl = targetGit.repository.config.getString('remote', 'origin', 'url') - - // In mirror mode, we mainly need the .git folder to push the whole git history, branches and tags. - // So copying source to target repo, .git folders are merged. - // git pack files are typically read-only, leading to - // IllegalArgumentException: File parameter 'destFile is not writable: .git/objects/pack/pack-123.pack - // Workaround: make .git writable. - // Note: Setting target remote in source repo and pushing from there causes other problems like - // IOException: Source ref someBranch doesn't resolve to any object. - FileSystemUtils.makeWritable(new File(targetRepo.absoluteLocalRepoTmpDir, '.git')) - - targetRepo.copyDirectoryContents(repoCoordinate.clonedContentRepo.absolutePath) - - // Restore remote, it could have been overwritten due to a copied .git folder in MIRROR mode - targetGit.repository.config.setString('remote', 'origin', 'url', remoteUrl) - targetGit.repository.config.save() - } - - if (repoCoordinate.repoConfig.ref) { - validateCommitReferences(repoCoordinate) - if (repoCoordinate.repoConfig.targetRef) { - log.debug("Mirroring repo '${repoCoordinate.repoConfig.url}' ref '${repoCoordinate.repoConfig.ref}' to target repo ${repoCoordinate.fullRepoName}, targetRef: '${repoCoordinate.repoConfig.targetRef}'") - targetRepo.pushRef(repoCoordinate.repoConfig.ref, repoCoordinate.repoConfig.targetRef, true) - } else { - log.debug("Mirroring repo '${repoCoordinate.repoConfig.url}' ref '${repoCoordinate.repoConfig.ref}' to target repo ${repoCoordinate.fullRepoName}") - targetRepo.pushRef(repoCoordinate.repoConfig.ref, true) - } - } else { - log.debug("Mirroring whole repo '${repoCoordinate.repoConfig.url}' to target repo ${repoCoordinate.fullRepoName}") - targetRepo.pushAll(true) - } - } - - private static void validateCommitReferences(RepoCoordinate repoCoordinate) { - if (GitRepo.isCommit(repoCoordinate.clonedContentRepo, repoCoordinate.repoConfig.ref)) { - // Mirroring detached commits does not make a lot of sense and is complicated - // We would have to branch, push, delete remote branch. Considering this an edge case at the moment! - throw new RuntimeException("Mirroring commit references is not supported for content repos at the moment. content repository '${repoCoordinate.repoConfig.url}', ref: ${repoCoordinate.repoConfig.ref}") - } - } - - private void createJenkinsJobIfApplicable(RepoCoordinate repoCoordinate, GitRepo repo) { - if (repoCoordinate.repoConfig.createJenkinsJob && jenkins.isEnabled()) { - if (GitRepo.existFileInSomeBranch(repo.absoluteLocalRepoTmpDir, 'Jenkinsfile')) { - jenkins.createJenkinsjob(repoCoordinate.namespace, repoCoordinate.namespace) - } - } - } - - /** - * Overwrite for testing purposes*/ - protected CloneCommand gitClone() { - Git.cloneRepository() - } - - /** - * Add new repoCoordinates to repos and ensure, newest one override last one. - * Except for MIRROR, which will have to run separately from COPY/FOLDER_BASED in order to allow overriding by COPY/FOLDER_BASED repoCoordinates for the same repo.*/ - static void addRepoCoordinates(List repoCoordinates, RepoCoordinate newRepoCoordinate) { - def existingRepoCoordinates = newRepoCoordinate.findSame(repoCoordinates) - - if (!existingRepoCoordinates.isEmpty()) { - log.debug("Found existing repo coordinates for ${newRepoCoordinate}: ${existingRepoCoordinates}") - - // Don't replace MIRROR coordinates, they are separate git operations - def repoCoordinateToOverwrite = newRepoCoordinate.findSameNotMirror(existingRepoCoordinates) - if (repoCoordinateToOverwrite) { - repoCoordinates.remove(repoCoordinateToOverwrite) - log.debug("Replacing existing repo coordinate ${existingRepoCoordinates} with new one: ${newRepoCoordinate}") - } - } - repoCoordinates << newRepoCoordinate - } - - /** - * Checks whether the repo already exists and overwrite Mode matches.*/ - static boolean isValidForPush(boolean isNewRepo, RepoCoordinate repoCoordinate) { - - if (!isNewRepo && OverwriteMode.INIT == repoCoordinate.repoConfig.overwriteMode) { - log.warn("OverwriteMode ${OverwriteMode.INIT} set for repo '${repoCoordinate.fullRepoName}' " + "and repo already exists in target: Not pushing content!" + - "If you want to override, set ${OverwriteMode.UPGRADE} or ${OverwriteMode.RESET} .") - return false - } - return true - } - - private void clearCache() { - if (mergedReposFolder) { - mergedReposFolder.deleteDir() - } - cachedRepoCoordinates.clear() - mergedReposFolder = null - } - - static class RepoCoordinate { - String namespace - String repoName - File clonedContentRepo - ContentRepositorySchema repoConfig - boolean refIsTag - - @Override - String toString() { - return "RepoCoordinates{ namespace='$namespace', repoName='$repoName', repoConfig.type='${repoConfig.type}', repoConfig.overwriteMode='${repoConfig.overwriteMode}', clonedContentRepo=$clonedContentRepo', refIsTag='${refIsTag}' }" - } - - String getFullRepoName() { - return "${namespace}/${repoName}" - } - - /** - * @return all epoCoordinate with the same fullRepoName. There can be one with either COPY/FOLDER_BASED and many MIRRORs. - */ - List findSame(List repoCoordinates) { - repoCoordinates.findAll() { it.fullRepoName == fullRepoName } - } - - /** - * @return RepoCoordinate with the same fullRepoName and repoConfig.type not MIRROR. There can only ever be one! - */ - RepoCoordinate findSameNotMirror(List repoCoordinates) { - repoCoordinates.find() { - it.fullRepoName == fullRepoName && ContentRepoType.MIRROR != it.repoConfig.type - } - } - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/application/orchestration/GitHandler.groovy b/src/main/groovy/com/cloudogu/gitops/application/orchestration/GitHandler.groovy deleted file mode 100644 index 94578837b..000000000 --- a/src/main/groovy/com/cloudogu/gitops/application/orchestration/GitHandler.groovy +++ /dev/null @@ -1,133 +0,0 @@ -package com.cloudogu.gitops.application.orchestration - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.util.ScmProviderType -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.gitlab.Gitlab -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManager -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.NetworkingUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(60) -class GitHandler extends Tool { - - Config config - - NetworkingUtils networkingUtils - HelmStrategy helmStrategy - FileSystemUtils fileSystemUtils - K8sClient k8sClient - - GitProvider tenant - GitProvider central - - GitHandler(Config config, HelmStrategy helmStrategy, FileSystemUtils fileSystemUtils, K8sClient k8sClient, NetworkingUtils networkingUtils) { - this.config = config - this.helmStrategy = helmStrategy - this.fileSystemUtils = fileSystemUtils - this.k8sClient = k8sClient - this.networkingUtils = networkingUtils - } - - @Override - boolean isEnabled() { - return true - } - - void validate() { - if (config.scm.scmManager.url) { - config.scm.scmManager.internal = false - config.scm.scmManager.urlForJenkins = config.scm.scmManager.url - } else { - log.debug("Setting configs for internal SCM-Manager") - // We use the K8s service as default name here, because it is the only option: - // "scmm.localhost" will not work inside the Pods and k3d-container IP + Port (e.g. 172.x.y.z:9091) - // will not work on Windows and MacOS. - config.scm.scmManager.urlForJenkins = "http://scmm.${config.application.namePrefix}${config.scm.scmManager.namespace}.svc.cluster.local/scm" - - // More internal fields are set lazily in ScmManger.groovy (after SCMM is deployed and ports are known) - } - config.scm.scmManager.gitOpsUsername = "${config.application.namePrefix}gitops" - - if (config.scm.gitlab.url) { - config.scm.scmProviderType = ScmProviderType.GITLAB - config.scm.scmManager = null - if (!config.scm.gitlab.password || !config.scm.gitlab.parentGroupId) { - throw new RuntimeException('GitLab configuration incomplete: please provide both password (PAT) and parentGroupId') - } - } - - } - - //Retrieves the appropriate SCM for cluster resources depending on whether the environment is multi-tenant or not. - GitProvider getResourcesScm() { - if (central) { - return central - } else if (tenant) { - return tenant - } else { - throw new IllegalStateException("No SCM provider found.") - } - } - - @Override - void enable() { - //TenantSCM - switch (config.scm.scmProviderType) { - case ScmProviderType.GITLAB: - this.tenant = new Gitlab(this.config, this.config.scm.gitlab) - break - case ScmProviderType.SCM_MANAGER: - def prefixedNamespace = "${config.application.namePrefix}${config.scm.scmManager.namespace}".toString() - config.scm.scmManager.namespace = prefixedNamespace - this.tenant = new ScmManager(this.config, config.scm.scmManager, helmStrategy, k8sClient, networkingUtils, true) - // this.tenant.setup() setup will be here in future - break - default: - throw new IllegalArgumentException("Unsupported SCM provider found in TenantSCM") - } - - if (config.multiTenant.useDedicatedInstance) { - switch (config.multiTenant.scmProviderType) { - case ScmProviderType.GITLAB: - this.central = new Gitlab(this.config, this.config.multiTenant.gitlab) - break - case ScmProviderType.SCM_MANAGER: - this.central = new ScmManager(this.config, config.multiTenant.scmManager, helmStrategy, k8sClient, networkingUtils) - break - default: - throw new IllegalArgumentException("Unsupported SCM-Central provider: ${config.scm.scmProviderType}") - } - } - - //can be removed if we combine argocd and cluster-resources - final String namePrefix = (config?.application?.namePrefix ?: "").trim() - if (this.central) { - setupRepos(this.central, namePrefix) - setupRepos(this.tenant, namePrefix) - } else { - setupRepos(this.tenant, namePrefix) - } - } - - static void setupRepos(GitProvider gitProvider, String namePrefix = "") { - gitProvider.createRepository(withOrgPrefix(namePrefix, "argocd/cluster-resources"), - "GitOps repo for basic cluster-resources") - } - - /** - * Adds a prefix to the group/namespace part (before the first '/'): - * Example: "argocd/argocd" + "foo-" => "foo-argocd/argocd"*/ - static String withOrgPrefix(String prefix, String repoPath) { - if (!prefix) return repoPath - return prefix + repoPath - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/cli/ApplicationConfigurator.groovy b/src/main/groovy/com/cloudogu/gitops/cli/ApplicationConfigurator.groovy deleted file mode 100644 index 34368e4f9..000000000 --- a/src/main/groovy/com/cloudogu/gitops/cli/ApplicationConfigurator.groovy +++ /dev/null @@ -1,337 +0,0 @@ -package com.cloudogu.gitops.cli - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.util.logging.Slf4j - -@Slf4j -class ApplicationConfigurator { - - private FileSystemUtils fileSystemUtils - - ApplicationConfigurator(FileSystemUtils fileSystemUtils = new FileSystemUtils()) { - this.fileSystemUtils = fileSystemUtils - } - - /** - * Sets dynamic fields and validates params*/ - Config initConfig(Config newConfig) { - - addAdditionalApplicationConfig(newConfig) - - addNamePrefix(newConfig) - - checkAndSetNamespaces(newConfig) - - addScmConfig(newConfig) - - addRegistryConfig(newConfig) - - addJenkinsConfig(newConfig) - - addFeatureConfig(newConfig) - - evaluateBaseUrl(newConfig) - - setResourceInclusionsCluster(newConfig) - - setMultiTenantModeConfig(newConfig) - - return newConfig - } - - private void addFeatureConfig(Config newConfig) { - if (newConfig.features.secrets.vault.mode) newConfig.features.secrets.active = true - - if (newConfig.features.mail.smtpAddress) newConfig.features.mail.active = true - - if (newConfig.features.ingress.active && !newConfig.application.baseUrl) { - log.warn("Ingress-controller is activated without baseUrl parameter. Services will not be accessible by hostnames. To avoid this use baseUrl with ingress. ") - } - } - - private void addNamePrefix(Config newConfig) { - String namePrefix = newConfig.application.namePrefix - if (namePrefix) { - if (!namePrefix.endsWith('-')) { - newConfig.application.namePrefix = "${namePrefix}-" - } - newConfig.application.namePrefixForEnvVars = "${(newConfig.application.namePrefix as String).toUpperCase().replace('-', '_')}" - } - } - - private void addRegistryConfig(Config newConfig) { - // Process image pull secrets first, they might even be relevant if no registry is set - if (newConfig.registry.createImagePullSecrets) { - String username = newConfig.registry.readOnlyUsername ?: newConfig.registry.username - String password = newConfig.registry.readOnlyPassword ?: newConfig.registry.password - if (!username || !password) { - throw new RuntimeException("createImagePullSecrets needs to be used with either registry username and password or the readOnly variants") - } - } - - if (newConfig.registry.url) { - newConfig.registry.internal = false - newConfig.registry.active = true - } else if (newConfig.registry.active) { - /* Internal Docker registry must be on localhost. Otherwise docker will use HTTPS, leading to errors on - docker push in the example application's Jenkins Jobs. - Both setting up HTTPS or allowing insecure registry via daemon.json makes the playground difficult to use. - So, always use localhost. - Allow overriding the port, in case multiple playground instance run on a single host in different - k3d clusters. */ - newConfig.registry.internal = true - newConfig.registry.url = "localhost:${newConfig.registry.internalPort}" - } else { - // Registry not active, no need to set the following values - return - } - - if (newConfig.registry.proxyUrl) { - newConfig.registry.twoRegistries = true - if (!newConfig.registry.proxyUsername || !newConfig.registry.proxyPassword) { - throw new RuntimeException("Proxy URL needs to be used with proxy-username and proxy-password") - } - } - } - - private void addAdditionalApplicationConfig(Config newConfig) { - if (System.getenv("KUBERNETES_SERVICE_HOST")) { - log.debug("installation is running in kubernetes.") - newConfig.application.runningInsideK8s = true - } - } - - private void addScmConfig(Config newConfig) { - log.debug("Adding additional config for SCM") - - if (newConfig.scm.scmManager.url) { - log.debug("Setting external scmm config") - newConfig.scm.scmManager.internal = false - newConfig.scm.scmManager.urlForJenkins = newConfig.scm.scmManager.url - } else { - log.debug("Setting configs for internal SCM-Manager") - // We use the K8s service as default name here, because it is the only option: - // "scmm.localhost" will not work inside the Pods and k3d-container IP + Port (e.g. 172.x.y.z:9091) - // will not work on Windows and MacOS. - newConfig.scm.scmManager.urlForJenkins = "http://scmm.${newConfig.application.namePrefix}${newConfig.scm.scmManager.namespace}.svc.cluster.local/scm" - - // More internal fields are set lazily in ScmManger.groovy (after SCMM is deployed and ports are known) - } - - // We probably could get rid of some of the complexity by refactoring url, host and ingress into a single var - if (newConfig.application.baseUrl) { - newConfig.scm.scmManager.ingress = new URL(injectSubdomain("scmm", - newConfig.application.baseUrl as String, newConfig.application.urlSeparatorHyphen as Boolean)).host - } - // When specific user/pw are not set, set them to global values - if (newConfig.scm.scmManager.password === Config.DEFAULT_ADMIN_PW) { - newConfig.scm.scmManager.password = newConfig.application.password - } - if (newConfig.scm.scmManager.username === Config.DEFAULT_ADMIN_USER) { - newConfig.scm.scmManager.username = newConfig.application.username - } - - } - - private void addJenkinsConfig(Config newConfig) { - log.debug("Adding additional config for Jenkins") - if (newConfig.jenkins.url) { - log.debug("Setting external jenkins config") - newConfig.jenkins.active = true - newConfig.jenkins.internal = false - newConfig.jenkins.urlForScm = newConfig.jenkins.url - } else if (newConfig.jenkins.active) { - log.debug("Setting configs for internal jenkins") - // We use the K8s service as default name here, because it is the only option: - // "jenkins.localhost" will not work inside the Pods and k3d-container IP + Port (e.g. 172.x.y.z:9090) - // will not work on Windows and MacOS. - String defaultNamespace = newConfig.jenkins.namespace - newConfig.jenkins.urlForScm = "http://jenkins.${newConfig.application.namePrefix}${defaultNamespace}.svc.cluster.local" - - // More internal fields are set lazily in Jenkins.groovy (after Jenkins is deployed and ports are known) - } else { - // Jenkins not active, no need to set the following values - return - } - - if (newConfig.application.baseUrl) { - newConfig.jenkins.ingress = new URL(injectSubdomain("jenkins", - newConfig.application.baseUrl, newConfig.application.urlSeparatorHyphen)).host - } - // When specific user/pw are not set, set them to global values - if (newConfig.jenkins.username === Config.DEFAULT_ADMIN_USER) { - newConfig.jenkins.username = newConfig.application.username - } - if (newConfig.jenkins.password === Config.DEFAULT_ADMIN_PW) { - newConfig.jenkins.password = newConfig.application.password - } - } - - private void evaluateBaseUrl(Config newConfig) { - String baseUrl = newConfig.application.baseUrl - if (!baseUrl) { - return - } - log.debug("Base URL set, adapting to individual tools") - def argocd = newConfig.features.argocd - def mail = newConfig.features.mail - def monitoring = newConfig.features.monitoring - def vault = newConfig.features.secrets.vault - boolean urlSeparatorHyphen = newConfig.application.urlSeparatorHyphen - - if (argocd.active && !argocd.url) { - argocd.url = injectSubdomain("argocd", baseUrl, urlSeparatorHyphen) - log.debug("Setting ArgoCD URL ${argocd.url}") - } - if (monitoring.active && !monitoring.grafanaUrl) { - monitoring.grafanaUrl = injectSubdomain('grafana', baseUrl, urlSeparatorHyphen) - log.debug("Setting Monitoring URL ${monitoring.grafanaUrl}") - } - if (newConfig.features.secrets.active && !vault.url) { - vault.url = injectSubdomain('vault', baseUrl, urlSeparatorHyphen) - log.debug("Setting Vault URL ${vault.url}") - } - - } - - void setMultiTenantModeConfig(Config newConfig) { - if (newConfig.multiTenant.useDedicatedInstance) { - if (!newConfig.application.namePrefix) { - throw new RuntimeException('To enable Central Multi-Tenant mode, you must define a name prefix to distinguish between instances.') - } - - if (!newConfig.features.argocd.operator) { - newConfig.features.argocd.operator = true - } - - // Removes trailing slash from the input URL to avoid duplicated slashes in further URL handling - if (newConfig.multiTenant.scmManager.url) { - String urlString = newConfig.multiTenant.scmManager.url.toString() - if (urlString.endsWith("/")) { - urlString = urlString[0..-2] - } - newConfig.multiTenant.scmManager.url = urlString - } - - //Disabling Ingress in DedicatedInstances Mode for now. - //Ingress has to be handled by Cluster, not by this tenant. - //Ingress has to be handled manually for local dev. - //See /scripts/local/ for local dev. - newConfig.features.ingress.active = false - } - } - - /** - * - * @param subdomain , e.g. argocd - * @param baseUrl e.g. http://localhost:8080 - * @param urlSeparatorHyphen - * @return e.g. http://argocd.localhost:8080 - */ - private String injectSubdomain(String subdomain, String baseUrl, boolean urlSeparatorHyphen) { - URL url = new URL(baseUrl) - String newUrl - - if (urlSeparatorHyphen) { - newUrl = url.getProtocol() + "://" + subdomain + "-" + url.getHost() - } else { - newUrl = url.getProtocol() + "://" + subdomain + "." + url.getHost() - } - if (url.getPort() != -1) { - newUrl += ":" + url.getPort() - } - newUrl += url.getPath() - return newUrl - } - - private void setResourceInclusionsCluster(Config configToSet) { - // Return early if NOT deploying via operator - if (!configToSet.features.argocd.operator) { - log.debug("ArgoCD operator is not enabled. Skipping features.argocd.resourceInclusionsCluster setup.") - return - } - log.info("Starting setup of features.argocd.resourceInclusionsCluster for ArgoCD Operator") - - if (!isUrlSetAndValid(configToSet)) { - // If features.argocd.resourceInclusionsClus 0 && args[0] == '-') { - println(prettyJson) - } else { - new File(SCHEMA_FILE).setText(prettyJson) - println "Wrote schema to ${SCHEMA_FILE}" - - new File(DOCS_FILE).setText(generateDocs()) - println "Wrote documentation to ${DOCS_FILE}" - } - } - - static String generateDocs() { - Config config = new Config() - StringBuilder md = new StringBuilder() - - md << '# Overview of all CLI and config options\n\n' - md << 'All options can be set via a [config file](./configuration.schema.json). ' - md << 'Most options are also available as CLI parameters.\n\n' - - List topFields = schemaFields(Config).findAll { Field field -> field.name !in ['features', 'stages'] } - - // Table of contents - md << '## Table of Contents\n\n' - topFields.each { f -> md << "- [${sectionTitle(f.name)}](#${anchor(f.name)})\n" } - md << '- [Tools](#tools)\n' - schemaFields(Config.FeaturesSchema).each { f -> md << " - [${sectionTitle(f.name)}](#tools-${anchor(f.name)})\n" - } - md << '\n' - - // Top-level sections - topFields.each { field -> - field.accessible = true - md << "## ${sectionTitle(field.name)}\n\n" - md << buildTable(field.get(config), field.type, field.name) - } - - // Tools sub-sections - md << '## Tools\n\n' - md << 'Configuration of optional tools supported by gitops-playground.\n\n' - schemaFields(Config.FeaturesSchema).each { field -> - field.accessible = true - md << "### Tool: ${sectionTitle(field.name)}\n\n" - md << buildTable(field.get(config.features), field.type, "features.${field.name}") - } - - return md.toString() - } - - static String buildTable(Object instance, Class clazz, String prefix) { - List rows = collectRows(instance, clazz, prefix) - if (!rows) { return '' } - - StringBuilder sb = new StringBuilder() - sb << '| CLI | Config key | Type | Default | Description |\n' - sb << '| :--- | :--- | :--- | :--- | :--- |\n' - rows.each { Map r -> sb << "| ${r.cli} | `${r.key}` | ${r.type} | `${r.default}` | ${r.desc} |\n" - } - sb << '\n' - return sb.toString() - } - - static List collectRows(Object instance, Class clazz, String prefix) { - List rows = [] - allFields(clazz).each { Field field -> - if (isInternalField(field)) { return } - - JsonPropertyDescription jsonDesc = field.getAnnotation(JsonPropertyDescription) - CliOption cliOpt = field.getAnnotation(CliOption) - if (!jsonDesc && !cliOpt) { return } - - field.accessible = true - String key = "${prefix}.${field.name}" - - if (isSchemaType(field.type)) { - rows.addAll(collectRows(safeGet(field, instance), field.type, key)) - } else { - rows << [cli : cliOpt ? cliOpt.names().collect { String opt -> "`${opt}`" }.join(', ') : '-', - key : key, - type : typeName(field), - default: formatDefault(safeGet(field, instance)), - desc : (jsonDesc?.value() ?: '-').replaceAll(/\s*\n\s*/, ' ').trim(),] - } - } - return rows - } - - static List allFields(Class clazz) { - List fields = [] - for (Class c = clazz; c && c != Object; c = c.superclass) { - fields.addAll(c.declaredFields) - } - return fields - } - - static List schemaFields(Class clazz) { - return clazz.declaredFields.findAll { Field field -> !isInternalField(field) && isSchemaType(field.type) } - } - - static boolean isInternalField(Field field) { - if (field.synthetic) { return true } - if (Modifier.isStatic(field.modifiers)) { return true } - if (field.getAnnotation(JsonIgnore)) { return true } - return (field.name in ['metaClass', '$staticClassInfo', '__$stMC']) - } - - static boolean isSchemaType(Class type) { - return type.name.startsWith('com.cloudogu.gitops') - } - - static Object safeGet(Field field, Object instance) { - try { field.accessible = true; return field.get(instance) } catch (e) { return null } - } - - static String formatDefault(Object value) { - switch (value) { - case null: return '-' - case Map: return value ? '[:]' : value.toString() - case Collection: return value ? '[]' : value.toString() - default: return value.toString() - } - } - - static String typeName(Field field) { - Class t = field.type - if (t == Boolean || t == boolean) { return 'Boolean' } - if (t == Integer || t == int) { return 'Integer' } - if (t == String) { return 'String' } - if (Map.isAssignableFrom(t)) { return 'Map' } - if (t.enum) { return t.simpleName } - if (field.genericType instanceof ParameterizedType) { - ParameterizedType pt = field.genericType as ParameterizedType - String args = pt.actualTypeArguments.collect { it -> it instanceof Class ? (it as Class).simpleName : it.toString() - }.join(', ') - return "${(pt.rawType as Class).simpleName}<${args}>" - } - return t.simpleName - } - - static String sectionTitle(String name) { - return name.replaceAll(/([A-Z])/, ' $1').trim().with { String title -> title[0].toUpperCase() + title[1..-1] } - } - - static String anchor(String name) { - return sectionTitle(name).toLowerCase().replaceAll(/\s+/, '-') - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCli.groovy b/src/main/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCli.groovy deleted file mode 100644 index 3311bf5f8..000000000 --- a/src/main/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCli.groovy +++ /dev/null @@ -1,284 +0,0 @@ -package com.cloudogu.gitops.cli - -import static com.cloudogu.gitops.config.ConfigConstants.APP_NAME -import static com.cloudogu.gitops.utils.MapUtils.deepMerge -import static com.cloudogu.gitops.utils.MapUtils.deepMergeDefaults - -import com.cloudogu.gitops.application.Application -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.schema.JsonSchemaValidator -import com.cloudogu.gitops.destroy.Destroyer -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.CommonToolConfig -import com.cloudogu.gitops.tools.common.Tool - -import io.micronaut.context.ApplicationContext - -import groovy.util.logging.Slf4j -import groovy.yaml.YamlSlurper - -import ch.qos.logback.classic.Level -import ch.qos.logback.classic.Logger -import ch.qos.logback.classic.LoggerContext -import ch.qos.logback.classic.encoder.PatternLayoutEncoder -import ch.qos.logback.classic.spi.ILoggingEvent -import ch.qos.logback.core.ConsoleAppender -import org.slf4j.LoggerFactory -import picocli.CommandLine - -/** - * Provides the entrypoint to the application as well as all config parameters. - * When changing parameters, make sure to update the Config for the config file as well - * - * @see Config - */ -@Slf4j -class GitopsPlaygroundCli { - - K8sClient k8sClient - ApplicationConfigurator applicationConfigurator - - GitopsPlaygroundCli(K8sClient k8sClient = new K8sClient(), - ApplicationConfigurator applicationConfigurator = new ApplicationConfigurator()) { - this.k8sClient = k8sClient - this.applicationConfigurator = applicationConfigurator - } - - ReturnCode run(String[] args) { - setLogging(args) - - log.debug("Reading initial CLI params") - def cliParams = new Config() - new CommandLine(cliParams).parseArgs(args) - - if (cliParams.application.usageHelpRequested) { - // if help is requested picocli help is used and printed by execute automatically - new CommandLine(cliParams).execute(args) - return ReturnCode.SUCCESS - } - - def version = createVersionOutput() - if (cliParams.application.versionInfoRequested) { - println version - return ReturnCode.SUCCESS - } - - def context = createApplicationContext() - Application app = context.getBean(Application) - - def config = readConfigs(args) - runHook(app, 'preConfigInit', config) - - if (config.application.outputConfigFile) { - println(config.toYaml(false)) - return ReturnCode.SUCCESS - } - - // Set internal values in config after help/version/output because these should work without connecting to k8s - // eg a simple docker run .. --help should not fail with connection refused - config = applicationConfigurator.initConfig(config) - log.debug("Actual config: ${config.toYaml(true)}") - runHook(app, 'postConfigInit', config) - - context = createApplicationContext() - register(config, context) - - if (config.application.destroy) { - log.info version - if (!confirm("Destroying gitops playground in kubernetes cluster '${k8sClient.currentContext}'.", config)) { - return ReturnCode.NOT_CONFIRMED - } - - Destroyer destroyer = context.getBean(Destroyer) - destroyer.destroy() - } else { - log.info version - if (!confirm("Applying gitops playground to kubernetes cluster '${k8sClient.currentContext}'.", config)) { - return ReturnCode.NOT_CONFIRMED - } - app = context.getBean(Application) - app.start() - - printWelcomeScreen(config.application.password) - } - - return ReturnCode.SUCCESS - } - - protected String createVersionOutput() { - def versionName = Version.NAME.replace('\\n', '\n') - - if (versionName.trim().startsWith('(')) { - // When there is no git tag, print commit without parentheses - versionName = versionName.trim() - .replace('(', '') - .replace(')', '') - } - return "${APP_NAME} ${versionName}" - } - - /** Can be used as a hook by child classes */ - @SuppressWarnings('GrMethodMayBeStatic') - // static methods cannot be overridden - protected void register(Config config, ApplicationContext context) { - context.registerSingleton(config) - } - - private static boolean confirm(String message, Config config) { - if (config.application.yes) { - return true - } - - log.info("\n${message}\nContinue? y/n [n]") - - def input = System.in.newReader().readLine() - - return input == 'y' - } - - /** Can be used as a hook by tests */ - protected ApplicationContext createApplicationContext() { - ApplicationContext.run() - } - - private void setLogging(String[] args) { - Logger logger = (Logger) LoggerFactory.getLogger("com.cloudogu.gitops") - if (args.contains('--trace') || args.contains('-x')) { - log.info("Setting loglevel to trace") - logger.setLevel(Level.TRACE) - // log levels can be set via picocli.trace sys env - defaults to 'WARN' - System.setProperty("picocli.trace", "DEBUG") - } else if (args.contains('--debug') || args.contains('-d')) { - System.setProperty("picocli.trace", "INFO") - logger.setLevel(Level.DEBUG) - log.info("Setting loglevel to debug") - } else { - setSimpleLogPattern() - } - } - - /** - * Changes log pattern to a simpler one, to reduce noise for normal users*/ - void setSimpleLogPattern() { - LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory() - def rootLogger = loggerContext.getLogger(Logger.ROOT_LOGGER_NAME) - def defaultPattern = ((rootLogger.getAppender('STDOUT') as ConsoleAppender) - .getEncoder() as PatternLayoutEncoder).pattern - - // Avoid duplicate output by existing appender - rootLogger.detachAppender('STDOUT') - PatternLayoutEncoder encoder = new PatternLayoutEncoder() - // Remove less relevant details from log pattern - encoder.setPattern(defaultPattern - .replaceAll(" \\S*%thread\\S* ", " ") - .replaceAll(" \\S*%logger\\S* ", " ")) - encoder.setContext(loggerContext) - encoder.start() - ConsoleAppender appender = new ConsoleAppender<>() - appender.setName('STDOUT') - appender.setContext(loggerContext) - appender.setEncoder(encoder) - appender.start() - rootLogger.addAppender(appender) - } - - private Config readConfigs(String[] args) { - def cliParams = new Config() - new CommandLine(cliParams).parseArgs(args) - - // first evaluate profile for setting predefined values e.g. examples, if applicable - Config profileConfig = extractProfile(cliParams) - - List configFile = [] - List configMap = [] - - for (String configFileItem : cliParams.application.configFiles) { - log.debug("Reading config file ${configFileItem}") - configFile.add(validateConfig(new File(configFileItem).text)) - } - - for (String configMapItem : cliParams.application.configMaps) { - log.debug("Reading config map ${configMapItem}") - def configValues = k8sClient.getConfigMap(configMapItem, 'config.yaml') - configMap.add(validateConfig(configValues)) - } - - // Last one takes precedence - def configPrecedence = [profileConfig.toMap(), configMap, configFile] - Map mergedConfigs = [:] - configPrecedence.flatten().each { element -> deepMerge(element as Map, mergedConfigs) - } - - // DeepMerge with default Config values to keep the default values defined in Config.groovy - mergedConfigs = deepMergeDefaults(mergedConfigs, new Config().toMap()) - - log.debug("Writing CLI params into config") - Config mergedConfig = Config.fromMap(mergedConfigs) - new CommandLine(mergedConfig).parseArgs(args) - - return mergedConfig - } - - static Map validateConfig(String configValues) { - def map = new YamlSlurper().parseText(configValues) - if (!(map instanceof Map)) { - throw new RuntimeException("Could not parse YAML as map: $map") - } - JsonSchemaValidator.validate(map as Map) - return map as Map - } - - void printWelcomeScreen(String password) { - log.info '''\n - |----------------------------------------------------------------------------------------------| - | Welcome to the GitOps playground by Cloudogu! - |----------------------------------------------------------------------------------------------| - | - | Please find the URLs of the individual applications in our README: - | https://github.com/cloudogu/gitops-playground/blob/main/README.md#table-of-contents - | - | A good starting point might also be the services or ingresses inside your cluster: - | kubectl get svc -A - | Or (depending on your config) - | kubectl get ing -A - | - | Please be aware, Jenkins and Argo CD may take some time to build and deploy all apps. - | - | ''' + "Your initial password for all apps (if not set manually): ${password}" + ''' - | - |----------------------------------------------------------------------------------------------| -''' - - } - - static void runHook(Application app, String methodName, def config) { - ([new CommonToolConfig(), *app.features]).each { feature -> - // Executing only the method if the derived feature class has implemented the passed methodName - def mm = feature.metaClass.getMetaMethod(methodName, config) - if (mm && mm.declaringClass.theClass != Tool) { - log.debug("Executing ${methodName} hook on feature ${feature.class.name}") - mm.invoke(feature, config) - } - } - } - - private static Config extractProfile(Config newConfig) { - - String profile = newConfig.application.profile - - Config profileConfig = new Config() - if (profile) { - String resourceName = "application-${profile}.yaml" - log.debug("Loading profile '${resourceName}' from classpath") - - def inputStream = GitopsPlaygroundCli.class.getResourceAsStream("/${resourceName}") - if (inputStream == null) { - throw new RuntimeException("Profile '${profile}' does not exist (resource '${resourceName}' not found).") - } - String content = inputStream.text - Map profileFile = validateConfig(content) - profileConfig = Config.fromMap(profileFile) - } - return profileConfig - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliMain.groovy b/src/main/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliMain.groovy deleted file mode 100644 index d34f10dd4..000000000 --- a/src/main/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliMain.groovy +++ /dev/null @@ -1,29 +0,0 @@ -package com.cloudogu.gitops.cli - -import groovy.util.logging.Slf4j - -@Slf4j -class GitopsPlaygroundCliMain { - - static void main(String[] args) throws Exception { - new GitopsPlaygroundCliMain().exec(args, GitopsPlaygroundCli.class) - } - - @SuppressWarnings('GrMethodMayBeStatic') - // Non-static for easier testing and reuse - void exec(String[] args, Class commandClass) { - GitopsPlaygroundCli app = commandClass.getDeclaredConstructor().newInstance() - - try { - System.exit(app.run(args).ordinal()) - } catch (RuntimeException e) { - if (log.isDebugEnabled()) { - log.error('', e) - } else { - log.error(e.message) - } - System.exit(ReturnCode.GENERIC_ERROR.ordinal()) - } - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/cli/ReturnCode.groovy b/src/main/groovy/com/cloudogu/gitops/cli/ReturnCode.groovy deleted file mode 100644 index 26e0a4631..000000000 --- a/src/main/groovy/com/cloudogu/gitops/cli/ReturnCode.groovy +++ /dev/null @@ -1,5 +0,0 @@ -package com.cloudogu.gitops.cli - -enum ReturnCode { - SUCCESS, NOT_CONFIRMED, GENERIC_ERROR -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/cli/package-info.java b/src/main/groovy/com/cloudogu/gitops/cli/package-info.java deleted file mode 100644 index 44a308d5b..000000000 --- a/src/main/groovy/com/cloudogu/gitops/cli/package-info.java +++ /dev/null @@ -1,7 +0,0 @@ -/** - * Creates class Version during build via annotation processing - */ -@VersionName(packageName = "com.cloudogu.gitops.cli") -package com.cloudogu.gitops.cli; - -import com.cloudogu.versionname.VersionName; \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/Config.groovy b/src/main/groovy/com/cloudogu/gitops/config/Config.groovy deleted file mode 100644 index 055f4f14c..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/Config.groovy +++ /dev/null @@ -1,810 +0,0 @@ -package com.cloudogu.gitops.config - -import static com.cloudogu.gitops.config.ConfigConstants.* -import static picocli.CommandLine.ScopeType - -import com.cloudogu.gitops.config.scm.ScmTenantSchema - -import java.security.SecureRandom -import jakarta.inject.Singleton -import groovy.transform.CompileStatic -import groovy.transform.MapConstructor - -import com.fasterxml.jackson.annotation.JsonIgnore -import com.fasterxml.jackson.annotation.JsonPropertyDescription -import com.fasterxml.jackson.core.JsonGenerator -import com.fasterxml.jackson.databind.* -import com.fasterxml.jackson.databind.module.SimpleModule -import com.fasterxml.jackson.databind.ser.BeanPropertyWriter -import com.fasterxml.jackson.databind.ser.BeanSerializerModifier -import com.fasterxml.jackson.dataformat.yaml.YAMLMapper -import picocli.CommandLine.Command -import picocli.CommandLine.Mixin -import picocli.CommandLine.Option - -/** - * The global configuration object. - * - * Also used to create the schema for the configuration file or map, which is used to validate the passed YAML file. - * - * Note that all properties marked with - * * {@link JsonPropertyDescription} (written into the Config for config file and config map) - * * {@link Option} (CLI Options) - * - * are external properties that can be changed by the user. - * All other properties are internal. - * - * When changing values make sure to recreate file configuration.schema.json using JsonSchemaGenerator - * (copy output into file an format using IDE). - * - * Make sure not to forget {@link Mixin} at sub types that contain CLI {@link Option}s. Otherwise they are ignored by - * picocli. - * - * Default values - * - Boolean is set to false - * - String uses empty string, because of too many null checks in freemarker and usages. - * - * @see com.cloudogu.gitops.cli.GitopsPlaygroundCli - initializes from file, and CLI - */ -@Singleton -@MapConstructor(noArg = true, includeSuperProperties = true, includeFields = true) -@Command(name = BINARY_NAME, description = APP_DESCRIPTION) -@CompileStatic -class Config { - - // When updating please also update in Dockerfile - public static final String HELM_IMAGE = "ghcr.io/cloudogu/helm:4.2.1-1" - // When updating please also adapt in Dockerfile, vars.tf and init-cluster.sh - public static final String K8S_VERSION = "1.36.2" - public static final String DEFAULT_ADMIN_USER = 'admin' - public static final String DEFAULT_ADMIN_PW = generatePassword() - public static final int DEFAULT_REGISTRY_PORT = 30000 - - @JsonPropertyDescription(REGISTRY_DESCRIPTION) - @Mixin - RegistrySchema registry = new RegistrySchema() - - @JsonPropertyDescription(JENKINS_DESCRIPTION) - @Mixin - JenkinsSchema jenkins = new JenkinsSchema() - - @JsonPropertyDescription(MULTITENANT_DESCRIPTION) - @Mixin - MultiTenantSchema multiTenant = new MultiTenantSchema() - - @JsonPropertyDescription(SCM_DESCRIPTION) - @Mixin - ScmTenantSchema scm = new ScmTenantSchema() - - @JsonPropertyDescription(APPLICATION_DESCRIPTION) - @Mixin - ApplicationSchema application = new ApplicationSchema() - - @JsonPropertyDescription(FEATURES_DESCRIPTION) - @Mixin - FeaturesSchema features = new FeaturesSchema() - - @JsonPropertyDescription(CONTENT_DESCRIPTION) - @Mixin - ContentSchema content = new ContentSchema() - /** - * Generates an admin password. - * @return - */ - private static generatePassword() { - return new SecureRandom() - .with { sr -> - (1..12).collect { - ('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@$%&')[sr.nextInt(62)] - }.join('') - } - } - - static class ContentSchema { - @JsonPropertyDescription(CONTENT_NAMESPACES_DESCRIPTION) - List namespaces = [] - - @JsonPropertyDescription(CONTENT_REPO_DESCRIPTION) - List repos = [] - - @JsonPropertyDescription(CONTENT_VARIABLES_DESCRIPTION) - Map variables = [:] - - // ✅ NEW: helm releases that should be deployed via ArgoCDApplicationStrategy without requiring a git repo - @JsonPropertyDescription() - //(CONTENT_HELM_RELEASES_DESCRIPTION) - List helmReleases = [] - - @Option(names = ['--content-whitelist'], description = CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION) - @JsonPropertyDescription(CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION) - Boolean useWhitelist = false - - @JsonPropertyDescription(CONTENT_STATICSWHITELIST_DESCRIPTION) - Set allowedStaticsWhitelist = ['java.lang.String', - 'java.lang.Integer', - 'java.lang.Long', - 'java.lang.Double', - 'java.lang.Float', - 'java.lang.Boolean', - 'java.lang.Math', - 'com.cloudogu.gitops.utils.DockerImageParser'] as Set - - static class ContentRepositorySchema { - static final String DEFAULT_PATH = '.' - // This is controversial. Forcing users to explicitly choose a type requires them to understand the concept - // of types. What would be a good default? The simplest use case ist MIRROR from url to target. - // COPY and FOLDER_BASED are more advanced use cases. So we choose MIRROR as the default. - static final ContentRepoType DEFAULT_TYPE = ContentRepoType.MIRROR - - @JsonPropertyDescription(CONTENT_REPO_URL_DESCRIPTION) - String url = '' - - @JsonPropertyDescription(CONTENT_REPO_PATH_DESCRIPTION) - String path = DEFAULT_PATH - - @JsonPropertyDescription(CONTENT_REPO_REF_DESCRIPTION) - String ref = '' - - @JsonPropertyDescription(CONTENT_REPO_TARGET_REF_DESCRIPTION) - String targetRef = '' - - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - Credentials credentials - - @JsonPropertyDescription(CONTENT_REPO_TEMPLATING_DESCRIPTION) - Boolean templating = false - - @JsonPropertyDescription(CONTENT_REPO_TYPE_DESCRIPTION) - ContentRepoType type = DEFAULT_TYPE - - @JsonPropertyDescription(CONTENT_REPO_TARGET_DESCRIPTION) - String target = '' - - @JsonPropertyDescription(CONTENT_REPO_TARGET_OVERWRITE_MODE_DESCRIPTION) - OverwriteMode overwriteMode = OverwriteMode.INIT - // Defensively use init to not override existing files by default - - @JsonPropertyDescription(CONTENT_REPO_CREATE_JENKINS_JOB_DESCRIPTION) - Boolean createJenkinsJob = false - - } - - static class HelmReleaseSchema { - @JsonPropertyDescription(CONTENT_HELM_RELEASE_NAME_DESCRIPTION) - String name = '' - // featureName/apps/, also default for releaseName - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_REPO_URL_DESCRIPTION) - String repoURL = '' - // helm repo url - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_CHART_DESCRIPTION) - String chart = '' - // chart name - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_VERSION_DESCRIPTION) - String version = '' - // chart version - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_NAMESPACE_DESCRIPTION) - String namespace = '' - // target namespace to deploy into - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_RELEASE_NAME_DESCRIPTION) - String releaseName = '' - // optional override; if empty => use name - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_VALUES_FILE_DESCRIPTION) - String valuesPath = '' - // optional local path or classpath resource, e.g. /foo/values.yaml - - @JsonPropertyDescription(CONTENT_HELM_RELEASE_VALUES_DESCRIPTION) - Map values = [:] - // optional inline values (merged with valuesFile) - } - } - - static class HelmConfig { - @JsonPropertyDescription(HELM_CONFIG_CHART_DESCRIPTION) - String chart = null - @JsonPropertyDescription(HELM_CONFIG_REPO_URL_DESCRIPTION) - String repoURL = null - @JsonPropertyDescription(HELM_CONFIG_VERSION_DESCRIPTION) - String version = null - } - - static class HelmConfigWithValues extends HelmConfig { - @JsonPropertyDescription(HELM_CONFIG_VALUES_DESCRIPTION) - Map values = [:] - } - - static class RegistrySchema { - Boolean internal = true - Boolean twoRegistries = false - - @Option(names = ['--registry'], description = REGISTRY_ENABLE_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_ENABLE_DESCRIPTION) - Boolean active = false - - @Option(names = ['--internal-registry-port'], description = REGISTRY_INTERNAL_PORT_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_INTERNAL_PORT_DESCRIPTION) - Integer internalPort = DEFAULT_REGISTRY_PORT - - @Option(names = ['--registry-url'], description = REGISTRY_URL_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_URL_DESCRIPTION) - String url = '' - - @Option(names = ['--registry-path'], description = REGISTRY_PATH_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_PATH_DESCRIPTION) - String path = '' - - @Option(names = ['--registry-username'], description = REGISTRY_USERNAME_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_USERNAME_DESCRIPTION) - String username = '' - - @Option(names = ['--registry-password'], description = REGISTRY_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_PASSWORD_DESCRIPTION) - String password = '' - - // Alternative: Use different registries, e.g. in air-gapped envs - // "Proxy" registry for 3rd party images, e.g. base images - @Option(names = ['--registry-proxy-url'], description = REGISTRY_PROXY_URL_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_PROXY_URL_DESCRIPTION) - String proxyUrl = '' - - @Option(names = ['--registry-proxy-path'], description = REGISTRY_PROXY_PATH_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_PROXY_PATH_DESCRIPTION) - String proxyPath = '' - - @Option(names = ['--registry-proxy-username'], description = REGISTRY_PROXY_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_PROXY_USERNAME_DESCRIPTION) - String proxyUsername = '' - - @Option(names = ['--registry-proxy-password'], description = 'Optional when --registry-proxy-url is set') - @JsonPropertyDescription(REGISTRY_PROXY_PASSWORD_DESCRIPTION) - String proxyPassword = '' - - // Alternative set of credentials for url, used only for image pull secrets - @Option(names = ['--registry-username-read-only'], description = REGISTRY_USERNAME_RO_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_USERNAME_RO_DESCRIPTION) - String readOnlyUsername = '' - - @Option(names = ['--registry-password-read-only'], description = REGISTRY_PASSWORD_RO_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_PASSWORD_RO_DESCRIPTION) - String readOnlyPassword = '' - - @Option(names = ['--create-image-pull-secrets'], description = REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION) - @JsonPropertyDescription(REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION) - Boolean createImagePullSecrets = false - - @Option(names = ['--registry-namespace'], description = REGISTRY_NAMESPACE) - @JsonPropertyDescription(REGISTRY_NAMESPACE) - String namespace = 'registry' - - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - HelmConfigWithValues helm = new HelmConfigWithValues(chart: 'docker-registry', - repoURL: 'https://twuni.github.io/docker-registry.helm', - version: '3.0.0') - - } - - static class JenkinsSchema { - Boolean internal = true - /* When installing via Docker we have to distinguish jenkins.url (which is a local IP address) from - the Jenkins URL used by SCMM. - - This is the URL configured in SCMM inside the Jenkins Plugin, e.g. at http://scmm.localhost/scm/admin/settings/jenkins - See addJenkinsConfig() and the comment at scmm.urlForJenkins */ - String urlForScm = '' - String ingress = '' - // Bash image used with internal Jenkins only - String internalBashImage = 'bash:5' - /* Docker client image, downloaded on internal Jenkins only - For updating, delete pvc jenkins-docker-client - When updating, we should not use too recent version, to not break support for LTS distros like debian - https://docs.docker.com/engine/install/debian/#os-requirements -> oldstable - For example: - $ curl -s https://download.docker.com/linux/debian/dists/bullseye/stable/binary-amd64/Packages | grep -EA5 'Package\: docker-ce$' | grep Version | sort | uniq | tail -n1 - Version: 5:27.1.1-1~debian.11~bullseye */ - String internalDockerClientVersion = '27.1.2' - - @Option(names = ['--jenkins'], description = JENKINS_ENABLE_DESCRIPTION) - @JsonPropertyDescription(JENKINS_ENABLE_DESCRIPTION) - Boolean active = false - - @Option(names = ['--jenkins-skip-restart'], description = JENKINS_SKIP_RESTART_DESCRIPTION) - @JsonPropertyDescription(JENKINS_SKIP_RESTART_DESCRIPTION) - Boolean skipRestart = false - - @Option(names = ['--jenkins-skip-plugins'], description = JENKINS_SKIP_PLUGINS_DESCRIPTION) - @JsonPropertyDescription(JENKINS_SKIP_PLUGINS_DESCRIPTION) - Boolean skipPlugins = false - - @Option(names = ['--jenkins-url'], description = JENKINS_URL_DESCRIPTION) - @JsonPropertyDescription(JENKINS_URL_DESCRIPTION) - String url = '' - - @Option(names = ['--jenkins-username'], description = JENKINS_USERNAME_DESCRIPTION) - @JsonPropertyDescription(JENKINS_USERNAME_DESCRIPTION) - String username = DEFAULT_ADMIN_USER - - @Option(names = ['--jenkins-password'], description = JENKINS_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(JENKINS_PASSWORD_DESCRIPTION) - String password = DEFAULT_ADMIN_PW - - @Option(names = ['--jenkins-metrics-username'], description = JENKINS_METRICS_USERNAME_DESCRIPTION) - @JsonPropertyDescription(JENKINS_METRICS_USERNAME_DESCRIPTION) - String metricsUsername = "metrics" - - @Option(names = ['--jenkins-metrics-password'], description = JENKINS_METRICS_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(JENKINS_METRICS_PASSWORD_DESCRIPTION) - String metricsPassword = "metrics" - - @Option(names = ['--maven-central-mirror'], description = MAVEN_CENTRAL_MIRROR_DESCRIPTION) - @JsonPropertyDescription(MAVEN_CENTRAL_MIRROR_DESCRIPTION) - String mavenCentralMirror = '' - - @Option(names = ["--jenkins-additional-envs"], description = JENKINS_ADDITIONAL_ENVS_DESCRIPTION, split = ",", required = false) - @JsonPropertyDescription(JENKINS_ADDITIONAL_ENVS_DESCRIPTION) - Map additionalEnvs = [:] - - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - HelmConfigWithValues helm = new HelmConfigWithValues(chart: 'jenkins', - repoURL: 'https://charts.jenkins.io', - version: '5.9.18') - - @Option(names = ['--jenkins-namespace'], description = JENKINS_NAMESPACE) - @JsonPropertyDescription(JENKINS_NAMESPACE) - String namespace = "jenkins" - - } - - static class ApplicationSchema { - Boolean runningInsideK8s = false - String namePrefixForEnvVars = '' - String internalKubernetesApiUrl = '' - String localHelmChartFolder = System.getenv('LOCAL_HELM_CHART_FOLDER') - - NamespaceSchema namespaces = new NamespaceSchema() - - @Option(names = ['--config-file'], description = CONFIG_FILE_DESCRIPTION, split = ',') - List configFiles = [] - - @Option(names = ['--config-map'], description = CONFIG_MAP_DESCRIPTION, split = ',') - List configMaps = [] - - @Option(names = ['-d', '--debug'], description = DEBUG_DESCRIPTION, scope = ScopeType.INHERIT) - Boolean debug - - @Option(names = ['-x', '--trace'], description = TRACE_DESCRIPTION, scope = ScopeType.INHERIT) - Boolean trace - - @Option(names = ['--output-config-file'], description = OUTPUT_CONFIG_FILE_DESCRIPTION, help = true) - Boolean outputConfigFile = false - - @Option(names = ["-v", "--version"], help = true, description = "Display version and license info") - Boolean versionInfoRequested = false - - // We define or own --version, so we need to define our own help param. - // The param itself is not used, "usageHelp = true" leads to hel being printed - @Option(names = ["-h", "--help"], usageHelp = true, description = "Display this help message") - Boolean usageHelpRequested = false - - @Option(names = ['--insecure'], description = INSECURE_DESCRIPTION) - @JsonPropertyDescription(INSECURE_DESCRIPTION) - Boolean insecure = false - - @Option(names = ['--openshift'], description = OPENSHIFT_DESCRIPTION) - @JsonPropertyDescription(OPENSHIFT_DESCRIPTION) - Boolean openshift = false - - @Option(names = ['--username'], description = USERNAME_DESCRIPTION) - @JsonPropertyDescription(USERNAME_DESCRIPTION) - String username = DEFAULT_ADMIN_USER - - @Option(names = ['--password'], description = PASSWORD_DESCRIPTION) - @JsonPropertyDescription(PASSWORD_DESCRIPTION) - String password = DEFAULT_ADMIN_PW - - @Option(names = ['-y', '--yes'], description = PIPE_YES_DESCRIPTION) - @JsonPropertyDescription(PIPE_YES_DESCRIPTION) - Boolean yes = false - - @Option(names = ['--name-prefix'], description = NAME_PREFIX_DESCRIPTION) - @JsonPropertyDescription(NAME_PREFIX_DESCRIPTION) - String namePrefix = '' - - @Option(names = ['--destroy'], description = DESTROY_DESCRIPTION) - @JsonPropertyDescription(DESTROY_DESCRIPTION) - Boolean destroy = false - - @Option(names = ['--pod-resources'], description = POD_RESOURCES_DESCRIPTION) - @JsonPropertyDescription(POD_RESOURCES_DESCRIPTION) - Boolean podResources = false - - @Option(names = ['--git-name'], description = GIT_NAME_DESCRIPTION) - @JsonPropertyDescription(GIT_NAME_DESCRIPTION) - String gitName = 'Cloudogu' - - @Option(names = ['--git-email'], description = GIT_EMAIL_DESCRIPTION) - @JsonPropertyDescription(GIT_EMAIL_DESCRIPTION) - String gitEmail = 'hello@cloudogu.com' - - @Option(names = ['--base-url'], description = BASE_URL_DESCRIPTION) - @JsonPropertyDescription(BASE_URL_DESCRIPTION) - String baseUrl = '' - - @Option(names = ['--url-separator-hyphen'], description = URL_SEPARATOR_HYPHEN_DESCRIPTION) - @JsonPropertyDescription(URL_SEPARATOR_HYPHEN_DESCRIPTION) - Boolean urlSeparatorHyphen = false - - @Option(names = ['--mirror-repos'], description = MIRROR_REPOS_DESCRIPTION) - @JsonPropertyDescription(MIRROR_REPOS_DESCRIPTION) - Boolean mirrorRepos = false - - @Option(names = ['--skip-crds'], description = SKIP_CRDS_DESCRIPTION) - @JsonPropertyDescription(SKIP_CRDS_DESCRIPTION) - Boolean skipCrds = false - - @Option(names = ['--namespace-isolation'], description = NAMESPACE_ISOLATION_DESCRIPTION) - @JsonPropertyDescription(NAMESPACE_ISOLATION_DESCRIPTION) - Boolean namespaceIsolation = false - - @Option(names = ['--netpols'], description = NETPOLS_DESCRIPTION) - @JsonPropertyDescription(NETPOLS_DESCRIPTION) - Boolean netpols = false - - @Option(names = ['--cluster-admin'], description = CLUSTER_ADMIN_DESCRIPTION) - @JsonPropertyDescription(CLUSTER_ADMIN_DESCRIPTION) - Boolean clusterAdmin = false - - @Option(names = ["-p", "--profile"], description = APPLICATION_PROFIL) - @JsonPropertyDescription(APPLICATION_PROFIL) - String profile - - @Option(names = ["--gop-namespace"], description = APPLICATION_GOP_NAMESPACE) - @JsonPropertyDescription(APPLICATION_GOP_NAMESPACE) - String gopNamespace = '' - - - @Option(names = ["-n","--namespace"], description = APPLICATION_NAMESPACE) - @JsonPropertyDescription(APPLICATION_NAMESPACE) - String namespace = '' - - - static class NamespaceSchema { - LinkedHashSet dedicatedNamespaces = new LinkedHashSet<>() - LinkedHashSet tenantNamespaces = new LinkedHashSet<>() - - LinkedHashSet getActiveNamespaces() { - return new LinkedHashSet<>(dedicatedNamespaces + tenantNamespaces) - } - } - - @JsonIgnore - String getTenantName() { - return namePrefix.replaceAll(/-$/, "") - } - } - - static class FeaturesSchema { - - @Mixin - @JsonPropertyDescription(ARGOCD_DESCRIPTION) - ArgoCDSchema argocd = new ArgoCDSchema() - - @Mixin - @JsonPropertyDescription(MAIL_DESCRIPTION) - MailSchema mail = new MailSchema() - - @Mixin - @JsonPropertyDescription(MONITORING_DESCRIPTION) - MonitoringSchema monitoring = new MonitoringSchema() - - @Mixin - @JsonPropertyDescription(SECRETS_DESCRIPTION) - SecretsSchema secrets = new SecretsSchema() - - @Mixin - @JsonPropertyDescription(INGRESS_DESCRIPTION) - IngressSchema ingress = new IngressSchema() - - @Mixin - @JsonPropertyDescription(CERTMANAGER_DESCRIPTION) - CertManagerSchema certManager = new CertManagerSchema() - } - - static class ArgoCDSchema { - Boolean configOnly = false - - @Option(names = ['--argocd'], description = ARGOCD_ENABLE_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_ENABLE_DESCRIPTION) - Boolean active = false - - @Option(names = ['--argocd-operator'], description = ARGOCD_OPERATOR_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_OPERATOR_DESCRIPTION) - Boolean operator = false - - @Option(names = ['--argocd-url'], description = ARGOCD_URL_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_URL_DESCRIPTION) - String url = '' - - @JsonPropertyDescription(ARGOCD_ENV_DESCRIPTION) - List> env - - @Option(names = ['--argocd-email-from'], description = ARGOCD_EMAIL_FROM_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_EMAIL_FROM_DESCRIPTION) - String emailFrom = 'argocd@example.org' - - @Option(names = ['--argocd-email-to-user'], description = ARGOCD_EMAIL_TO_USER_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_EMAIL_TO_USER_DESCRIPTION) - String emailToUser = 'app-team@example.org' - - @Option(names = ['--argocd-email-to-admin'], description = ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION) - String emailToAdmin = 'infra@example.org' - - @Option(names = ['--argocd-resource-inclusions-cluster'], description = ARGOCD_RESOURCE_INCLUSIONS_CLUSTER) - @JsonPropertyDescription(ARGOCD_RESOURCE_INCLUSIONS_CLUSTER) - String resourceInclusionsCluster = '' - - @Option(names = ['--argocd-namespace'], description = ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION) - @JsonPropertyDescription(ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION) - String namespace = 'argocd' - - @JsonPropertyDescription(HELM_CONFIG_VALUES_DESCRIPTION) - Map values = [:] - } - - static class MailSchema { - - Boolean active = false - - @Option(names = ['--smtp-address'], description = SMTP_ADDRESS_DESCRIPTION) - @JsonPropertyDescription(SMTP_ADDRESS_DESCRIPTION) - String smtpAddress = '' - - @Option(names = ['--smtp-port'], description = SMTP_PORT_DESCRIPTION) - @JsonPropertyDescription(SMTP_PORT_DESCRIPTION) - Integer smtpPort = null - - @Option(names = ['--smtp-user'], description = SMTP_USER_DESCRIPTION) - @JsonPropertyDescription(SMTP_USER_DESCRIPTION) - String smtpUser = '' - - @Option(names = ['--smtp-password'], description = SMTP_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(SMTP_PASSWORD_DESCRIPTION) - String smtpPassword = '' - } - - static class MonitoringSchema { - @Option(names = ['--metrics', '--monitoring'], description = MONITORING_ENABLE_DESCRIPTION) - @JsonPropertyDescription(MONITORING_ENABLE_DESCRIPTION) - Boolean active = false - - @Option(names = ['--grafana-url'], description = GRAFANA_URL_DESCRIPTION) - @JsonPropertyDescription(GRAFANA_URL_DESCRIPTION) - String grafanaUrl = '' - - @Option(names = ['--grafana-email-from'], description = GRAFANA_EMAIL_FROM_DESCRIPTION) - @JsonPropertyDescription(GRAFANA_EMAIL_FROM_DESCRIPTION) - String grafanaEmailFrom = 'grafana@example.org' - - @Option(names = ['--grafana-email-to'], description = GRAFANA_EMAIL_TO_DESCRIPTION) - @JsonPropertyDescription(GRAFANA_EMAIL_TO_DESCRIPTION) - String grafanaEmailTo = 'infra@example.org' - - @Mixin - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - @SuppressWarnings('GroovyAssignabilityCheck') - // Because of values - MonitoringHelmSchema helm = new MonitoringHelmSchema(chart: 'kube-prometheus-stack', - repoURL: 'https://prometheus-community.github.io/helm-charts', - /* When updating this make sure to also test if air-gapped mode still works */ - version: '80.2.2', - values: [:] // Otherwise values is null 🤷‍♂️ - ) - - @Option(names = ['--monitoring-namespace'], description = MONITORING_NAMESPACE) - @JsonPropertyDescription(MONITORING_NAMESPACE) - String namespace = 'monitoring' - - static class MonitoringHelmSchema extends HelmConfigWithValues { - @Option(names = ['--grafana-image'], description = GRAFANA_IMAGE_DESCRIPTION) - @JsonPropertyDescription(GRAFANA_IMAGE_DESCRIPTION) - String grafanaImage = '' - - @Option(names = ['--grafana-sidecar-image'], description = GRAFANA_SIDECAR_IMAGE_DESCRIPTION) - @JsonPropertyDescription(GRAFANA_SIDECAR_IMAGE_DESCRIPTION) - String grafanaSidecarImage = '' - - @Option(names = ['--prometheus-image'], description = PROMETHEUS_IMAGE_DESCRIPTION) - @JsonPropertyDescription(PROMETHEUS_IMAGE_DESCRIPTION) - String prometheusImage = '' - - @Option(names = ['--prometheus-operator-image'], description = PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION) - @JsonPropertyDescription(PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION) - String prometheusOperatorImage = '' - - @Option(names = ['--prometheus-config-reloader-image'], description = PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION) - @JsonPropertyDescription(PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION) - String prometheusConfigReloaderImage = '' - } - } - - static class SecretsSchema { - Boolean active = false - - @Mixin - @JsonPropertyDescription(ESO_DESCRIPTION) - ESOSchema externalSecrets = new ESOSchema() - - @Mixin - @JsonPropertyDescription(VAULT_DESCRIPTION) - VaultSchema vault = new VaultSchema() - - @Option(names = ['--secrets-namespace'], description = SECRETS_NAMESPACE) - @JsonPropertyDescription(SECRETS_NAMESPACE) - String namespace = 'secrets' - - static class ESOSchema { - - @Mixin - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - ESOHelmSchema helm = new ESOHelmSchema(chart: 'external-secrets', - repoURL: 'https://charts.external-secrets.io', - version: '0.9.16') - - static class ESOHelmSchema extends HelmConfigWithValues { - @Option(names = ['--external-secrets-image'], description = EXTERNAL_SECRETS_IMAGE_DESCRIPTION) - @JsonPropertyDescription(EXTERNAL_SECRETS_IMAGE_DESCRIPTION) - String image = '' - - @Option(names = ['--external-secrets-certcontroller-image'], description = EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION) - @JsonPropertyDescription(EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION) - String certControllerImage = '' - - @Option(names = ['--external-secrets-webhook-image'], description = EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION) - @JsonPropertyDescription(EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION) - String webhookImage = '' - } - } - - static class VaultSchema { - @Option(names = ['--vault'], description = VAULT_ENABLE_DESCRIPTION) - @JsonPropertyDescription(VAULT_ENABLE_DESCRIPTION) - VaultMode mode - - @Option(names = ['--vault-url'], description = VAULT_URL_DESCRIPTION) - @JsonPropertyDescription(VAULT_URL_DESCRIPTION) - String url = '' - - @Mixin - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - VaultHelmSchema helm = new VaultHelmSchema(chart: 'vault', - repoURL: 'https://helm.releases.hashicorp.com', - version: '0.25.0') - static class VaultHelmSchema extends HelmConfigWithValues { - @Option(names = ['--vault-image'], description = VAULT_IMAGE_DESCRIPTION) - @JsonPropertyDescription(VAULT_IMAGE_DESCRIPTION) - String image = '' - } - } - } - - static class IngressSchema { - - @Option(names = ['--ingress'], description = INGRESS_ENABLE_DESCRIPTION) - @JsonPropertyDescription(INGRESS_ENABLE_DESCRIPTION) - Boolean active = false - - @Mixin - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - IngressHelmSchema helm = new IngressHelmSchema(chart: 'traefik', - repoURL: 'https://traefik.github.io/charts', - version: '39.0.0') - static class IngressHelmSchema extends HelmConfigWithValues { - @Option(names = ['--ingress-image'], description = HELM_CONFIG_IMAGE_DESCRIPTION) - @JsonPropertyDescription(HELM_CONFIG_IMAGE_DESCRIPTION) - String image = '' - } - @Option(names = ['--ingress-namespace'], description = INGRESS_NAMESPACE) - @JsonPropertyDescription(INGRESS_NAMESPACE) - String ingressNamespace = 'ingress' - } - - static class CertManagerSchema { - @Option(names = ['--cert-manager'], description = CERTMANAGER_ENABLE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_ENABLE_DESCRIPTION) - Boolean active = false - - @Option(names = ['--cert-manager-issuer'], description = CERTMANAGER_ENABLE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_ENABLE_DESCRIPTION) - String issuer = 'cluster-selfsigned' - - @Option(names = ['--cert-manager-namespace'], description = CERTMANAGER_NAMESPACE) - @JsonPropertyDescription(CERTMANAGER_NAMESPACE) - String namespace = 'cert-manager' - - @Mixin - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - CertManagerHelmSchema helm = new CertManagerHelmSchema(chart: 'cert-manager', - repoURL: 'https://charts.jetstack.io', - version: '1.19.4') - - static class CertManagerHelmSchema extends HelmConfigWithValues { - - @Option(names = ['--cert-manager-image'], description = CERTMANAGER_IMAGE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_IMAGE_DESCRIPTION) - String image = '' - - @Option(names = ['--cert-manager-webhook-image'], description = CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION) - String webhookImage = '' - - @Option(names = ['--cert-manager-cainjector-image'], description = CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION) - String cainjectorImage = '' - - @Option(names = ['--cert-manager-acme-solver-image'], description = CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION) - String acmeSolverImage = '' - - @Option(names = ['--cert-manager-startup-api-check-image'], description = CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION) - @JsonPropertyDescription(CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION) - String startupAPICheckImage = '' - - } - } - - static enum ContentRepoType { - FOLDER_BASED, COPY, MIRROR - } - - static enum VaultMode { - dev, prod - } - - /** - * This defines, how customer repos will be updated. - * See {@link ConfigConstants#CONTENT_REPO_TARGET_OVERWRITE_MODE_DESCRIPTION} - */ - static enum OverwriteMode { - INIT, RESET, UPGRADE - } - - private static final ObjectMapper objectMapper = new ObjectMapper() - .registerModule(new SimpleModule().addSerializer(GString, new JsonSerializer() { - @Override - void serialize(GString value, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException { - jsonGenerator.writeString(value.toString()) - } - })) - - static Config fromMap(Map map) { - objectMapper.convertValue(map, Config) - } - - Map toMap() { - objectMapper.convertValue(this, Map) - } - - String toYaml(boolean includeInternals) { - createYamlMapper(includeInternals) - .writeValueAsString(this) - } - - private static YAMLMapper createYamlMapper(boolean includeInternals) { - if (!includeInternals) { - new YAMLMapper() - .registerModule(new SimpleModule().setSerializerModifier(new BeanSerializerModifier() { - @Override - List changeProperties(SerializationConfig serializationConfig, BeanDescription beanDesc, List beanProperties) { - beanProperties.findAll { writer -> writer.getAnnotation(JsonPropertyDescription) != null } - } - })) as YAMLMapper - } else { - new YAMLMapper() - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/ConfigConstants.groovy b/src/main/groovy/com/cloudogu/gitops/config/ConfigConstants.groovy deleted file mode 100644 index 330756676..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/ConfigConstants.groovy +++ /dev/null @@ -1,177 +0,0 @@ -package com.cloudogu.gitops.config - -interface ConfigConstants { - - public static final String BINARY_NAME = 'apply-ng' - public static final String APP_NAME = 'gitops-playground (GOP)' - public static final String APP_DESCRIPTION = 'CLI-tool to deploy gitops-playground.' - - // group registry - String REGISTRY_ENABLE_DESCRIPTION = 'Installs a simple cluster-local registry for demonstration purposes. Warning: Registry does not provide authentication!' - String REGISTRY_DESCRIPTION = 'Config parameters for Registry' - String REGISTRY_INTERNAL_PORT_DESCRIPTION = 'Port of registry registry. Ignored when a registry*url params are set' - String REGISTRY_URL_DESCRIPTION = 'The url of your external registry, used for pushing images' - String REGISTRY_PATH_DESCRIPTION = 'Optional when registry-url is set' - String REGISTRY_USERNAME_DESCRIPTION = 'Optional when registry-url is set' - String REGISTRY_PASSWORD_DESCRIPTION = 'Optional when registry-url is set' - - String REGISTRY_PROXY_URL_DESCRIPTION = 'The url of your proxy-registry. Used in pipelines to authorize pull base images. Use in conjunction with petclinic base image. Used in helm charts when create-image-pull-secrets is set. Use in conjunction with helm.*image fields.' - String REGISTRY_PROXY_PATH_DESCRIPTION = 'Optional when registry-proxy-url is set and the registry is running on a non root web path.' - String REGISTRY_PROXY_USERNAME_DESCRIPTION = 'Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set.' - String REGISTRY_PROXY_PASSWORD_DESCRIPTION = 'Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set.' - - String REGISTRY_USERNAME_RO_DESCRIPTION = 'Optional alternative username for registry-url with read-only permissions that is used when create-image-pull-secrets is set.' - String REGISTRY_PASSWORD_RO_DESCRIPTION = 'Optional alternative password for registry-url with read-only permissions that is used when create-image-pull-secrets is set.' - String REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION = 'Create image pull secrets for registry and proxy-registry for all GOP namespaces and helm charts. Uses proxy-username, read-only-username or registry-username (in this order). Use this if your cluster is not auto-provisioned with credentials for your private registries or if you configure individual helm images to be pulled from the proxy-registry that requires authentication.' - String REGISTRY_NAMESPACE = 'Optional defines the kubernetes namespace for registry.' - - String FEATURES_DESCRIPTION = 'Config parameters for features or tools' - - String CONTENT_DESCRIPTION = 'Config parameters for content, i.e. end-user or tenant applications as opposed to cluster-resources' - - // ContentLoader - String CONTENT_NAMESPACES_DESCRIPTION = 'Additional kubernetes namespaces. These are authorized to Argo CD, supplied with image pull secrets, monitored by prometheus, etc. Namespaces can be templates, e.g. ${config.application.namePrefix}staging' - String CONTENT_REPO_DESCRIPTION = "ContentLoader repos to push into target environment" - String CONTENT_REPO_URL_DESCRIPTION = "URL of the content repo. Mandatory for each type." - String CONTENT_REPO_PATH_DESCRIPTION = "Path within the content repo to process" - String CONTENT_REPO_REF_DESCRIPTION = "Reference for a specific branch, tag, or commit. Emtpy defaults to default branch of the repo. With type MIRROR: ref must not be a commit hash; Choosing a ref only mirrors the ref but does not delete other branches/tags!" - String CONTENT_REPO_TARGET_REF_DESCRIPTION = "Reference for a specific branch or tag in the target repo of a MIRROR or COPY repo. If ref is a tag, targetRef is treated as tag as well. Except: targetRef is full ref like refs/heads/my-branch or refs/tags/my-tag. Empty defaults to the source ref." - String CONTENT_REPO_CREDENTIALS_DESCRIPTION = "Credentials Object to authenticate against content repo. Allows using a K8s Secret" - String CONTENT_REPO_TEMPLATING_DESCRIPTION = "When true, template all files ending in .ftl within the repo" - String CONTENT_REPO_TYPE_DESCRIPTION = "ContentLoader Repos can either be:\ncopied (only the files, starting on ref, starting at path within the repo. Requires target)\n, mirrored (FORCE pushes ref or the whole git repo if no ref set). Requires target, does not allow path and template.)\nfolderBased (folder structure is interpreted as repos. That is, root folder becomes namespace in SCM, sub folders become repository names in SCM, files are copied. Requires target.)" - String CONTENT_REPO_TARGET_DESCRIPTION = "Target repo for the repository in the for of namespace/name. Must contain one slash to separate namespace from name." - String CONTENT_REPO_TARGET_OVERWRITE_MODE_DESCRIPTION = "This defines, how customer repos will be updated.\nINIT - push only if repo does not exist.\nRESET - delete all files after cloning source - files not in content are deleted\nUPGRADE - clone and copy - existing files will be overwritten, files not in content are kept. For type: MIRROR reset and upgrade have same result: in both cases source repo will be force pushed to target repo." - String CONTENT_REPO_CREATE_JENKINS_JOB_DESCRIPTION = "If true, creates a Jenkins job, if jenkinsfile exists in one of the content repo's branches." - String CONTENT_VARIABLES_DESCRIPTION = "Additional variables to use in custom templates." - String CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION = 'Enables the whitelist for statics in content templating' - String CONTENT_STATICSWHITELIST_DESCRIPTION = 'Whitelist for Statics freemarker is allowing in user templates' - String CONTENT_HELM_RELEASE_NAME_DESCRIPTION = "Logical name of the Helm release. Used as the feature folder name under 'apps/' and as default for 'releaseName' if not set." - - String CONTENT_HELM_RELEASE_REPO_URL_DESCRIPTION = "Helm repository URL to fetch the chart from. Use an HTTP(S) Helm repo (must provide an index.yaml) or an OCI registry URL (oci://...)." - String CONTENT_HELM_RELEASE_CHART_DESCRIPTION = "Helm chart name to install. For HTTP(S) repos this is the chart name from the repo index; for OCI this is the chart artifact name." - String CONTENT_HELM_RELEASE_VERSION_DESCRIPTION = "Chart version to deploy. Required for Helm charts in Argo CD. For HTTP(S) Helm repos you may use a SemVer range like '*' to always pick the newest version. For OCI registries, specify an explicit version/tag." - String CONTENT_HELM_RELEASE_NAMESPACE_DESCRIPTION = "Kubernetes namespace to deploy the release into." - String CONTENT_HELM_RELEASE_RELEASE_NAME_DESCRIPTION = "Helm release name. If empty, the value of 'name' is used." - String CONTENT_HELM_RELEASE_VALUES_FILE_DESCRIPTION = "Optional path to a YAML values file to load Helm values from.The file must be accessible locally on the machine running GOP. Inline 'values' will be merged on top (inline overrides file)." - String CONTENT_HELM_RELEASE_VALUES_DESCRIPTION = "Optional inline Helm values. These values are merged on top of 'valuesFile' (if set) and override keys from the file. Use this for small overrides without maintaining a separate file." - - // group jenkins - String JENKINS_ENABLE_DESCRIPTION = 'Installs Jenkins as CI server' - String JENKINS_SKIP_RESTART_DESCRIPTION = 'Skips restarting Jenkins after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.' - String JENKINS_SKIP_PLUGINS_DESCRIPTION = 'Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.' - String JENKINS_DESCRIPTION = 'Config parameters for Jenkins CI/CD Pipeline Server' - String JENKINS_URL_DESCRIPTION = 'The url of your external jenkins' - String JENKINS_USERNAME_DESCRIPTION = 'Mandatory when jenkins-url is set' - String JENKINS_PASSWORD_DESCRIPTION = 'Mandatory when jenkins-url is set' - String JENKINS_METRICS_USERNAME_DESCRIPTION = 'Mandatory when jenkins-url is set and monitoring enabled' - String JENKINS_METRICS_PASSWORD_DESCRIPTION = 'Mandatory when jenkins-url is set and monitoring enabled' - String MAVEN_CENTRAL_MIRROR_DESCRIPTION = 'URL for maven mirror, used by applications built in Jenkins' - String JENKINS_ADDITIONAL_ENVS_DESCRIPTION = 'Set additional environments to Jenkins' - String JENKINS_NAMESPACE = 'Optional defines the kubernetes namespace for Jenkins.' - - // group scmm - String SCM_DESCRIPTION = 'Config parameters for Scm' - String GIT_NAME_DESCRIPTION = 'Sets git author and committer name used for initial commits' - String GIT_EMAIL_DESCRIPTION = 'Sets git author and committer email used for initial commits' - - //MutliTentant - String MULTITENANT_DESCRIPTION = 'Multi Tenant Configs' - - // group remote - String INSECURE_DESCRIPTION = 'Sets insecure-mode in cURL which skips cert validation' - - // group tool configuration - String APPLICATION_DESCRIPTION = 'Application configuration parameter for GOP' - String GRAFANA_IMAGE_DESCRIPTION = 'Sets image for grafana' - String GRAFANA_SIDECAR_IMAGE_DESCRIPTION = 'Sets image for grafana\'s sidecar' - String PROMETHEUS_IMAGE_DESCRIPTION = 'Sets image for prometheus' - String PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION = 'Sets image for prometheus-operator' - String PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION = 'Sets image for prometheus-operator\'s config-reloader' - String EXTERNAL_SECRETS_IMAGE_DESCRIPTION = 'Sets image for external secrets operator' - String EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION = 'Sets image for external secrets operator\'s controller' - String EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION = 'Sets image for external secrets operator\'s webhook' - String VAULT_IMAGE_DESCRIPTION = 'Sets image for vault' - String BASE_URL_DESCRIPTION = 'the external base url (TLD) for all tools, e.g. https://example.com or http://localhost:8080. The individual -url params for argocd, grafana and vault take precedence.' - String URL_SEPARATOR_HYPHEN_DESCRIPTION = 'Use hyphens instead of dots to separate application name from base-url' - String SKIP_CRDS_DESCRIPTION = 'Skip installation of CRDs. This requires prior installation of CRDs' - String NAMESPACE_ISOLATION_DESCRIPTION = 'Configure tools to explicitly work with the given namespaces only, and not cluster-wide. This way GOP can be installed without having cluster-admin permissions.' - String MIRROR_REPOS_DESCRIPTION = 'Changes the sources of deployed tools so they are not pulled from the internet, but are pulled from git and work in air-gapped environments.' - String NETPOLS_DESCRIPTION = 'Sets Network Policies' - String CLUSTER_ADMIN_DESCRIPTION = 'Binds ArgoCD controllers to cluster-admin ClusterRole' - String OPENSHIFT_DESCRIPTION = 'When set, openshift specific resources and configurations are applied' - String APPLICATION_PROFIL = 'Use predefined profile (full, only-argocd, operator-mandants aso.)' - String APPLICATION_GOP_NAMESPACE = 'If set, GOP stores specific information in this namespace.' - String APPLICATION_NAMESPACE = 'If set, GOP uses the same Kubernetes namespace for all tools and examples. Attention! Only use for test purposes.' - // group metrics - String MONITORING_DESCRIPTION = 'Config parameters for the Monitoring system (prometheus)' - String MONITORING_ENABLE_DESCRIPTION = 'Installs the Kube-Prometheus-Stack. This includes Prometheus, the Prometheus operator, Grafana and some extra resources' - String MONITORING_NAMESPACE = 'Optional defines the kubernetes namespace for monitoring.' - String GRAFANA_URL_DESCRIPTION = 'Sets url for grafana' - String GRAFANA_EMAIL_FROM_DESCRIPTION = 'Notifications, define grafana alerts sender email address' - String GRAFANA_EMAIL_TO_DESCRIPTION = 'Notifications, define grafana alerts recipient email address' - - // group vault / secrets - String SECRETS_DESCRIPTION = 'Config parameters for the secrets management' - String ESO_DESCRIPTION = 'Config parameters for the external secrets operator' - String VAULT_DESCRIPTION = 'Config parameters for the secrets-vault' - String VAULT_ENABLE_DESCRIPTION = "Installs Hashicorp vault and the external secrets operator. Possible values: dev, prod." - String VAULT_URL_DESCRIPTION = 'Sets url for vault ui' - String SECRETS_NAMESPACE = 'Optional defines the kubernetes namespace for secrets.' - - // group external Mailserver - String MAIL_DESCRIPTION = 'Config parameters for mail servers' - String SMTP_ADDRESS_DESCRIPTION = 'Sets smtp port of external Mailserver' - String SMTP_PORT_DESCRIPTION = 'Sets smtp port of external Mailserver' - String SMTP_USER_DESCRIPTION = 'Sets smtp username for external Mailserver' - String SMTP_PASSWORD_DESCRIPTION = 'Sets smtp password of external Mailserver' - - // group debug - String DEBUG_DESCRIPTION = 'Debug output' - String TRACE_DESCRIPTION = 'Debug + Show each command executed (set -x)' - - // group configuration - String USERNAME_DESCRIPTION = 'Set initial admin username' - String PASSWORD_DESCRIPTION = 'Set initial admin passwords' - String PIPE_YES_DESCRIPTION = 'Skip confirmation' - String NAME_PREFIX_DESCRIPTION = 'Set name-prefix for repos, jobs, namespaces' - String DESTROY_DESCRIPTION = 'Unroll playground' - String CONFIG_FILE_DESCRIPTION = 'Config file for the application' - String CONFIG_MAP_DESCRIPTION = 'Kubernetes configuration map. Should contain a key `config.yaml`.' - String OUTPUT_CONFIG_FILE_DESCRIPTION = 'Output current config as config file as much as possible' - String POD_RESOURCES_DESCRIPTION = 'Write kubernetes resource requests and limits on each pod' - - // group ArgoCD Operator - String ARGOCD_DESCRIPTION = 'Config Parameter for the ArgoCD Operator' - String ARGOCD_ENABLE_DESCRIPTION = 'Install ArgoCD' - String ARGOCD_URL_DESCRIPTION = 'The URL where argocd is accessible. It has to be the full URL with http:// or https://' - String ARGOCD_EMAIL_FROM_DESCRIPTION = 'Notifications, define Argo CD sender email address' - String ARGOCD_EMAIL_TO_USER_DESCRIPTION = 'Notifications, define Argo CD user / app-team recipient email address' - String ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION = 'Notifications, define Argo CD admin recipient email address' - String ARGOCD_OPERATOR_DESCRIPTION = 'Install ArgoCD via an already running ArgoCD Operator' - String ARGOCD_ENV_DESCRIPTION = 'Pass a list of env vars to Argo CD components. Currently only works with operator' - String ARGOCD_RESOURCE_INCLUSIONS_CLUSTER = 'Internal Kubernetes API Server URL https://IP:PORT (kubernetes.default.svc). Needed in argocd-operator resourceInclusions. Use this parameter if argocd.operator=true and NOT running inside a Pod (remote mode). Full URL needed, for example: https://100.125.0.1:443' - String ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION = 'Defines the kubernetes namespace for ArgoCD' - - // group ingress-class - String INGRESS_DESCRIPTION = 'Config parameters for the Ingress Controller' - String INGRESS_ENABLE_DESCRIPTION = 'Sets and enables Ingress Controller' - String INGRESS_NAMESPACE = 'Optional defines the kubernetes namespace for Ingress Controller' - - // group CERTMANAGER - String CERTMANAGER_DESCRIPTION = 'Config parameters for the Cert Manager' - String CERTMANAGER_ENABLE_DESCRIPTION = 'Sets and enables Cert Manager' - String CERTMANAGER_IMAGE_DESCRIPTION = 'Sets image for Cert Manager' - String CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION = 'Sets webhook Image for Cert Manager' - String CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION = 'Sets cainjector Image for Cert Manager' - String CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION = 'Sets acmeSolver Image for Cert Manager' - String CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION = 'Sets startupAPICheck Image for Cert Manager' - String CERTMANAGER_NAMESPACE = 'Optional defines the kubernetes namespace for Cert Manager' - - // group helm - String HELM_CONFIG_DESCRIPTION = 'Common Config parameters for the Helm package manager: Name of Chart (chart), URl of Helm-Repository (repoURL) and Chart Version (version). Note: These config is intended to obtain the chart from a different source (e.g. in air-gapped envs), not to use a different version of a helm chart. Using a different helm chart or version to the one used in the GOP version will likely cause errors.' - String HELM_CONFIG_CHART_DESCRIPTION = 'Name of the Helm chart' - String HELM_CONFIG_REPO_URL_DESCRIPTION = 'Repository url from which the Helm chart should be obtained' - String HELM_CONFIG_VERSION_DESCRIPTION = 'The version of the Helm chart to be installed' - String HELM_CONFIG_IMAGE_DESCRIPTION = 'The image of the Helm chart to be installed' - String HELM_CONFIG_VALUES_DESCRIPTION = 'Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration' -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/Credentials.groovy b/src/main/groovy/com/cloudogu/gitops/config/Credentials.groovy deleted file mode 100644 index a7933e5e8..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/Credentials.groovy +++ /dev/null @@ -1,44 +0,0 @@ -package com.cloudogu.gitops.config - -import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_CREDENTIALS_DESCRIPTION - -import groovy.transform.ToString - -import com.fasterxml.jackson.annotation.JsonIgnore -import com.fasterxml.jackson.annotation.JsonPropertyDescription - -@ToString -class Credentials { - - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - String username - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - @JsonIgnore - String password - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - String secretNamespace - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - String secretName - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - String usernameKey = 'username' - @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) - String passwordKey = 'password' - - Credentials() {} - - Credentials(String username, String password, String secretName = '', String secretNamespace = '', String usernameKey = "username", String passwordKey = 'password') { - this.username = username - this.password = password - this.secretNamespace = secretNamespace - this.secretName = secretName - this.usernameKey = usernameKey - this.passwordKey = passwordKey - } - - Credentials(Credentials unsafeCredentials) { - this.secretNamespace = unsafeCredentials.secretNamespace - this.secretName = unsafeCredentials.secretName - this.usernameKey = unsafeCredentials.usernameKey - this.passwordKey = unsafeCredentials.passwordKey - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/MultiTenantSchema.groovy b/src/main/groovy/com/cloudogu/gitops/config/MultiTenantSchema.groovy deleted file mode 100644 index 31ce19b0e..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/MultiTenantSchema.groovy +++ /dev/null @@ -1,40 +0,0 @@ -package com.cloudogu.gitops.config - -import com.cloudogu.gitops.config.scm.ScmCentralSchema.GitlabCentralConfig -import com.cloudogu.gitops.config.scm.ScmCentralSchema.ScmManagerCentralConfig -import com.cloudogu.gitops.config.scm.util.ScmProviderType -import com.fasterxml.jackson.annotation.JsonPropertyDescription -import picocli.CommandLine.Mixin -import picocli.CommandLine.Option - -class MultiTenantSchema { - - static final String SCM_PROVIDER_TYPE_DESCRIPTION = 'The SCM provider type. Possible values: SCM_MANAGER, GITLAB' - static final String GITLAB_CONFIG_DESCRIPTION = 'Config for GITLAB' - static final String SCMM_CONFIG_DESCRIPTION = 'Config for GITLAB' - static final String CENTRAL_ARGOCD_NAMESPACE_DESCRIPTION = 'Namespace for the centralized Argocd' - static final String CENTRAL_USEDEDICATED_DESCRIPTION = 'Toggles the Dedicated Instances Mode. See docs for more info' - - @Option(names = ['--central-scm-provider'], - description = SCM_PROVIDER_TYPE_DESCRIPTION, - defaultValue = "SCM_MANAGER") - @JsonPropertyDescription(SCM_PROVIDER_TYPE_DESCRIPTION) - ScmProviderType scmProviderType = ScmProviderType.SCM_MANAGER - - @JsonPropertyDescription(GITLAB_CONFIG_DESCRIPTION) - @Mixin - GitlabCentralConfig gitlab - - @JsonPropertyDescription(SCMM_CONFIG_DESCRIPTION) - @Mixin - ScmManagerCentralConfig scmManager - - @Option(names = ['--central-argocd-namespace'], description = CENTRAL_ARGOCD_NAMESPACE_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_ARGOCD_NAMESPACE_DESCRIPTION) - String centralArgocdNamespace = 'argocd' - - @Option(names = ['--dedicated-instance'], description = CENTRAL_USEDEDICATED_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_USEDEDICATED_DESCRIPTION) - Boolean useDedicatedInstance = false - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/schema/JsonSchemaGenerator.groovy b/src/main/groovy/com/cloudogu/gitops/config/schema/JsonSchemaGenerator.groovy deleted file mode 100644 index 9c8ff68f6..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/schema/JsonSchemaGenerator.groovy +++ /dev/null @@ -1,36 +0,0 @@ -package com.cloudogu.gitops.config.schema - -import com.cloudogu.gitops.config.Config - -import jakarta.inject.Singleton - -import com.fasterxml.jackson.annotation.JsonPropertyDescription -import com.github.victools.jsonschema.generator.* -import com.github.victools.jsonschema.module.jackson.JacksonModule -import tools.jackson.databind.node.ObjectNode - -@Singleton -class JsonSchemaGenerator { - static ObjectNode createSchema() { - SchemaGeneratorConfigBuilder configBuilder = - new SchemaGeneratorConfigBuilder(SchemaVersion.DRAFT_2020_12, OptionPreset.PLAIN_JSON) - // Make the schema strict: Only allow our fields, warn when additional fields are passed - .with(Option.FORBIDDEN_ADDITIONAL_PROPERTIES_BY_DEFAULT) - // Exception to the above: For Maps allow additional fields. - // We use this to allow inline helm values without having to validate them - .with(Option.MAP_VALUES_AS_ADDITIONAL_PROPERTIES) - // All fields can be set to null to use the default - .with(Option.NULLABLE_FIELDS_BY_DEFAULT) - .with(new JacksonModule(/* no options for now */)) - // Apply the rule to include only fields with @JsonProperty annotation - configBuilder.forFields() - .withIgnoreCheck((FieldScope field) -> { - // Only include fields that are annotated with @JsonProperty - return field.getAnnotation(JsonPropertyDescription) == null - }) - - SchemaGenerator generator = new SchemaGenerator(configBuilder.build()) - - return generator.generateSchema(Config) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/schema/JsonSchemaValidator.groovy b/src/main/groovy/com/cloudogu/gitops/config/schema/JsonSchemaValidator.groovy deleted file mode 100644 index 853619fc1..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/schema/JsonSchemaValidator.groovy +++ /dev/null @@ -1,29 +0,0 @@ -package com.cloudogu.gitops.config.schema - -import groovy.util.logging.Slf4j - -import com.networknt.schema.Schema -import com.networknt.schema.SchemaRegistry -import tools.jackson.databind.JsonNode -import tools.jackson.databind.ObjectMapper - -@Slf4j -class JsonSchemaValidator { - - private static ObjectMapper objectMapper = new ObjectMapper() - private static SchemaRegistry schemaRegistry = SchemaRegistry.builder().build() - - static void validate(Map yaml) { - def json = objectMapper.convertValue(yaml, JsonNode) - def schemaNode = JsonSchemaGenerator.createSchema() - Schema schema = schemaRegistry.getSchema(schemaNode) - - log.debug("yaml configuration converted to json for validate {}", json) - - def validationMessages = schema.validate(json) - - if (!validationMessages.isEmpty()) { - throw new RuntimeException("Config file invalid: " + validationMessages.join("\n")) - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/schema/Schema.groovy b/src/main/groovy/com/cloudogu/gitops/config/schema/Schema.groovy deleted file mode 100644 index e69de29bb..000000000 diff --git a/src/main/groovy/com/cloudogu/gitops/config/scm/ScmCentralSchema.groovy b/src/main/groovy/com/cloudogu/gitops/config/scm/ScmCentralSchema.groovy deleted file mode 100644 index da79b06e6..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/scm/ScmCentralSchema.groovy +++ /dev/null @@ -1,90 +0,0 @@ -package com.cloudogu.gitops.config.scm - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.config.scm.util.GitlabConfig -import com.cloudogu.gitops.config.scm.util.ScmManagerConfig -import com.fasterxml.jackson.annotation.JsonPropertyDescription -import picocli.CommandLine.Option - -class ScmCentralSchema { - - static class GitlabCentralConfig implements GitlabConfig { - - public static final String CENTRAL_GITLAB_URL_DESCRIPTION = "URL for external Gitlab" - public static final String CENTRAL_GITLAB_USERNAME_DESCRIPTION = "GitLab username for API access. Must be 'oauth2' when using Personal Access Token (PAT) authentication" - public static final String CENTRAL_GITLAB_PASSWORD_DESCRIPTION = "Password for SCM Manager authentication" - public static final String CENTRAL_GITLAB_PARENTGROUP_ID_DESCRIPTION = "Main Group for Gitlab where the GOP creates it's groups/repos" - - // Only supports external Gitlab for now - @Option(names = ['--central-gitlab-url'], description = CENTRAL_GITLAB_URL_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_GITLAB_URL_DESCRIPTION) - String url = 'https://gitlab.com/' - - @Option(names = ['--central-gitlab-username'], description = CENTRAL_GITLAB_USERNAME_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_GITLAB_USERNAME_DESCRIPTION) - String username = 'oauth2.0' - - @Option(names = ['--central-gitlab-token'], description = CENTRAL_GITLAB_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_GITLAB_PASSWORD_DESCRIPTION) - String password = '' - - @Option(names = ['--central-gitlab-group-id'], description = CENTRAL_GITLAB_PARENTGROUP_ID_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_GITLAB_PARENTGROUP_ID_DESCRIPTION) - String parentGroupId = '' - - Credentials getCredentials() { - return new Credentials(username, password) - } - - String gitOpsUsername = '' - String defaultVisibility = '' - } - - static class ScmManagerCentralConfig implements ScmManagerConfig { - - public static final String CENTRAL_SCMM_INTERNAL_DESCRIPTION = 'SCM for Central Management is running on the same cluster, so k8s internal URLs can be used for access' - public static final String CENTRAL_SCMM_URL_DESCRIPTION = 'URL for the centralized Management Repo' - public static final String CENTRAL_SCMM_USERNAME_DESCRIPTION = 'CENTRAL SCMM username' - public static final String CENTRAL_SCMM_PASSWORD_DESCRIPTION = 'CENTRAL SCMM password' - public static final String CENTRAL_SCMM_PATH_DESCRIPTION = 'Root path for SCM Manager. In SCM-Manager it is always "repo"' - public static final String CENTRAL_SCMM_NAMESPACE_DESCRIPTION = 'Namespace where to find the Central SCMM' - - @Option(names = ['--central-scmm-internal'], description = CENTRAL_SCMM_INTERNAL_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_SCMM_INTERNAL_DESCRIPTION) - Boolean internal = false - - @Option(names = ['--central-scmm-url'], description = CENTRAL_SCMM_URL_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_SCMM_URL_DESCRIPTION) - String url = '' - - @Option(names = ['--central-scmm-username'], description = CENTRAL_SCMM_USERNAME_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_SCMM_USERNAME_DESCRIPTION) - String username = '' - - @Option(names = ['--central-scmm-password'], description = CENTRAL_SCMM_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_SCMM_PASSWORD_DESCRIPTION) - String password = '' - - @Option(names = ['--central-scmm-namespace'], description = CENTRAL_SCMM_NAMESPACE_DESCRIPTION) - @JsonPropertyDescription(CENTRAL_SCMM_NAMESPACE_DESCRIPTION) - String namespace = 'scm-manager' - - @Override - String getIngress() { - return null //Needed for setup - } - - @Override - Config.HelmConfigWithValues getHelm() { - return null //Needed for setup - } - - Credentials getCredentials() { - return new Credentials(username, password) - } - - String gitOpsUsername = '' - - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/scm/ScmTenantSchema.groovy b/src/main/groovy/com/cloudogu/gitops/config/scm/ScmTenantSchema.groovy deleted file mode 100644 index 08b3399c9..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/scm/ScmTenantSchema.groovy +++ /dev/null @@ -1,156 +0,0 @@ -package com.cloudogu.gitops.config.scm - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.config.scm.util.GitlabConfig -import com.cloudogu.gitops.config.scm.util.ScmManagerConfig -import com.cloudogu.gitops.config.scm.util.ScmProviderType -import com.cloudogu.gitops.utils.NetworkingUtils -import com.fasterxml.jackson.annotation.JsonIgnore -import com.fasterxml.jackson.annotation.JsonMerge -import com.fasterxml.jackson.annotation.JsonPropertyDescription -import picocli.CommandLine.Mixin -import picocli.CommandLine.Option - -import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_DESCRIPTION - -class ScmTenantSchema { - - static final String GITLAB_CONFIG_DESCRIPTION = 'Config for GITLAB' - static final String SCMM_CONFIG_DESCRIPTION = 'Config for GITLAB' - static final String SCM_PROVIDER_TYPE_DESCRIPTION = 'The SCM provider type. Possible values: SCM_MANAGER, GITLAB' - static final String GITOPSUSERNAME_DESCRIPTION = 'Username for the Gitops User' - - @Option(names = ['--scm-provider'], - description = SCM_PROVIDER_TYPE_DESCRIPTION, - defaultValue = "SCM_MANAGER") - @JsonPropertyDescription(SCM_PROVIDER_TYPE_DESCRIPTION) - ScmProviderType scmProviderType = ScmProviderType.SCM_MANAGER - - @JsonPropertyDescription(GITLAB_CONFIG_DESCRIPTION) - @Mixin - GitlabTenantConfig gitlab - - @JsonPropertyDescription(SCMM_CONFIG_DESCRIPTION) - @Mixin - ScmManagerTenantConfig scmManager - - @JsonIgnore - Boolean internal = { -> return (gitlab.internal || scmManager.internal) - } - - static class GitlabTenantConfig implements GitlabConfig { - - static final String GITLAB_INTERNAL_DESCRIPTION = 'True if Gitlab is running in the same K8s cluster. For now we only support access by external URL' - static final String GITLAB_URL_DESCRIPTION = "Base URL for the Gitlab instance" - static final String GITLAB_USERNAME_DESCRIPTION = 'Defaults to: oauth2.0 when PAT token is given.' - static final String GITLAB_TOKEN_DESCRIPTION = 'PAT Token for the account. Needs read/write repo permissions. See docs for mor information' - static final String GITLAB_PARENT_GROUP_ID = 'Number for the Gitlab Group where the repos and subgroups should be created' - - @JsonPropertyDescription(GITLAB_INTERNAL_DESCRIPTION) - Boolean internal = false - - @Option(names = ['--gitlab-url'], description = GITLAB_URL_DESCRIPTION) - @JsonPropertyDescription(GITLAB_URL_DESCRIPTION) - String url - - @Option(names = ['--gitlab-username'], description = GITLAB_USERNAME_DESCRIPTION) - @JsonPropertyDescription(GITLAB_USERNAME_DESCRIPTION) - String username = 'oauth2.0' - - @Option(names = ['--gitlab-token'], description = GITLAB_TOKEN_DESCRIPTION) - @JsonPropertyDescription(GITLAB_TOKEN_DESCRIPTION) - String password - - @Option(names = ['--gitlab-group-id'], description = GITLAB_PARENT_GROUP_ID) - @JsonPropertyDescription(GITLAB_PARENT_GROUP_ID) - String parentGroupId = '' - - @JsonIgnore - Credentials getCredentials() { - return new Credentials(username, password) - } - - @JsonPropertyDescription(GITOPSUSERNAME_DESCRIPTION) - String gitOpsUsername = '' - String defaultVisibility = '' - - } - - static class ScmManagerTenantConfig implements ScmManagerConfig { - - static final String SCMM_SKIP_RESTART_DESCRIPTION = 'Skips restarting SCM-Manager after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.\'' - static final String SCMM_SKIP_PLUGINS_DESCRIPTION = 'Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.' - static final String SCMM_URL_DESCRIPTION = 'The host of your external scm-manager' - static final String SCMM_USERNAME_DESCRIPTION = 'Mandatory when scmm-url is set' - static final String SCMM_PASSWORD_DESCRIPTION = 'Mandatory when scmm-url is set' - static final String SCMM_ROOT_PATH_DESCRIPTION = 'Sets the root path for the Git Repositories. In SCM-Manager it is always "repo"' - static final String SCMM_NAMESPACE_DESCRIPTION = 'Namespace where SCM-Manager should run' - - Boolean internal = true - - @Option(names = ['--scmm-url'], description = SCMM_URL_DESCRIPTION) - @JsonPropertyDescription(SCMM_URL_DESCRIPTION) - String url = '' - - @Option(names = ['--scmm-namespace'], description = SCMM_NAMESPACE_DESCRIPTION) - @JsonPropertyDescription(SCMM_NAMESPACE_DESCRIPTION) - String namespace = 'scm-manager' - - @Option(names = ['--scmm-username'], description = SCMM_USERNAME_DESCRIPTION) - @JsonPropertyDescription(SCMM_USERNAME_DESCRIPTION) - String username = Config.DEFAULT_ADMIN_USER - - @Option(names = ['--scmm-password'], description = SCMM_PASSWORD_DESCRIPTION) - @JsonPropertyDescription(SCMM_PASSWORD_DESCRIPTION) - String password = Config.DEFAULT_ADMIN_PW - - @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) - @JsonMerge - Config.HelmConfigWithValues helm = new Config.HelmConfigWithValues(chart: 'scm-manager', - repoURL: 'https://packages.scm-manager.org/repository/helm-v2-releases/', - version: '3.11.6', - values: [:]) - - /* When installing from via Docker we have to distinguish scmm.url (which is a local IP address) from - the SCMM URL used by jenkins. - - This is necessary to make the build on push feature (webhooks from SCMM to Jenkins that trigger builds) work - in k3d. - The webhook contains repository URLs that start with the "Base URL" Setting of SCMM. - Jenkins checks these repo URLs and triggers all builds that match repo URLs. - - This value is set as "Base URL" in SCMM Settings and in Jenkins Job. - - See ApplicationConfigurator.addScmmConfig() and the comment at jenkins.urlForScmm */ - - String urlForJenkins = '' - - @JsonIgnore - String getHost() { - return NetworkingUtils.getHost(url) - } - - @JsonIgnore - String getProtocol() { - return NetworkingUtils.getProtocol(url) - } - String ingress = '' - - @Option(names = ['--scmm-skip-restart'], description = SCMM_SKIP_RESTART_DESCRIPTION) - @JsonPropertyDescription(SCMM_SKIP_RESTART_DESCRIPTION) - Boolean skipRestart = false - - @Option(names = ['--scmm-skip-plugins'], description = SCMM_SKIP_PLUGINS_DESCRIPTION) - @JsonPropertyDescription(SCMM_SKIP_PLUGINS_DESCRIPTION) - Boolean skipPlugins = false - - @JsonPropertyDescription(GITOPSUSERNAME_DESCRIPTION) - String gitOpsUsername = '' - - @JsonIgnore - Credentials getCredentials() { - return new Credentials(username, password) - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/scm/util/GitlabConfig.groovy b/src/main/groovy/com/cloudogu/gitops/config/scm/util/GitlabConfig.groovy deleted file mode 100644 index 32588a473..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/scm/util/GitlabConfig.groovy +++ /dev/null @@ -1,15 +0,0 @@ -package com.cloudogu.gitops.config.scm.util - -import com.cloudogu.gitops.config.Credentials - -interface GitlabConfig { - String getUrl() - - String getParentGroupId() - - String getDefaultVisibility() - - String getGitOpsUsername() - - Credentials getCredentials() -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/scm/util/ScmManagerConfig.groovy b/src/main/groovy/com/cloudogu/gitops/config/scm/util/ScmManagerConfig.groovy deleted file mode 100644 index c34404835..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/scm/util/ScmManagerConfig.groovy +++ /dev/null @@ -1,24 +0,0 @@ -package com.cloudogu.gitops.config.scm.util - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials - -interface ScmManagerConfig { - Boolean getInternal() - - String getUrl() - - String getUsername() - - String getPassword() - - String getNamespace() - - String getIngress() - - Config.HelmConfigWithValues getHelm() - - String getGitOpsUsername() - - Credentials getCredentials() -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/config/scm/util/ScmProviderType.groovy b/src/main/groovy/com/cloudogu/gitops/config/scm/util/ScmProviderType.groovy deleted file mode 100644 index 89566786c..000000000 --- a/src/main/groovy/com/cloudogu/gitops/config/scm/util/ScmProviderType.groovy +++ /dev/null @@ -1,6 +0,0 @@ -package com.cloudogu.gitops.config.scm.util - -enum ScmProviderType { - GITLAB, - SCM_MANAGER -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/dependencyinjection/HttpClientFactory.groovy b/src/main/groovy/com/cloudogu/gitops/dependencyinjection/HttpClientFactory.groovy deleted file mode 100644 index e80fe97c4..000000000 --- a/src/main/groovy/com/cloudogu/gitops/dependencyinjection/HttpClientFactory.groovy +++ /dev/null @@ -1,100 +0,0 @@ -package com.cloudogu.gitops.dependencyinjection - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.dependencyinjection.okhttp.RetryInterceptor -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.AuthorizationInterceptor -import groovy.transform.TupleConstructor -import io.micronaut.context.annotation.Factory -import jakarta.inject.Named -import jakarta.inject.Singleton -import okhttp3.JavaNetCookieJar -import okhttp3.OkHttpClient -import okhttp3.logging.HttpLoggingInterceptor -import org.jetbrains.annotations.NotNull -import org.slf4j.LoggerFactory - -import javax.net.ssl.HostnameVerifier -import javax.net.ssl.SSLContext -import javax.net.ssl.SSLSocketFactory -import javax.net.ssl.X509TrustManager -import java.security.SecureRandom -import java.security.cert.CertificateException -import java.security.cert.X509Certificate - -@Factory -class HttpClientFactory { - - static OkHttpClient buildOkHttpClient(Credentials credentials, Boolean isInsecure) { - def builder = new OkHttpClient.Builder() - .addInterceptor(new AuthorizationInterceptor(credentials.username, credentials.password)) - .addInterceptor(createLoggingInterceptor()) - .addInterceptor(new RetryInterceptor()) - - if (isInsecure) { - def context = insecureSslContext() - builder.sslSocketFactory(context.socketFactory, context.trustManager) - } - - builder.hostnameVerifier({ hostname, session -> true } as HostnameVerifier) - - return builder.build() - } - - @Singleton - @Named("jenkins") - OkHttpClient okHttpClientJenkins(Config config) { - def builder = new OkHttpClient.Builder() - .cookieJar(new JavaNetCookieJar(new CookieManager())) - .addInterceptor(createLoggingInterceptor()) - .addInterceptor(new RetryInterceptor()) - - if (config.application.insecure) { - def context = insecureSslContext() - builder.sslSocketFactory(context.socketFactory, context.trustManager) - } - - return builder.build() - } - - static HttpLoggingInterceptor createLoggingInterceptor() { - def logger = LoggerFactory.getLogger("com.cloudogu.gitops.HttpClient") - - def ret = new HttpLoggingInterceptor(new HttpLoggingInterceptor.Logger() { - @Override - void log(@NotNull String msg) { - logger.trace(msg) - } - }) - - ret.setLevel(HttpLoggingInterceptor.Level.HEADERS) - ret.redactHeader("Authorization") - - return ret - } - - static InsecureSslContext insecureSslContext() { - def noCheckTrustManager = new X509TrustManager() { - @Override - void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {} - - @Override - void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {} - - @Override - X509Certificate[] getAcceptedIssuers() { - return new X509Certificate[0] - } - } - def sslCtxt = SSLContext.getInstance('SSL') - sslCtxt.init(null, [noCheckTrustManager] as X509TrustManager[], new SecureRandom()) - - return new InsecureSslContext(sslCtxt.socketFactory, noCheckTrustManager) - } - - @TupleConstructor(defaults = false) - static class InsecureSslContext { - final SSLSocketFactory socketFactory - final X509TrustManager trustManager - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptor.groovy b/src/main/groovy/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptor.groovy deleted file mode 100644 index bf7d1a921..000000000 --- a/src/main/groovy/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptor.groovy +++ /dev/null @@ -1,77 +0,0 @@ -package com.cloudogu.gitops.dependencyinjection.okhttp - -import groovy.util.logging.Slf4j -import okhttp3.Interceptor -import okhttp3.Response -import org.jetbrains.annotations.NotNull - -/** - * Retries request on specific status codes as well as timeouts. - * Both error codes (like temporary (!) 500 or 401/403) and timeouts occur often during our jenkins initialization, - * due to necessary restarts, e.g. after plugin installs.*/ -@Slf4j -class RetryInterceptor implements Interceptor { - private int retries - private int waitPeriodInMs - - // Number of retries in uncommonly high, because we might have to outlive a unexpected Jenkins restart - RetryInterceptor(int retries = 180, int waitPeriodInMs = 2000) { - this.waitPeriodInMs = waitPeriodInMs - this.retries = retries - } - - @Override - Response intercept(@NotNull Chain chain) throws IOException { - def i = 0 - Response response = null - IOException lastException = null - - do { - try { - response = chain.proceed(chain.request()) - - if (response.code() !in getStatusCodesToRetry()) { - // Success or non-retriable error - return the response - return response - } - - log.trace("Retry HTTP Request to {} due to status code {}", chain.request().url().toString(), response.code()) - response.close() - - } catch (SocketTimeoutException e) { - lastException = e - log.trace("Retry HTTP Request to {} due to SocketTimeoutException: {}", chain.request().url().toString(), e.message) - } - - // Wait before next retry (but not after the last attempt) - if (i < retries) { - Thread.sleep(waitPeriodInMs) - } - ++i - - } while (i <= retries) - - // If we got here, all retries failed - if (response != null) { - // Return the last failed response - return response - } else if (lastException != null) { - // All attempts resulted in timeout - throw the last exception - throw lastException - } else { - // This should never happen, but as a safety net - throw new IOException("Request failed after ${retries} retries") - } - } - - private List getStatusCodesToRetry() { - return [// list of codes from curl --retry - 408, // Request Timeout - 429, // Too Many Requests - 500, // Internal Server Error - 502, // Bad Gateway - 503, // Service Unavailable - 504, // Gateway Timeout - ] - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/destroy/ArgoCDDestructionHandler.groovy b/src/main/groovy/com/cloudogu/gitops/destroy/ArgoCDDestructionHandler.groovy deleted file mode 100644 index 849c41e52..000000000 --- a/src/main/groovy/com/cloudogu/gitops/destroy/ArgoCDDestructionHandler.groovy +++ /dev/null @@ -1,95 +0,0 @@ -package com.cloudogu.gitops.destroy - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.transform.CompileStatic -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -import java.nio.file.Path - -@Singleton -@Order(100) -@CompileStatic -class ArgoCDDestructionHandler implements DestructionHandler { - private K8sClient k8sClient - private HelmClient helmClient - private GitRepoFactory repoProvider - private Config config - private FileSystemUtils fileSystemUtils - private GitHandler gitHandler - - ArgoCDDestructionHandler(Config config, - K8sClient k8sClient, - HelmClient helmClient, - GitRepoFactory repoProvider, - FileSystemUtils fileSystemUtils, - GitHandler gitHandler) { - this.k8sClient = k8sClient - this.helmClient = helmClient - this.repoProvider = repoProvider - this.config = config - this.fileSystemUtils = fileSystemUtils - this.gitHandler = gitHandler - } - - @Override - void destroy() { - - def repo = repoProvider.getRepo("argocd/cloud-resources", gitHandler.resourcesScm) - repo.cloneRepo() - - for (def app in k8sClient.getCustomResource("app")) { - if (app.name == 'bootstrap' || app.name == 'argocd' || app.name == 'projects') { - // we don't want bootstrap to kill everything - // argocd and projects are needed for argocd to function and run finalizers - continue - } - - k8sClient.patch("app", - app.name, - app.namespace, - 'merge', - [metadata: [finalizers: ["resources-finalizer.argocd.argoproj.io"]]]) - } - - List> appsToBeDeleted = [new Tuple2("argocd", "bootstrap"), // first to prevent recreation - new Tuple2("argocd", "cluster-resources"), - new Tuple2("argocd", "example-apps"),] - - for (def app in appsToBeDeleted) { - k8sClient.delete("app", app.v1, app.v2) - } - - installArgoCDViaHelm(repo) - helmClient.uninstall('argocd', 'argocd') - for (def project in k8sClient.getCustomResource('appprojects')) { - k8sClient.delete("appproject", project.namespace, project.name) - } - - k8sClient.delete("app", 'argocd', "projects") - k8sClient.delete("app", 'argocd', "argocd") - - k8sClient.delete('secret', 'default', 'jenkins-credentials') - k8sClient.delete('secret', 'default', 'argocd-repo-creds-scm') - } - - void installArgoCDViaHelm(GitRepo repo) { - // this is a hack to be able to uninstall using helm - def namePrefix = config.application.namePrefix - def argocdNamespace = namePrefix + config.features.argocd.namespace - // Install umbrella chart from folder - String umbrellaChartPath = Path.of(repo.getAbsoluteLocalRepoTmpDir(), 'argocd/') - // Even if the Chart.lock already contains the repo, we need to add it before resolving it - // See https://github.com/helm/helm/issues/8036#issuecomment-872502901 - List helmDependencies = fileSystemUtils.readYaml(Path.of(umbrellaChartPath, 'Chart.yaml'))['dependencies'].collect { it } - helmClient.addRepo('argo', helmDependencies[0]['repository'] as String) - helmClient.dependencyBuild(umbrellaChartPath) - helmClient.upgrade('argocd', umbrellaChartPath, [namespace: "${argocdNamespace}"]) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/destroy/Destroyer.groovy b/src/main/groovy/com/cloudogu/gitops/destroy/Destroyer.groovy deleted file mode 100644 index edb8f0e96..000000000 --- a/src/main/groovy/com/cloudogu/gitops/destroy/Destroyer.groovy +++ /dev/null @@ -1,28 +0,0 @@ -package com.cloudogu.gitops.destroy - -import jakarta.inject.Singleton -import groovy.util.logging.Slf4j - -@Singleton -@Slf4j -class Destroyer { - - final List destructionHandlers - - Destroyer(List destructionHandlers) { - this.destructionHandlers = destructionHandlers - } - - void destroy() { - log.info("Start destroying") - for (def handler in destructionHandlers) { - log.info("Running handler $handler.class.simpleName") - handler.destroy() - } - log.info("Finished destroying") - } - - List getDestructionHandlers() { - return destructionHandlers - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/destroy/DestructionHandler.groovy b/src/main/groovy/com/cloudogu/gitops/destroy/DestructionHandler.groovy deleted file mode 100644 index d049bc31b..000000000 --- a/src/main/groovy/com/cloudogu/gitops/destroy/DestructionHandler.groovy +++ /dev/null @@ -1,5 +0,0 @@ -package com.cloudogu.gitops.destroy - -interface DestructionHandler { - void destroy() -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/destroy/JenkinsDestructionHandler.groovy b/src/main/groovy/com/cloudogu/gitops/destroy/JenkinsDestructionHandler.groovy deleted file mode 100644 index fca2a8cc8..000000000 --- a/src/main/groovy/com/cloudogu/gitops/destroy/JenkinsDestructionHandler.groovy +++ /dev/null @@ -1,33 +0,0 @@ -package com.cloudogu.gitops.destroy - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.jenkins.GlobalPropertyManager -import com.cloudogu.gitops.infrastructure.jenkins.JobManager -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Singleton -@Order(300) -class JenkinsDestructionHandler implements DestructionHandler { - private JobManager jobManager - private GlobalPropertyManager globalPropertyManager - private Config configuration - - JenkinsDestructionHandler(JobManager jobManager, Config configuration, GlobalPropertyManager globalPropertyManager) { - this.jobManager = jobManager - this.configuration = configuration - this.globalPropertyManager = globalPropertyManager - } - - @Override - void destroy() { - jobManager.deleteJob("${configuration.application.namePrefix}example-apps") - globalPropertyManager.deleteGlobalProperty("SCMM_URL") - globalPropertyManager.deleteGlobalProperty("${configuration.application.namePrefixForEnvVars}REGISTRY_URL") - globalPropertyManager.deleteGlobalProperty("${configuration.application.namePrefixForEnvVars}REGISTRY_PATH") - globalPropertyManager.deleteGlobalProperty("${configuration.application.namePrefixForEnvVars}REGISTRY_PROXY_URL") - globalPropertyManager.deleteGlobalProperty("${configuration.application.namePrefixForEnvVars}REGISTRY_PROXY_PATH") - - globalPropertyManager.deleteGlobalProperty("${configuration.application.namePrefixForEnvVars}K8S_VERSION") - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/destroy/ScmmDestructionHandler.groovy b/src/main/groovy/com/cloudogu/gitops/destroy/ScmmDestructionHandler.groovy deleted file mode 100644 index ef4b8513b..000000000 --- a/src/main/groovy/com/cloudogu/gitops/destroy/ScmmDestructionHandler.groovy +++ /dev/null @@ -1,47 +0,0 @@ -package com.cloudogu.gitops.destroy - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Singleton -@Order(200) -class ScmmDestructionHandler implements DestructionHandler { - private ScmManagerApiClient scmmApiClient - private Config config - - ScmmDestructionHandler(Config config) { - this.config = config - this.scmmApiClient = scmmApiClient - } - - @Override - void destroy() { - deleteUser("gitops") - deleteRepository("argocd", "argocd") - deleteRepository("argocd", "cluster-resources") - deleteRepository("argocd", "example-apps") - deleteRepository("3rd-party-dependencies", "ces-build-lib", false) - deleteRepository("3rd-party-dependencies", "gitops-build-lib", false) - deleteRepository("3rd-party-dependencies", "spring-boot-helm-chart", false) - deleteRepository("3rd-party-dependencies", "spring-boot-helm-chart-with-dependency", false) - } - - private void deleteRepository(String namespace, String repository, boolean prefixNamespace = true) { - def namePrefix = prefixNamespace ? config.application.namePrefix : '' - def response = scmmApiClient.repositoryApi().delete("${namePrefix}$namespace", repository).execute() - - if (response.code() != 204) { - throw new RuntimeException("Could not delete user $namespace/$repository (${response.code()} ${response.message()}): ${response.errorBody().string()}") - } - } - - private void deleteUser(String name) { - def response = scmmApiClient.usersApi().delete("${config.application.namePrefix}$name").execute() - - if (response.code() != 204) { - throw new RuntimeException("Could not delete user $name (${response.code()} ${response.message()}): ${response.errorBody().string()}") - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategy.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategy.groovy deleted file mode 100644 index 2b1176963..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategy.groovy +++ /dev/null @@ -1,136 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.utils.FileSystemUtils -import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator -import com.fasterxml.jackson.dataformat.yaml.YAMLMapper -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton - -import java.nio.file.Path - -@Singleton -@Slf4j -class ArgoCdApplicationStrategy implements DeploymentStrategy { - private FileSystemUtils fileSystemUtils - private Config config - private final GitRepoFactory gitRepoProvider - - private GitHandler gitHandler - - ArgoCdApplicationStrategy(Config config, - FileSystemUtils fileSystemUtils, - GitRepoFactory gitRepoProvider, - GitHandler gitHandler) { - this.gitRepoProvider = gitRepoProvider - this.fileSystemUtils = fileSystemUtils - this.config = config - this.gitHandler = gitHandler - } - - @Override - @SuppressWarnings('GroovyGStringKey') - // Using dynamic strings as keys seems an easy to read way to avoid more ifs - void deployFeature(String repoURL, String repoName, String chartOrPath, String version, String namespace, - String releaseName, Path helmValuesPath, RepoType repoType) { - log.trace("Deploying helm chart via ArgoCD: ${releaseName}. Reading values from ${helmValuesPath}") - def namePrefix = config.application.namePrefix - def shallCreateNamespace = config.features['argocd']['operator'] ? "CreateNamespace=false" : "CreateNamespace=true" - - GitRepo clusterResourcesRepo = gitRepoProvider.getRepo('argocd/cluster-resources', this.gitHandler.resourcesScm) - clusterResourcesRepo.cloneRepo() - - String project = "cluster-resources" - String namespaceName = "${namePrefix}" + config.features.argocd.namespace - String featureName = repoName - //DedicatedInstances - if (config.multiTenant.useDedicatedInstance) { - repoName = "${config.application.namePrefix}${repoName}" - namespaceName = "${config.multiTenant.centralArgocdNamespace}" - project = config.application.namePrefix.replaceFirst(/-$/, "") - } - - // Feature-Name -> Ordner under apps/ - String featurePath = "apps/${featureName}" - - // --- ensure folders exist before writing files --- - String repoRoot = clusterResourcesRepo.getAbsoluteLocalRepoTmpDir() - Path.of(repoRoot, featurePath).toFile().mkdirs() - - // 1) GOP-managed values (may be overwritten each run) - String gopValuesPath = "${featurePath}/${featureName}-gop-helm.yaml" - // relative to repo-root - def inlineValues = helmValuesPath.toFile().text - clusterResourcesRepo.writeFile(gopValuesPath, inlineValues) - - // 2) User values (must NEVER be overwritten by GOP) - String userValuesPath = "${featurePath}/${featureName}-user-values.yaml" - Path userValuesAbsPath = Path.of(repoRoot, userValuesPath) - if (!userValuesAbsPath.toFile().exists()) { - clusterResourcesRepo.writeFile(userValuesPath, "") - } - - // 1) helm source (external chart source) - def helmSource = [repoURL : repoURL, - (chooseKeyChartOrPath(repoType)): chartOrPath, - targetRevision : version, - helm : [releaseName : releaseName, - valueFiles : ["\$values/${gopValuesPath}".toString(), - "\$values/${userValuesPath}".toString()], - ignoreMissingValueFiles: true]] - - // 2) Git source for values - // - repoURL: cluster-resources repo - // - ref: values → used in valueFiles as $values - // - path: apps/ → additional manifests - def featureRepoUrl = "${clusterResourcesRepo.gitProvider.repoPrefix()}argocd/cluster-resources.git".toString() - def gitSource = [repoURL : featureRepoUrl, - targetRevision: "main", - ref : "values", - path : featurePath, - directory : [recurse: true]] - - def sources = [helmSource, gitSource] - - // Prepare ArgoCD Application YAML - def yamlMapper = YAMLMapper.builder() - .enable(YAMLGenerator.Feature.LITERAL_BLOCK_STYLE) - .build() - - def yamlResult = yamlMapper.writeValueAsString([apiVersion: "argoproj.io/v1alpha1", - kind : "Application", - metadata : [name : repoName, - namespace: namespaceName], - spec : [destination: [server : "https://kubernetes.default.svc", - namespace: namespace], - project : project, - sources : sources, - syncPolicy : [automated : [prune : true, - selfHeal: true], - syncOptions: [// So that we can apply very large resources (e.g. prometheus CRD) - "ServerSideApply=true", - // Create namespaces for helm charts (while not using the argocd-operater mode) - shallCreateNamespace]]]]) - - String appManifestPath = "apps/argocd/applications/${releaseName}.yaml" - - clusterResourcesRepo.writeFile(appManifestPath, yamlResult) - - log.debug("Deploying helm release ${releaseName} basing on chart ${chartOrPath} from ${repoURL}, version " + "${version}, into namespace ${namespace}. Using Argo CD application:\n${yamlResult}") - - clusterResourcesRepo.commitAndPush("Added $repoName/$chartOrPath to ArgoCD") - } - - String chooseKeyChartOrPath(RepoType repoType) { - switch (repoType) { - case RepoType.HELM: 'chart' - break - case RepoType.GIT: 'path' - break - default: throw new RuntimeException("Repo type ${repoType} not implemented for ${this.class.simpleName}") - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/Deployer.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/Deployer.groovy deleted file mode 100644 index dd0ab8b07..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/Deployer.groovy +++ /dev/null @@ -1,31 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import com.cloudogu.gitops.config.Config -import io.micronaut.context.annotation.Primary -import jakarta.inject.Singleton - -import java.nio.file.Path - -@Singleton -@Primary -class Deployer implements DeploymentStrategy { - private Config config - private ArgoCdApplicationStrategy argoCdStrategy - private HelmStrategy helmStrategy - - Deployer(Config config, ArgoCdApplicationStrategy argoCdStrategy, HelmStrategy helmStrategy) { - this.helmStrategy = helmStrategy - this.argoCdStrategy = argoCdStrategy - this.config = config - } - - @Override - void deployFeature(String repoURL, String repoName, String chartOrPath, String version, String namespace, - String releaseName, Path helmValuesPath, RepoType repoType) { - if (config.features['argocd']['active']) { - argoCdStrategy.deployFeature(repoURL, repoName, chartOrPath, version, namespace, releaseName, helmValuesPath, repoType) - } else { - helmStrategy.deployFeature(repoURL, repoName, chartOrPath, version, namespace, releaseName, helmValuesPath, repoType) - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/DeploymentStrategy.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/DeploymentStrategy.groovy deleted file mode 100644 index 342bdfe33..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/DeploymentStrategy.groovy +++ /dev/null @@ -1,17 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import java.nio.file.Path - -interface DeploymentStrategy { - void deployFeature(String repoURL, String repoName, String chartOrPath, String version, String namespace, - String releaseName, Path helmValuesPath, RepoType repoType) - - default void deployFeature(String repoURL, String repoName, String chart, String version, String namespace, - String releaseName, Path helmValuesPath) { - deployFeature(repoURL, repoName, chart, version, namespace, releaseName, helmValuesPath, RepoType.HELM) - } - - enum RepoType { - HELM, GIT - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/HelmStrategy.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/HelmStrategy.groovy deleted file mode 100644 index 87f857752..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/deployment/HelmStrategy.groovy +++ /dev/null @@ -1,38 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton - -import java.nio.file.Path - -@Slf4j -@Singleton -class HelmStrategy implements DeploymentStrategy { - private HelmClient helmClient - private Config config - - HelmStrategy(Config config, HelmClient helmClient) { - this.config = config - this.helmClient = helmClient - } - - @Override - void deployFeature(String repoURL, String repoName, String chartOrPath, String version, String namespace, - String releaseName, Path helmValuesPath, RepoType repoType) { - - if (repoType == RepoType.GIT) { - // This would be possible with plugins or by pulling the repo first, but for now, we don't need it - throw new RuntimeException("Unable to deploy helm chart via Helm CLI from Git URL, because helm does not support this out of the box.\n" + "Repo URL: ${repoURL}") - } - - log.debug("Imperatively deploying helm release ${releaseName} basing on chart ${chartOrPath} from ${repoURL}, " + "version ${version}, into namespace ${namespace}. Using values:\n${helmValuesPath.toFile().text}") - - helmClient.addRepo(repoName, repoURL) - helmClient.upgrade(releaseName, "$repoName/$chartOrPath", - [namespace: namespace, - version : version, - values : helmValuesPath.toString()]) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/GitRepo.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/GitRepo.groovy deleted file mode 100644 index bc151a3d4..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/GitRepo.groovy +++ /dev/null @@ -1,286 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git - -import com.cloudogu.gitops.cli.Version -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope -import com.cloudogu.gitops.infrastructure.git.providers.Scope -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.TemplatingEngine -import com.cloudogu.gitops.utils.jgit.helpers.InsecureCredentialProvider -import groovy.util.logging.Slf4j -import org.eclipse.jgit.api.Git -import org.eclipse.jgit.api.ListBranchCommand -import org.eclipse.jgit.api.PushCommand -import org.eclipse.jgit.lib.ObjectId -import org.eclipse.jgit.lib.Ref -import org.eclipse.jgit.revwalk.RevCommit -import org.eclipse.jgit.revwalk.RevWalk -import org.eclipse.jgit.transport.ChainingCredentialsProvider -import org.eclipse.jgit.transport.CredentialsProvider -import org.eclipse.jgit.transport.RefSpec -import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider -import org.eclipse.jgit.treewalk.TreeWalk -import org.eclipse.jgit.treewalk.filter.PathFilter - -@Slf4j -class GitRepo { - - static final String NAMESPACE_3RD_PARTY_DEPENDENCIES = '3rd-party-dependencies' - - private final Config config - public GitProvider gitProvider - private final FileSystemUtils fileSystemUtils - - private final String repoTarget - private final boolean insecure - private final String gitName - private final String gitEmail - - private Git gitMemoization - private final String absoluteLocalRepoTmpDir - - GitRepo(Config config, - GitProvider gitProvider, - String repoTarget, - FileSystemUtils fileSystemUtils) { - def tmpDir = File.createTempDir() - tmpDir.deleteOnExit() - this.absoluteLocalRepoTmpDir = tmpDir.absolutePath - this.config = config - this.gitProvider = gitProvider - this.fileSystemUtils = fileSystemUtils - - this.repoTarget = "${config.application.namePrefix}${repoTarget}" - - this.insecure = config.application.insecure - this.gitName = config.application.gitName - this.gitEmail = config.application.gitEmail - } - - String getRepoTarget() { - return repoTarget - } - - boolean createRepositoryAndSetPermission(String description, boolean initialize = true) { - def isNewRepo = this.gitProvider.createRepository(repoTarget, description, initialize) - if (gitProvider.getGitOpsUsername()) { - gitProvider.setRepositoryPermission(repoTarget, - gitProvider.getGitOpsUsername(), - AccessRole.WRITE, - Scope.USER) - } - return isNewRepo - - } - - String getAbsoluteLocalRepoTmpDir() { - return absoluteLocalRepoTmpDir - } - - void cloneRepo() { - def cloneUrl = getGitRepositoryUrl() - log.debug("Cloning ${repoTarget}, Origin: ${cloneUrl}") - Git.cloneRepository() - .setURI(cloneUrl) - .setDirectory(new File(absoluteLocalRepoTmpDir)) - .setCredentialsProvider(getCredentialProvider()) - .call() - } - - void commitAndPush(String message, String tag) { - commitAndPush(message, tag, 'HEAD:refs/heads/main') - } - - void commitAndPush(String commitMessage, String tag, String refSpec) { - log.debug("Adding files to ${repoTarget}") - def git = getGit() - git.add().addFilepattern(".").call() - - if (git.status().call().hasUncommittedChanges()) { - log.debug("Commiting ${repoTarget}") - git.commit() - .setSign(false) - .setMessage(commitMessage) - .setAuthor(gitName, gitEmail) - .setCommitter("${gitName} - GOP v${Version.NAME.split(',')[0].replace('(', '')}", gitEmail) //parsing the Versions from the full text in Version.Name. In local Dev there is no Tag->Version is empty - .call() - - def pushCommand = createPushCommand(refSpec) - - if (tag) { - log.debug("Setting tag '${tag}' on repo: ${repoTarget}") - // Delete existing tags first to get idempotence - git.tagDelete().setTags(tag).call() - git.tag() - .setName(tag) - .call() - pushCommand.setPushTags() - } - - log.debug("Pushing repo: ${repoTarget}, refSpec: ${refSpec}") - pushCommand.call() - } else { - log.debug("No changes after add, nothing to commit or push on repo: ${repoTarget}") - } - } - - void commitAndPush(String commitMessage) { - commitAndPush(commitMessage, null, 'HEAD:refs/heads/main') - } - - /** - * Push all refs, i.e. all tags and branches*/ - - void pushAll(boolean force) { - createPushCommand('refs/*:refs/*').setForce(force).call() - } - - void pushRef(String ref, boolean force) { - pushRef(ref, ref, force) - } - - void pushRef(String ref, String targetRef, boolean force) { - createPushCommand("${ref}:${targetRef}").setForce(force).call() - } - - /** - * Delete all files in this repository*/ - void clearRepo() { - fileSystemUtils.deleteFilesExcept(new File(absoluteLocalRepoTmpDir), ".git") - } - - void copyDirectoryContents(String srcDir) { - copyDirectoryContents(srcDir, (FileFilter) null) - } - - void copyDirectoryContents(String srcDir, FileFilter fileFilter) { - if (!srcDir) { - log.warn("Source directory is not defined. Nothing to copy?") - return - } - - log.debug("Initializing repo $repoTarget from $srcDir") - String absoluteSrcDirLocation = new File(srcDir).isAbsolute() ? srcDir : "${fileSystemUtils.getRootDir()}/${srcDir}" - fileSystemUtils.copyDirectory(absoluteSrcDirLocation, absoluteLocalRepoTmpDir, fileFilter) - } - - void writeFile(String path, String content) { - def file = new File("$absoluteLocalRepoTmpDir/$path") - fileSystemUtils.createDirectory(file.parent) - file.createNewFile() - file.text = content - } - - void replaceTemplates(Map parameters) { - new TemplatingEngine().replaceTemplates(new File(absoluteLocalRepoTmpDir), parameters) - } - - String getGitRepositoryUrl() { - return this.gitProvider.repoUrl(repoTarget, RepoUrlScope.CLIENT) - } - - static boolean isCommit(File repoPath, String ref) { - if (!ref) { - return false - } - - try (Git git = Git.open(repoPath)) { - // Get all branch and tag names - def allRefs = [] - - // Add all branch names (without refs/heads/ prefix) - git.branchList().call().each { branch -> allRefs.add(branch.name.replaceFirst('refs/heads/', '')) - } - - // Add all tag names (without refs/tags/ prefix) - git.tagList().call().each { tag -> allRefs.add(tag.name.replaceFirst('refs/tags/', '')) - } - - // If the ref matches any branch or tag name, it's not a commit hash - if (allRefs.contains(ref)) { - return false - } - - // If it's not a branch or tag, try to resolve it as a commit - def objectId = git.repository.resolve(ref) - return objectId != null - - } - } - - /** - * checks, if file exists in repo in some branch. - * @param pathToRepo - * @param filename - */ - static boolean existFileInSomeBranch(String repo, String filename) { - String filenameToSearch = filename - File repoPath = new File(repo + '/.git') - - try (def git = Git.open(repoPath)) { - List branches = git - .branchList() - .setListMode(ListBranchCommand.ListMode.ALL) - .call() - - for (Ref branch : branches) { - String branchName = branch.getName() - - ObjectId commitId = git.repository.resolve(branchName) - if (commitId == null) { - continue - } - try (RevWalk revWalk = new RevWalk(git.repository)) { - RevCommit commit = revWalk.parseCommit(commitId) - try (TreeWalk treeWalk = new TreeWalk(git.repository)) { - - treeWalk.addTree(commit.getTree()) - treeWalk.setFilter(PathFilter.create(filenameToSearch)) - - if (treeWalk.next()) { - log.debug("File ${filename} found in branch ${branchName}") - - return true - } - } - } - } - } - log.debug("File ${filename} not found in repository ${repoPath}") - return false - } - - static boolean isTag(File repo, String ref) { - if (!ref) { - return false - } - try (def git = Git.open(repo)) { - git.tagList().call().any { it.name.endsWith("/" + ref) || it.name == ref } - } - } - - private PushCommand createPushCommand(String refSpec) { - getGit() - .push() - .setRemote(getGitRepositoryUrl()) - .setRefSpecs(new RefSpec(refSpec)) - .setCredentialsProvider(getCredentialProvider()) - } - - private Git getGit() { - if (gitMemoization != null) { - return gitMemoization - } - - return gitMemoization = Git.open(new File(absoluteLocalRepoTmpDir)) - } - - private CredentialsProvider getCredentialProvider() { - def auth = this.gitProvider.getCredentials() - def passwordAuthentication = new UsernamePasswordCredentialsProvider(auth.username, auth.password) - return insecure ? new ChainingCredentialsProvider(new InsecureCredentialProvider(), passwordAuthentication) : passwordAuthentication - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/GitRepoFactory.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/GitRepoFactory.groovy deleted file mode 100644 index f72180c57..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/GitRepoFactory.groovy +++ /dev/null @@ -1,22 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.utils.FileSystemUtils -import jakarta.inject.Singleton - -@Singleton -class GitRepoFactory { - protected final Config config - protected final FileSystemUtils fileSystemUtils - - GitRepoFactory(Config config, FileSystemUtils fileSystemUtils) { - this.fileSystemUtils = fileSystemUtils - this.config = config - } - - GitRepo getRepo(String repoTarget, GitProvider gitProvider) { - return new GitRepo(config, gitProvider, repoTarget, fileSystemUtils) - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/GitProvider.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/GitProvider.groovy deleted file mode 100644 index 1415a59d3..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/GitProvider.groovy +++ /dev/null @@ -1,74 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers - -import com.cloudogu.gitops.config.Credentials - -interface GitProvider { - - default boolean createRepository(String repoTarget, String description) { - return createRepository(repoTarget, description, true); - } - - boolean createRepository(String repoTarget, String description, boolean initialize) - - void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) - - default String repoUrl(String repoTarget) { - return repoUrl(repoTarget, RepoUrlScope.IN_CLUSTER); - } - - String repoUrl(String repoTarget, RepoUrlScope scope); - - String repoPrefix() - - Credentials getCredentials() - - URI prometheusMetricsEndpoint() - - /** - * Deletes the given repository on the provider, if supported. - * Note: This capability is not used by the current destruction flow, - * which talks directly to provider-specific clients (e.g. ScmManagerApiClient).*/ - void deleteRepository(String namespace, String repository, boolean prefixNamespace) - - /** - * Deletes a user account on the provider, if supported. - * Note: Not used by the current destruction flow; kept as an optional capability - * on the GitProvider abstraction */ - void deleteUser(String name) - - /** - * Sets the default branch of a repository, if supported by the provider; - * kept as an optional capability on the GitProvider abstraction */ - void setDefaultBranch(String repoTarget, String branch) - - String getUrl() - - String getProtocol() - - String getHost() - - String getGitOpsUsername() - -} - -enum AccessRole { - READ, WRITE, MAINTAIN, ADMIN, OWNER -} - -enum Scope { - USER, GROUP -} - -/** - * IN_CLUSTER: URLs intended for workloads running inside the Kubernetes cluster - * (e.g., ArgoCD, Jobs, in-cluster automation). - * - * CLIENT : URLs intended for interactive or CI clients performing push/clone operations, - * regardless of their location. - * If the application itself runs inside Kubernetes, the Service DNS is used; - * otherwise, NodePort (for internal installations) or externalBase (for external ones) - * is selected automatically.*/ -enum RepoUrlScope { - IN_CLUSTER, - CLIENT -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/gitlab/Gitlab.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/gitlab/Gitlab.groovy deleted file mode 100644 index d596d29c2..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/gitlab/Gitlab.groovy +++ /dev/null @@ -1,393 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.gitlab - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.config.scm.util.GitlabConfig -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope -import com.cloudogu.gitops.infrastructure.git.providers.Scope -import groovy.util.logging.Slf4j -import org.gitlab4j.api.GitLabApi -import org.gitlab4j.api.GitLabApiException -import org.gitlab4j.api.models.AccessLevel -import org.gitlab4j.api.models.Group -import org.gitlab4j.api.models.Project -import org.gitlab4j.api.models.Visibility - -import java.util.logging.Level - -@Slf4j -class Gitlab implements GitProvider { - - private final Config config - private final GitLabApi api - private GitlabConfig gitlabConfig - - Gitlab(Config config, GitlabConfig gitlabConfig) { - this.config = config - this.gitlabConfig = gitlabConfig - - String url = Objects.requireNonNull(gitlabConfig.getUrl(), "Missing gitlab url in config.scm.gitlab.url").trim() - String pat = Objects.requireNonNull(gitlabConfig.getCredentials()?.password, "Missing gitlab token").trim() - this.api = new GitLabApi(url, pat) - this.api.enableRequestResponseLogging(Level.ALL) - } - - @Override - boolean createRepository(String repoTarget, String description, boolean initialize) { - def repoNamespace = repoTarget.split('/', 2)[0] - def repoName = repoTarget.split('/', 2)[1] - - // def repoNamespacePrefixed = config.application.namePrefix + repoNamespace - // 1) Resolve parent by numeric ID (do NOT treat the ID as a path!) - Group parent = parentGroup() - String repoNamespacePath = repoNamespace.toLowerCase() - String projectPath = repoName.toLowerCase() - - long subgroupId = ensureSubgroupUnderParentId(parent, repoNamespacePath) - String fullProjectPath = "${parentFullPath()}/${repoNamespacePath}/${projectPath}" - - if (findProject(fullProjectPath).present) { - log.info("GitLab project already exists: ${fullProjectPath}") - return false - } - - def project = new Project() - .withName(repoName) - .withPath(projectPath) - .withDescription(description ?: "") - .withIssuesEnabled(false) - .withMergeRequestsEnabled(false) - .withWikiEnabled(false) - .withSnippetsEnabled(false) - .withNamespaceId(subgroupId) - .withInitializeWithReadme(initialize) - project.visibility = toVisibility(gitlabConfig.defaultVisibility) - - def created = api.projectApi.createProject(project) - log.info("Created GitLab project ${created.getPathWithNamespace()} (id=${created.id})") - return true - } - - @Override - void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { - String fullPath = resolveFullPath(repoTarget) - Project project = findProjectOrThrow(fullPath) - AccessLevel level = toAccessLevel(role, scope) - if (scope == Scope.GROUP) { - def group = api.groupApi.getGroups(principal) - .find { it.fullPath == principal || it.path == principal || it.name == principal } - if (!group) throw new IllegalArgumentException("Group '${principal}' not found") - api.projectApi.shareProject(project.id, group.id, level, null) - } else { - def user = api.userApi.findUsers(principal) - .find { it.username == principal || it.email == principal } - if (!user) throw new IllegalArgumentException("User '${principal}' not found") - api.projectApi.addMember(project.id, user.id, level) - } - } - - @Override - String repoUrl(String repoTarget, RepoUrlScope scope) { - String base = gitlabConfig.url.strip() - return "${base}/${parentFullPath()}/${repoTarget}.git" - } - - @Override - String repoPrefix() { - String base = gitlabConfig.url.strip() - def prefix = (config.application.namePrefix ?: "").strip() - return "${base}/${parentFullPath()}/${prefix}" - - } - - @Override - Credentials getCredentials() { - return this.gitlabConfig.credentials - } - - @Override - String getProtocol() { - return gitlabConfig.url - } - - String getHost() { - return gitlabConfig.url - } - - @Override - String getGitOpsUsername() { - return gitlabConfig.gitOpsUsername - } - - @Override - String getUrl() { - return this.gitlabConfig.url - } - - /** - * Prometheus integration is only required for SCM-Manager. - * GitLab provides its own built-in Prometheus metrics, so we don't expose an endpoint here.*/ - @Override - URI prometheusMetricsEndpoint() { - return null - } - - /** - * No-op by design. GitLab repository deletion is not managed through this abstraction. - * Kept for interface compatibility only.*/ - @Override - void deleteRepository(String namespace, String repository, boolean prefixNamespace) { - // intentionally left blank - } - - /** - * No-op by design. User deletion is not supported or handled through this provider. - * Kept for interface compatibility only.*/ - @Override - void deleteUser(String name) { - // intentionally left blank - } - - /** - * No-op by design. Default branch management is not implemented via this abstraction. - * Kept for interface compatibility only.*/ - @Override - void setDefaultBranch(String repoTarget, String branch) { - // intentionally left blank - } - - private Group parentGroup() { - String raw = gitlabConfig?.parentGroupId?.trim() - if (!raw) throw new IllegalArgumentException("--gitlab-group-id is required") - - boolean isNumeric = raw ==~ /\d+/ - - def groupApi = api.getGroupApi() - if (isNumeric) { - return groupApi.getGroup(Long.parseLong(raw)) - } else { - return groupApi.getGroup(raw.replaceAll('^/+', '')) - } - } - - private String parentFullPath() { - parentGroup().fullPath - } - - /** Ensure a single-level subgroup exists under 'parent'; return its namespace (group) ID. */ - private long ensureSubgroupUnderParentId(Group parent, String segPath) { - // 1) Already there? - Group existing = findDirectSubgroupByPath(parent.id as Long, segPath) - if (existing != null) return existing.id as Long - - - // 2) Guard against project/subgroup name collision in the same parent - Project collision = findDirectProjectByPath(parent.id as Long, segPath) - if (collision != null) { - throw new IllegalStateException("Cannot create subgroup '${segPath}' under '${parent.fullPath}': " + "a project with that path already exists at '${parent.fullPath}/${segPath}'. " + - "Rename/transfer the project first or choose a different subgroup name.") - } - - // 3) Create subgroup - Group toCreate = new Group() - .withName(segPath) // display name - .withPath(segPath) // (lowercase etc.) - .withParentId(parent.id) - - try { - Group created = api.groupApi.addGroup(toCreate) - log.info("Created group {}", created.fullPath) - return created.id as Long - } catch (GitLabApiException e) { - // If someone created it in parallel, treat 400/409 as "exists" and re-fetch - if (e.httpStatus in [400, 409]) { - Group retry = findDirectSubgroupByPath(parent.id as Long, segPath) - if (retry != null) return retry.id as Long - } - def ve = e.hasValidationErrors() ? e.getValidationErrors() : null - log.error("addGroup failed (parent={}, segPath={}, status={}, message={}, validationErrors={})", - parent.fullPath, segPath, e.httpStatus, e.getMessage(), ve) - throw e - } - } - - /** Find a direct subgroup of 'parentId' with the exact path . */ - private Group findDirectSubgroupByPath(Long parentId, String segPath) { - // uses the overload: getSubGroups(Object idOrPath) - List subGroups = api.groupApi.getSubGroups(parentId) - return subGroups?.find { Group subGroup -> subGroup.path == segPath } - } - - /** Find a direct project of 'parentId' with the exact path . */ - private Project findDirectProjectByPath(Long parentId, String path) { - // uses the overload: getProjects(Object idOrPath) - List projects = api.groupApi.getProjects(parentId) - return projects?.find { Project project -> project.path == path } - } - - // ---- Helpers ---- - private Optional findProject(String fullPath) { - try { - return Optional.ofNullable(api.projectApi.getProject(fullPath)) - } catch (Exception ignore) { - return Optional.empty() - } - } - - private Project findProjectOrThrow(String fullPath) { - return findProject(fullPath).orElseThrow { - new IllegalStateException("GitLab project '${fullPath}' not found") - } - } - - private String resolveFullPath(String repoTarget) { - if (!gitlabConfig.parentGroupId) { - throw new IllegalStateException("gitlab.parentGroup is not set") - } - return "${gitlabConfig.parentGroupId}/${repoTarget}" - } - - private static Visibility toVisibility(String s) { - switch ((s ?: "private").toLowerCase()) { - case "public": return Visibility.PUBLIC - case "internal": return Visibility.INTERNAL - default: return Visibility.PRIVATE - } - } - - // provider-agnostic AccessRole → GitLab AccessLevel - private static AccessLevel toAccessLevel(AccessRole role, Scope scope) { - switch (role) { - case AccessRole.READ: - // GitLab: Guests usually can't read private repo code; Reporter can. - return AccessLevel.REPORTER - case AccessRole.WRITE: - // Typical push/merge permissions - return AccessLevel.DEVELOPER - case AccessRole.MAINTAIN: - return AccessLevel.MAINTAINER - case AccessRole.ADMIN: - // No separate project-level "admin" → cap at Maintainer - return AccessLevel.MAINTAINER - case AccessRole.OWNER: - // OWNER is meaningful for groups/namespaces; for users on a project we cap to MAINTAINER - return (scope == Scope.GROUP) ? AccessLevel.OWNER : AccessLevel.MAINTAINER - default: - throw new IllegalArgumentException("Unknown role: ${role}") - } - } - - //TODO when git abctraction feature is ready, we will create before merge to main a branch, that - // contain this code as preservation for oop - /* ================================= SETUP CODE ==================================== - void setup() { - log.info("Creating Gitlab Groups") - def mainGroupName = "${config.application.namePrefix}scm".toString() - Group mainSCMGroup = this.gitlabApi.groupApi.getGroup(mainGroupName) - if (!mainSCMGroup) { - def tempGroup = new Group() - .withName(mainGroupName) - .withPath(mainGroupName.toLowerCase()) - .withParentId(null) - - mainSCMGroup = this.gitlabApi.groupApi.addGroup(tempGroup) - } - - String argoCDGroupName = 'argocd' - Optional argoCDGroup = getGroup("${mainGroupName}/${argoCDGroupName}") - if (argoCDGroup.isEmpty()) { - def tempGroup = new Group() - .withName(argoCDGroupName) - .withPath(argoCDGroupName.toLowerCase()) - .withParentId(mainSCMGroup.id) - - argoCDGroup = addGroup(tempGroup) - } - - argoCDGroup.ifPresent(this.&createArgoCDRepos) - - String dependencysGroupName = '3rd-party-dependencies' - Optional dependencysGroup = getGroup("${mainGroupName}/${dependencysGroupName}") - if (dependencysGroup.isEmpty()) { - def tempGroup = new Group() - .withName(dependencysGroupName) - .withPath(dependencysGroupName.toLowerCase()) - .withParentId(mainSCMGroup.id) - - addGroup(tempGroup) - } - - String exercisesGroupName = 'exercises' - Optional exercisesGroup = getGroup("${mainGroupName}/${exercisesGroupName}") - if (exercisesGroup.isEmpty()) { - def tempGroup = new Group() - .withName(exercisesGroupName) - .withPath(exercisesGroupName.toLowerCase()) - .withParentId(mainSCMGroup.id) - - exercisesGroup = addGroup(tempGroup) - } - - exercisesGroup.ifPresent(this.&createExercisesRepos) - } - - void createRepo(String name, String description) { - Optional project = getProject("${parentGroup.getFullPath()}/${name}".toString()) - if (project.isEmpty()) { - Project projectSpec = new Project() - .withName(name) - .withDescription(description) - .withIssuesEnabled(true) - .withMergeRequestsEnabled(true) - .withWikiEnabled(true) - .withSnippetsEnabled(true) - .withPublic(false) - .withNamespaceId(this.gitlabConfig.parentGroup.toLong()) - .withInitializeWithReadme(true) - - project = Optional.ofNullable(this.gitlabApi.projectApi.createProject(projectSpec)) - log.info("Project ${projectSpec} created in Gitlab!") - } - removeBranchProtection(project.get()) - } - - void removeBranchProtection(Project project) { - try { - this.gitlabApi.getProtectedBranchesApi().unprotectBranch(project.getId(), project.getDefaultBranch()) - log.debug("Unprotected default branch: " + project.getDefaultBranch()) - } catch (Exception ex) { - log.error("Failed to unprotect default branch '${project.getDefaultBranch()}' for project '${project.getName()}' (ID: ${project.getId()})", ex) - } - } - - - private Optional getGroup(String groupName) { - try { - return Optional.ofNullable(this.gitlabApi.groupApi.getGroup(groupName)) - } catch (Exception e) { - return Optional.empty() - } - } - - private Optional addGroup(Group group) { - try { - return Optional.ofNullable(this.gitlabApi.groupApi.addGroup(group)) - } catch (Exception e) { - return Optional.empty() - } - } - - private Optional getProject(String projectPath) { - try { - return Optional.ofNullable(this.gitlabApi.projectApi.getProject(projectPath)) - } catch (Exception e) { - return Optional.empty() - - - } - } - */ - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/Permission.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/Permission.groovy deleted file mode 100644 index c518a5aa4..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/Permission.groovy +++ /dev/null @@ -1,24 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager - -class Permission { - final String name - final Role role - final List verbs - final boolean groupPermission - - Permission(String name, Role role, boolean groupPermission = false, List verbs = []) { - this.name = name - this.role = role - this.verbs = verbs - this.groupPermission = groupPermission - } - - @Override - String toString() { - "Permission{name='$name', role=$role, verbs=$verbs, groupPermission=$groupPermission}" - } - - enum Role { - READ, WRITE, OWNER - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManager.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManager.groovy deleted file mode 100644 index 8f089ecbd..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManager.groovy +++ /dev/null @@ -1,190 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.config.scm.util.ScmManagerConfig -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope -import com.cloudogu.gitops.infrastructure.git.providers.Scope -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.core.ScmManagerSetup -import com.cloudogu.gitops.utils.NetworkingUtils -import groovy.util.logging.Slf4j -import retrofit2.Response - -@Slf4j -class ScmManager implements GitProvider { - - ScmManagerUrlResolver urls - ScmManagerApiClient apiClient - ScmManagerConfig scmmConfig - - NetworkingUtils networkingUtils - HelmStrategy helmStrategy - K8sClient k8sClient - Config config - ScmManagerSetup scmManagerSetup - - ScmManager(Config config, ScmManagerConfig scmmConfig, HelmStrategy helmStrategy, K8sClient k8sClient, NetworkingUtils networkingUtils, Boolean installNeeded = false) { - this.scmmConfig = scmmConfig - this.config = config - this.helmStrategy = helmStrategy - this.k8sClient = k8sClient - this.networkingUtils = networkingUtils - init(installNeeded) - } - - void init(installNeeded) { - // --- Init Setup --- - if (this.scmmConfig.internal && installNeeded) { - this.scmManagerSetup = new ScmManagerSetup(this) - this.scmManagerSetup.setupHelm() - this.urls = new ScmManagerUrlResolver(this.config, this.scmmConfig, this.k8sClient, this.networkingUtils) - this.apiClient = new ScmManagerApiClient(this.urls.clientApiBase().toString(), this.scmmConfig.credentials, this.config.application.insecure) - this.scmManagerSetup.waitForScmmAvailable() - this.scmManagerSetup.configure() - } else { - this.urls = new ScmManagerUrlResolver(this.config, this.scmmConfig, this.k8sClient, this.networkingUtils) - this.apiClient = new ScmManagerApiClient(this.urls.clientApiBase().toString(), this.scmmConfig.credentials, this.config.application.insecure) - } - } - - // --- Git operations --- - @Override - boolean createRepository(String repoTarget, String description, boolean initialize) { - def repoNamespace = repoTarget.split('/', 2)[0] - def repoName = repoTarget.split('/', 2)[1] - def repo = new Repository(repoNamespace, repoName, description ?: "") - Response response = apiClient.repositoryApi().create(repo, initialize).execute() - return handle201or409(response, "Repository ${repoNamespace}/${repoName}") - } - - @Override - void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { - def repoNamespace = repoTarget.split('/', 2)[0] - def repoName = repoTarget.split('/', 2)[1] - - boolean isGroup = (scope == Scope.GROUP) - Permission.Role scmManagerRole = mapToScmManager(role) - def permission = new Permission(principal, scmManagerRole, isGroup) - - Response response = apiClient.repositoryApi().createPermission(repoNamespace, repoName, permission).execute() - handle201or409(response, "Permission on ${repoNamespace}/${repoName}") - } - - @Override - Credentials getCredentials() { - return this.scmmConfig.credentials - } - - @Override - String getGitOpsUsername() { - return scmmConfig.gitOpsUsername - } - - // --- In-cluster / Endpoints --- - /** In-cluster base …/scm (without trailing slash) */ - @Override - String getUrl() { - return urls.inClusterBase().toString() - } - - /** In-cluster repo prefix: …/scm/repo/[] */ - @Override - String repoPrefix() { - return urls.inClusterRepoPrefix() - } - - /** …/scm/repo// */ - @Override - String repoUrl(String repoTarget, RepoUrlScope scope) { - switch (scope) { - case RepoUrlScope.CLIENT: - return urls.clientRepoUrl(repoTarget) - case RepoUrlScope.IN_CLUSTER: - return urls.inClusterRepoUrl(repoTarget) - default: - return urls.inClusterRepoUrl(repoTarget) - } - } - - @Override - String getProtocol() { - return urls.inClusterBase().scheme // e.g. "http" - } - - @Override - String getHost() { - return urls.inClusterBase().host // e.g. "scmm.ns.svc.cluster.local" - } - - /** …/scm/api/v2/metrics/prometheus — client-side, typically scraped externally */ - @Override - URI prometheusMetricsEndpoint() { - return urls.prometheusEndpoint() - } - - /** - * No-op by design. Not used: ScmmDestructionHandler deletes repositories via ScmManagerApiClient. - * Kept for interface compatibility only. */ - @Override - void deleteRepository(String namespace, String repository, boolean prefixNamespace) { - // intentionally left blank - } - - /** - * No-op by design. Not used: ScmmDestructionHandler deletes users via ScmManagerApiClient. - * Kept for interface compatibility only. */ - @Override - void deleteUser(String name) { - // intentionally left blank - } - - /** - * No-op by design. Default branch management is not implemented via this abstraction. - * Kept for interface compatibility only.*/ - @Override - void setDefaultBranch(String repoTarget, String branch) { - // intentionally left blank - } - - // --- helpers --- - private static Permission.Role mapToScmManager(AccessRole role) { - switch (role) { - case AccessRole.READ: return Permission.Role.READ - case AccessRole.WRITE: return Permission.Role.WRITE - case AccessRole.MAINTAIN: - // SCM-manager doesn't know MAINTAIN -> downgrade to WRITE - log.warn("SCM-Manager: Mapping MAINTAIN → WRITE") - return Permission.Role.WRITE - case AccessRole.ADMIN: return Permission.Role.OWNER - case AccessRole.OWNER: return Permission.Role.OWNER - } - } - - private static boolean handle201or409(Response response, String what) { - int code = response.code() - if (code == 409) { - log.debug("${what} already exists — ignoring (HTTP 409)") - return false - } else if (code != 201) { - throw new RuntimeException("Could not create ${what}" + "HTTP Details: ${response.code()} ${response.message()}: ${response.errorBody().string()}") - } - return true // because its created - } - - /** Test-only constructor (package-private on purpose). */ - ScmManager(Config config, ScmManagerConfig scmmConfig, - ScmManagerUrlResolver urls, - ScmManagerApiClient apiClient) { - this.scmmConfig = Objects.requireNonNull(scmmConfig, "scmmConfig must not be null") - this.urls = Objects.requireNonNull(urls, "urls must not be null") - this.apiClient = apiClient ?: new ScmManagerApiClient(urls.clientApiBase().toString(), - scmmConfig.credentials, - Objects.requireNonNull(config, "config must not be null").application.insecure) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolver.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolver.groovy deleted file mode 100644 index 34bc98798..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolver.groovy +++ /dev/null @@ -1,133 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.util.ScmManagerConfig -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.utils.NetworkingUtils -import groovy.util.logging.Slf4j - -@Slf4j -class ScmManagerUrlResolver { - - private final Config config - private final ScmManagerConfig scmm - private final K8sClient k8s - private final NetworkingUtils net - - private URI cachedClusterBind - - private final String releaseName = 'scmm' - - ScmManagerUrlResolver(Config config, ScmManagerConfig scmm, K8sClient k8s, NetworkingUtils net) { - this.config = config - this.scmm = scmm - this.k8s = k8s - this.net = net - } - - // ---------- Public API used by ScmManager ---------- - - /** Client base …/scm (no trailing slash) */ - URI clientBase() { - noTrailSlash(ensureScm(clientBaseRaw())) - } - - /** Client API base …/scm/api/ */ - URI clientApiBase() { - withSlash(clientBase()).resolve("api/") - } - - /** Client repo base …/scm/repo (no trailing slash) */ - URI clientRepoBase() { - noTrailSlash(withSlash(clientBase()).resolve("${root()}/")) - } - - /** In-cluster base …/scm (no trailing slash) */ - URI inClusterBase() { - noTrailSlash(ensureScm(inClusterBaseRaw())) - } - - /** In-cluster repo prefix …/scm/repo/[] */ - String inClusterRepoPrefix() { - def prefix = (config.application.namePrefix ?: "").strip() - def base = withSlash(inClusterBase()) - def url = withSlash(base.resolve(root())) - - return URI.create(url.toString() + prefix).toString() - } - - /** In-cluster repo URL …/scm/repo// */ - String inClusterRepoUrl(String repoTarget) { - def repo = repoTarget.strip() - noTrailSlash(withSlash(inClusterBase()).resolve("${root()}/${repo}/")).toString() - } - - /** Client repo URL …/scm/repo// (no trailing slash) */ - String clientRepoUrl(String repoTarget) { - def repo = repoTarget.strip() - noTrailSlash(withSlash(clientRepoBase()).resolve("${repo}/")).toString() - } - - /** …/scm/api/v2/metrics/prometheus */ - URI prometheusEndpoint() { - withSlash(clientBase()).resolve("api/v2/metrics/prometheus") - } - - // ---------- Base resolution ---------- - - private URI clientBaseRaw() { - if (Boolean.TRUE == scmm.internal) return config.application.runningInsideK8s ? serviceDnsBase() : nodePortBase() - return externalBase() - } - - private URI inClusterBaseRaw() { - return scmm.internal ? serviceDnsBase() : externalBase() - } - - private URI serviceDnsBase() { - def namespace = (scmm.namespace ?: "scm-manager").strip() - URI.create("http://scmm.${namespace}.svc.cluster.local") - } - - private URI externalBase() { - def url = (scmm.url ?: "").strip() - if (url) return URI.create(url) - - def ingress = (scmm.ingress ?: "").strip() - if (ingress) return URI.create("http://${ingress}") - throw new IllegalArgumentException("Either scmm.url or scmm.ingress must be set when internal=false") - } - - private URI nodePortBase() { - if (cachedClusterBind) return cachedClusterBind - - def namespace = (scmm.namespace ?: "scm-manager").strip() - - final def port = k8s.waitForNodePort(releaseName, namespace) - final def host = net.findClusterBindAddress() - cachedClusterBind = new URI("http://${host}:${port}") - return cachedClusterBind - } - - // ---------- Helpers ---------- - - private String root() { - return 'repo' - } - - private static URI ensureScm(URI u) { - def us = withSlash(u) - def path = us.path ?: "" - path.endsWith("/scm/") ? us : us.resolve("scm/") - } - - private static URI withSlash(URI u) { - def s = u.toString() - s.endsWith('/') ? u : URI.create(s + '/') - } - - private static URI noTrailSlash(URI u) { - def s = u.toString() - s.endsWith('/') ? URI.create(s.substring(0, s.length() - 1)) : u - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/AuthorizationInterceptor.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/AuthorizationInterceptor.groovy deleted file mode 100644 index ed88e2435..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/AuthorizationInterceptor.groovy +++ /dev/null @@ -1,25 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import okhttp3.Credentials -import okhttp3.Interceptor -import okhttp3.Response -import org.jetbrains.annotations.NotNull - -class AuthorizationInterceptor implements Interceptor { - private String username - private String password - - AuthorizationInterceptor(String username, String password) { - this.username = username - this.password = password - } - - @Override - Response intercept(@NotNull Chain chain) throws IOException { - def newRequest = chain.request().newBuilder() - .header("Authorization", Credentials.basic(username, password)) - .build() - - return chain.proceed(newRequest) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/PluginApi.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/PluginApi.groovy deleted file mode 100644 index 6b4c9fe19..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/PluginApi.groovy +++ /dev/null @@ -1,13 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import retrofit2.Call -import retrofit2.http.* - -interface PluginApi { - @POST("v2/plugins/available/{name}/install") - Call install(@Path("name") String name, @Query("restart") Boolean restart) - - @PUT("v2/config/jenkins/") - @Headers("Content-Type: application/json") - Call configureJenkinsPlugin(@Body Map config) -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/Repository.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/Repository.groovy deleted file mode 100644 index 3c2d2a7de..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/Repository.groovy +++ /dev/null @@ -1,26 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -class Repository { - final String name - final String namespace - final String type - final String contact - final String description - - Repository(String namespace, String name, String description = null, String contact = null, String type = 'git') { - this.namespace = namespace - this.name = name - this.type = type - this.contact = contact - this.description = description - } - - String getFullRepoName() { - return "${namespace}/${name}" - } - - @Override - String toString() { - "Repository{name='$name', namespace='$namespace', type='$type', contact='$contact', description='$description'}" - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/RepositoryApi.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/RepositoryApi.groovy deleted file mode 100644 index dbd834713..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/RepositoryApi.groovy +++ /dev/null @@ -1,18 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.Permission -import retrofit2.Call -import retrofit2.http.* - -interface RepositoryApi { - @DELETE("v2/repositories/{namespace}/{name}") - Call delete(@Path("namespace") String namespace, @Path("name") String name) - - @POST("v2/repositories/") - @Headers("Content-Type: application/vnd.scmm-repository+json;v=2") - Call create(@Body Repository repository, @Query("initialize") boolean initialize) - - @POST("v2/repositories/{namespace}/{name}/permissions/") - @Headers("Content-Type: application/vnd.scmm-repositoryPermission+json") - Call createPermission(@Path("namespace") String namespace, @Path("name") String name, @Body Permission permission) -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApi.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApi.groovy deleted file mode 100644 index 88f4f2b44..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApi.groovy +++ /dev/null @@ -1,17 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import retrofit2.Call -import retrofit2.http.Body -import retrofit2.http.GET -import retrofit2.http.Headers -import retrofit2.http.PUT - -interface ScmManagerApi { - - @GET("v2") - Call checkScmmAvailable() - - @PUT("v2/config") - @Headers("Content-Type: application/vnd.scmm-config+json;v=2") - Call setConfig(@Body Map config) -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApiClient.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApiClient.groovy deleted file mode 100644 index 327d8efef..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApiClient.groovy +++ /dev/null @@ -1,71 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.dependencyinjection.HttpClientFactory -import groovy.util.logging.Slf4j -import okhttp3.OkHttpClient -import retrofit2.Call -import retrofit2.Response -import retrofit2.Retrofit -import retrofit2.converter.jackson.JacksonConverterFactory - -/** - * Parent class for all SCMM Apis that lazily creates the APIs*/ -@Slf4j -class ScmManagerApiClient { - Credentials credentials - OkHttpClient okHttpClient - String url - - ScmManagerApiClient(String url, Credentials credentials, Boolean isInsecure) { - this.url = url - this.credentials = credentials - this.okHttpClient = HttpClientFactory.buildOkHttpClient(credentials, isInsecure) - } - - UsersApi usersApi() { - return retrofit().create(UsersApi) - } - - RepositoryApi repositoryApi() { - return retrofit().create(RepositoryApi) - } - - ScmManagerApi generalApi() { - return retrofit().create(ScmManagerApi) - } - - PluginApi pluginApi() { - return retrofit().create(PluginApi) - } - - static handleApiResponse(Call apiCall, String additionalMessage = "") { - try { - Response response = apiCall.execute() - - if (!response.isSuccessful() && response.code() != 409 && response.code() != 201) { - def errorMessage = "API call failed!'. HTTP Status: ${response.code()} - ${response.message()}" - if (additionalMessage) { - errorMessage += " Additional Info: ${additionalMessage}" - } - log.error(errorMessage) - throw new RuntimeException(errorMessage) - } else { - log.debug("Successfully completed ${apiCall}") - } - } catch (Exception e) { - def errorMessage = "Error executing API: ${e.message}" - log.error(errorMessage, e) - throw new RuntimeException(errorMessage, e) - } - } - - protected Retrofit retrofit() { - return new Retrofit.Builder() - .baseUrl(this.url) - .client(okHttpClient) - // Converts HTTP body objects from groovy to JSON - .addConverterFactory(JacksonConverterFactory.create()) - .build() - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerUser.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerUser.groovy deleted file mode 100644 index cffa6b7b5..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerUser.groovy +++ /dev/null @@ -1,11 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -class ScmManagerUser { - String name - String displayName - String mail - boolean external = false - String password - boolean active = true - Map _links = [:] -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApi.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApi.groovy deleted file mode 100644 index f7e918b27..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApi.groovy +++ /dev/null @@ -1,18 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import retrofit2.Call -import retrofit2.http.* - -interface UsersApi { - @DELETE("v2/users/{id}") - Call delete(@Path("id") String id) - - @Headers(["Content-Type: application/vnd.scmm-user+json;v=2"]) - @POST("v2/users") - Call addUser(@Body ScmManagerUser user) - - @Headers(["Content-Type: application/vnd.scmm-permissionCollection+json;v=2"]) - @PUT("v2/users/{username}/permissions") - Call setPermissionForUser(@Path("username") String username, - @Body Map> permissions) -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/helm/HelmClient.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/helm/HelmClient.groovy deleted file mode 100644 index 39ee9d691..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/helm/HelmClient.groovy +++ /dev/null @@ -1,51 +0,0 @@ -package com.cloudogu.gitops.infrastructure.helm - -import com.cloudogu.gitops.utils.CommandExecutor -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton - -@Slf4j -@Singleton -class HelmClient { - - private CommandExecutor commandExecutor - - HelmClient(CommandExecutor commandExecutor) { - this.commandExecutor = commandExecutor - } - - String addRepo(String repoName, String url) { - helm(['repo', 'add', repoName, url]) - } - - String dependencyBuild(String path) { - helm(['dependency', 'build', path]) - } - - String upgrade(String release, String chartOrPath, Map args = [:]) { - helm(['upgrade', '-i', release, chartOrPath, '--create-namespace'], args) - } - - String template(String release, String chartOrPath, Map args = [:]) { - helm(['template', release, chartOrPath], args) - } - - String uninstall(String release, String namespace) { - String[] command = ["helm", "uninstall", release, '--namespace', namespace] - commandExecutor.execute(command).stdOut - } - - private String helm(List verbAndParams, Map args = [:]) { - List command = ['helm'] + verbAndParams - - for (entry in args) { - String key = entry.key - String value = entry.value - command += "--${key}".toString() - command += value - } - - log.trace("Executing helm command: ${command.join(' ')}") - commandExecutor.execute(command as String[]).stdOut - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManager.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManager.groovy deleted file mode 100644 index 7b5215831..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManager.groovy +++ /dev/null @@ -1,67 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import jakarta.inject.Singleton -import org.intellij.lang.annotations.Language - -@Singleton -class GlobalPropertyManager { - private JenkinsApiClient apiClient - - GlobalPropertyManager(JenkinsApiClient apiClient) { - this.apiClient = apiClient - } - - void setGlobalProperty(String key, String value) { - @Language("groovy") - def script = """ - instance = Jenkins.getInstance() - globalNodeProperties = instance.getGlobalNodeProperties() - envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) - - def newEnvVarsNodeProperty - def envVars - - if ( envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0 ) { - newEnvVarsNodeProperty = new hudson.slaves.EnvironmentVariablesNodeProperty() - globalNodeProperties.add(newEnvVarsNodeProperty) - envVars = newEnvVarsNodeProperty.getEnvVars() - } else { - envVars = envVarsNodePropertyList.get(0).getEnvVars() - - } - - envVars.put("$key", "$value") - - instance.save() - print("Done") - """ - - def result = apiClient.runScript(script) - if (result != 'Done') { - throw new RuntimeException("Could not create global property: $result") - } - } - - void deleteGlobalProperty(String key) { - @Language("groovy") - def script = """ - def instance = Jenkins.getInstance() - def globalNodeProperties = instance.getGlobalNodeProperties() - def envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) - - if (envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0) { - print("Nothing to do") - return - } - - envVars = envVarsNodePropertyList.get(0).getEnvVars() - envVars.remove("$key") - print("Done") - """ - - def result = apiClient.runScript(script) - if (result != 'Nothing to do' && result != 'Done') { - throw new RuntimeException("Could not delete global property: $result") - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClient.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClient.groovy deleted file mode 100644 index 271dbbf73..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClient.groovy +++ /dev/null @@ -1,114 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import com.cloudogu.gitops.config.Config -import groovy.json.JsonSlurper -import groovy.util.logging.Slf4j -import jakarta.inject.Named -import jakarta.inject.Singleton -import okhttp3.* - -@Slf4j -@Singleton -class JenkinsApiClient { - private Config config - - private OkHttpClient client - - // Number of retries in uncommonly high, because we might have to outlive a unexpected Jenkins restart - private int maxRetries = 180 - private int waitPeriodInMs = 2000 - - JenkinsApiClient(Config config, - @Named("jenkins") OkHttpClient client) { - - if (config.application.insecure) { - this.client = client.newBuilder() - .hostnameVerifier({ hostname, session -> true }) - .build() - } else { - this.client = client - } - this.config = config - } - - String runScript(String code) { - log.trace("Running groovy script in Jenkins: {}", code) - def response = postRequestWithCrumb("scriptText", new FormBody.Builder().add("script", code).build()) - if (response.code() != 200) { - throw new RuntimeException("Could not run script. Status code ${response.code()}") - } - - return response.body().string() - } - - Response postRequestWithCrumb(String url, RequestBody postData = null) { - return sendRequestWithRetries { - Request.Builder request = buildRequest(url) - .header("Jenkins-Crumb", getCrumb()) - - if (postData != null) { - request.method("POST", postData) - } else { - // Explicitly set empty body. Otherwise okhttp sends GET - RequestBody emptyBody = RequestBody.create("", null) - request.method("POST", emptyBody) - } - - request.build() - } - } - - private String getCrumb() { - log.trace("Getting Crumb for Jenkins") - def response = sendRequestWithRetries { buildRequest("crumbIssuer/api/json").build() } - - if (response.code() != 200) { - throw new RuntimeException("Could not create crumb. Status code ${response.code()}") - } - - def json = new JsonSlurper().parse(response.body().byteStream()) - - if (!json instanceof Map || !(json as Map).containsKey('crumb')) { - throw new RuntimeException("Could not create crumb. Invalid json.") - } - - return json['crumb'] - } - - private Request.Builder buildRequest(String url) { - return new Request.Builder() - .url("${config.jenkins.url}/$url") - .header("Authorization", Credentials.basic(config.jenkins.username, config.jenkins.password)) - } - - // We pass a closure, so that we actually refetch a new crumb for a failed request - // The Jenkins ApiClient has it's own retry logic on top of RetryInterceptor, because of crumb lifetime and restarts - private Response sendRequestWithRetries(Closure request) { - def retry = 0 - Response response = null - do { - response = client.newCall(request()).execute() - if (!shouldRetryRequest(response)) { - break - } - Thread.sleep(waitPeriodInMs) - } while (++retry < maxRetries) - - return response - } - - private boolean shouldRetryRequest(Response response) { - // We might run into a 403 due to an invalid crumb from a previous session before jenkins was restarted. - // Here in the ApiClient, we simply retry all 401 and 403 including fetching a new crumb - return response.code() in [401, 403] - } - - protected void setMaxRetries(int retries) { - this.maxRetries = retries - } - - protected setWaitPeriodInMs(int waitPeriodInMs) { - this.waitPeriodInMs = waitPeriodInMs - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/JobManager.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/JobManager.groovy deleted file mode 100644 index 239b40ef7..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/JobManager.groovy +++ /dev/null @@ -1,90 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import com.cloudogu.gitops.utils.TemplatingEngine -import groovy.json.JsonOutput -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton -import okhttp3.FormBody -import okhttp3.MediaType -import okhttp3.RequestBody -import org.intellij.lang.annotations.Language - -@Singleton -@Slf4j -class JobManager { - private JenkinsApiClient apiClient - - JobManager(JenkinsApiClient apiClient) { - this.apiClient = apiClient - } - - void createCredential(String jobName, String id, String username, String password, String description) { - def response = apiClient.postRequestWithCrumb("job/$jobName/credentials/store/folder/domain/_/createCredentials", - new FormBody.Builder() - .add("json", JsonOutput.toJson([credentials: [scope : "GLOBAL", - id : id, - username : username, - password : password, - description: description, - $class : "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl",]])) - .build()) - - if (response.code() != 200) { - throw new RuntimeException("Could not create credential id=$id,job=$jobName. StatusCode: ${response.code()}") - } - } - - /** - * @return true, if created; false if job already exists and nothing was changed. - */ - boolean createJob(String name, String serverUrl, String jobNamespace, String credentialsId) { - if (jobExists(name)) { - log.warn("Job '${name}' already exists, ignoring.") - return false - } else { - // Note for development: the XML representation of an existing job can be exporting by adding /config.xml to the URL - String payloadXml = new TemplatingEngine().template(new File('argocd/cluster-resources/apps/jenkins/templates/namespaceJobTemplate.xml.ftl'), - [SCMM_NAMESPACE_JOB_SERVER_URL : serverUrl, - SCMM_NAMESPACE_JOB_NAMESPACE : jobNamespace, - SCMM_NAMESPACE_JOB_CREDENTIALS_ID: credentialsId]) - - RequestBody body = RequestBody.create(payloadXml, MediaType.get("text/xml")) - - def response = apiClient.postRequestWithCrumb("createItem?name=$name", body) - - if (response.code() != 200) { - throw new RuntimeException("Could not create job '${name}'. StatusCode: ${response.code()}") - } - } - return true - } - - boolean jobExists(String name) { - def response = apiClient.postRequestWithCrumb("job/$name") - - return response.code() == 200 - } - - void deleteJob(String name) { - if (name.contains("'")) { - throw new RuntimeException('Job name cannot contain quotes.') - } - - @Language("groovy") - String script = "print(Jenkins.instance.getItem('$name')?.delete())" - def result = apiClient.runScript(script) - - if (result != 'null') { - throw new RuntimeException("Could not delete job $name") - } - } - - void startJob(String jobName) { - - def response = apiClient.postRequestWithCrumb("job/$jobName/build?delay=0sec") - - if (response.code() != 200) { - throw new RuntimeException("Could not trigger build of Jenkins job: $jobName. StatusCode: ${response.code()}") - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/PrometheusConfigurator.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/PrometheusConfigurator.groovy deleted file mode 100644 index 4eb3466d0..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/PrometheusConfigurator.groovy +++ /dev/null @@ -1,27 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import jakarta.inject.Singleton - -@Singleton -class PrometheusConfigurator { - private final JenkinsApiClient apiClient - - PrometheusConfigurator(JenkinsApiClient apiClient) { - this.apiClient = apiClient - } - - void enableAuthentication() { - def result = apiClient.runScript(""" - import org.jenkinsci.plugins.prometheus.config.* - - def config = Jenkins.instance.getDescriptor(PrometheusConfiguration) - config.setUseAuthenticatedEndpoint(true) - - print(config.useAuthenticatedEndpoint) - """) - - if (result != "true") { - throw new RuntimeException("Cannot enable authentication for prometheus: $result") - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/UserManager.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/UserManager.groovy deleted file mode 100644 index 9a9ab0113..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/jenkins/UserManager.groovy +++ /dev/null @@ -1,104 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton -import org.intellij.lang.annotations.Language - -@Singleton -@Slf4j -class UserManager { - private JenkinsApiClient apiClient - - UserManager(JenkinsApiClient apiClient) { - this.apiClient = apiClient - } - - void createUser(String username, String password) { - log.debug("Add user $username to jenkins") - - @Language("Groovy") - def script = """ - def realm = Jenkins.getInstance().getSecurityRealm() - def user = realm.createAccount('${escapeString(username)}', '${escapeString(password)}') - - print(user) - """ - - def result = apiClient.runScript(script) - - if (result != username) { - throw new RuntimeException("Error when creating user: $result") - } - } - - void grantPermission(String username, Permissions permission) { - if (!isUsingMatrixBasedPermissions()) { - log.debug("Is not using matrix based permission. Does not need to add permission.") - return - } - - log.debug("Grant user $username permission $permission") - - @Language("Groovy") - def script = """ - import org.jenkinsci.plugins.matrixauth.PermissionEntry - import org.jenkinsci.plugins.matrixauth.AuthorizationType - - def permissions = Jenkins.getInstance().getAuthorizationStrategy().getGrantedPermissionEntries() - permissions.computeIfAbsent(${permission.toJenkinsPermissionEnum()}) { - new HashSet<>() - } - print(permissions[${permission.toJenkinsPermissionEnum()}].add(new PermissionEntry(AuthorizationType.USER, '${escapeString(username)}'))) - """ - def result = apiClient.runScript(script) - - if (result !in ["true", "false"]) { - // Both are valid return values for Set.add(). true == was already in set, false == was not already in set - throw new RuntimeException("Failed to add permission $permission to $username: $result") - } - } - - boolean isUsingMatrixBasedPermissions() { - def result = apiClient.runScript("print(Jenkins.getInstance().getAuthorizationStrategy().class)") - - if (!result.startsWith("class ")) { - throw new RuntimeException("Error when trying to determine authorization strategy: $result") - } - - return result == "class hudson.security.GlobalMatrixAuthorizationStrategy" || result == "class hudson.security.ProjectMatrixAuthorizationStrategy" - } - - boolean isUsingCasSecurityRealm() { - def result = apiClient.runScript("print(Jenkins.getInstance().getSecurityRealm().class)") - - if (!result.startsWith("class ")) { - throw new RuntimeException("Error when trying to determine security realm: $result") - } - - return result == "class org.jenkinsci.plugins.cas.CasSecurityRealm" - } - - private String escapeString(String str) { - if (str.contains("\\")) { - // We don't want get in trouble with escaping, - // e.g. `foo\'foo` => `foo\\'foo`. Now we would have a backslash followed by an unescaped quote. - throw new IllegalArgumentException("Backslashes within the escaped variables are forbidden.") - } - - return str.replace("'", "\\'") - } - - enum Permissions { - METRICS_VIEW("jenkins.metrics.api.Metrics.VIEW") - - private final String value - - Permissions(String value) { - this.value = value - } - - String toJenkinsPermissionEnum() { - return value - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClient.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClient.groovy deleted file mode 100644 index c3274a708..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClient.groovy +++ /dev/null @@ -1,1345 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.api - -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.utils.MapUtils - -import jakarta.inject.Singleton -import groovy.io.FileType -import groovy.json.JsonBuilder -import groovy.json.JsonSlurper -import groovy.transform.CompileStatic -import groovy.transform.Immutable -import groovy.transform.TypeCheckingMode -import groovy.util.logging.Slf4j - -import io.fabric8.kubernetes.api.model.* -import io.fabric8.kubernetes.client.Config -import io.fabric8.kubernetes.client.ConfigBuilder -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.KubernetesClientBuilder -import io.fabric8.kubernetes.client.dsl.base.PatchContext -import io.fabric8.kubernetes.client.dsl.base.PatchType -import io.fabric8.kubernetes.client.dsl.base.ResourceDefinitionContext -import io.fabric8.kubernetes.client.utils.Serialization -import io.fabric8.openshift.api.model.Project -import io.fabric8.openshift.api.model.ProjectBuilder -import io.fabric8.openshift.client.OpenShiftClient - -/** - * Kubernetes client using Fabric8 Kubernetes Client.*/ -@Slf4j -@Singleton -class K8sClient { - - // ======================================== - // Constants - // ======================================== - - private static final String DEFAULT_NAMESPACE = "default" - private static final String INTERNAL_IP_TYPE = "InternalIP" - private static final String DOCKER_CONFIG_JSON_TYPE = "kubernetes.io/dockerconfigjson" - private static final String DOCKER_CONFIG_JSON_KEY = ".dockerconfigjson" - - private static final int DEFAULT_TIMEOUT_SECONDS = 60 - private static final int DEFAULT_CHECK_INTERVAL_SECONDS = 1 - private static final int FABRIC8_REQUEST_TIMEOUT_MILLIS = 60_000 - private static final int FABRIC8_CONNECTION_TIMEOUT_MILLIS = 10_000 - - // ======================================== - // Instance Variables - // ======================================== - - protected int SLEEPTIME = 1000 - protected int DEFAULT_RETRIES = 120 - - KubernetesClient client - com.cloudogu.gitops.config.Config gopConfig - - K8sClient(com.cloudogu.gitops.config.Config gopConfig = null) { - Config config = new ConfigBuilder() - .withRequestTimeout(FABRIC8_REQUEST_TIMEOUT_MILLIS) - .withConnectionTimeout(FABRIC8_CONNECTION_TIMEOUT_MILLIS) - .build() - - this.client = new KubernetesClientBuilder() - .withConfig(config) - .build() - /* OpenShift client includes Kubernetes client APIs. */ - this.gopConfig = gopConfig - } - - // ======================================== - // Public API Methods - Node Operations - // ======================================== - - /** - * Waits for the first node in the cluster to become available. - * - * @return The name of the first available node (e.g., "k3d-gitops-playground-server-0") - * @throws RuntimeException if no node becomes available within the retry limit - */ - String waitForNode() { - log.debug("Waiting for first node of the cluster to become ready") - - String nodeName = waitForResourceWithRetry("node") { -> - NodeList nodes = client.nodes().list() - if (nodes?.items && !nodes.items.isEmpty()) { - return nodes.items[0].metadata.name - } - return null - } - - log.debug("First node of the cluster is ready: $nodeName") - return nodeName - } - - /** - * Waits for and retrieves the internal IP address of the first node. - * For k3d, this is either the host's IP or the k3d API server's container IP. - * - * @return The internal IP address of the node (IPv4) - * @throws RuntimeException if the internal IP cannot be retrieved - */ - String waitForInternalNodeIp() { - String nodeName = waitForNode() - log.debug("Waiting for internal IP of node $nodeName") - - String internalIp = waitForResourceWithRetry("internal IP of node $nodeName") { -> - Node node = client.nodes().withName(nodeName).get() - if (node?.status?.addresses) { - def internalIpAddress = node.status.addresses.find { it.type == INTERNAL_IP_TYPE } - return internalIpAddress?.address - } - return null - } - - log.debug("Internal IP of node $nodeName: $internalIp") - return internalIp - } - - // ======================================== - // Public API Methods - Service Operations - // ======================================== - - /** - * Waits for a service's NodePort to become available. - * - * @param serviceName The name of the service - * @param namespace The namespace of the service - * @return The NodePort as a string - * @throws RuntimeException if the NodePort cannot be retrieved - */ - String waitForNodePort(String serviceName, String namespace) { - log.debug("Getting node port for service $serviceName, ns=$namespace") - - String nodePort = waitForResourceWithRetry("node port for service $serviceName") { -> - Service service = client.services().inNamespace(namespace).withName(serviceName).get() - if (service?.spec?.ports && !service.spec.ports.isEmpty()) { - Integer port = service.spec.ports[0].nodePort - return port?.toString() - } - return null - } - - log.debug("Node port for service $serviceName, ns=$namespace: $nodePort") - return nodePort - } - - /** - * Creates a NodePort service (idempotent). - * - * @param name The name of the service - * @param tcp Port pairs specified as ':' - * @param nodePort The NodePort (optional) - * @param namespace The namespace (defaults to "default") - */ - void createServiceNodePort(String name, String tcp, String nodePort = '', String namespace = '') { - log.debug("Creating NodePort service $name in namespace $namespace") - - def ports = tcp.split(':') - int port = Integer.parseInt(ports[0]) - int targetPort = ports.size() > 1 ? Integer.parseInt(ports[1]) : port - - def portBuilder = new ServiceBuilder() - .withNewMetadata() - .withName(name) - .withNamespace(resolveNamespace(namespace)) - .endMetadata() - .withNewSpec() - .withType("NodePort") - .addNewPort() - .withPort(port) - .withTargetPort(new IntOrString(targetPort)) - - if (nodePort) { - portBuilder = portBuilder.withNodePort(Integer.parseInt(nodePort)) - } - - Service service = portBuilder - .endPort() - .endSpec() - .build() - - executeWithErrorHandling("create NodePort service $name") { - client.services() - .inNamespace(resolveNamespace(namespace)) - .resource(service) - .createOrReplace() - } - - log.debug("NodePort service $name created/updated successfully") - } - - /** - * Patches the nodePort of a specific port in a service. - * - * @param serviceName The name of the service to patch - * @param namespace The namespace of the service - * @param portName The name of the port to patch - * @param newNodePort The new nodePort value to set - * @throws IllegalArgumentException if parameters are invalid - * @throws RuntimeException if the port is not found or patching fails - */ - void patchServiceNodePort(String serviceName, String namespace, String portName, int newNodePort) { - validateServiceNodePortPatch(serviceName, namespace, portName, newNodePort) - - log.debug("Patching service $serviceName port $portName with nodePort $newNodePort") - - Service service = client.services().inNamespace(namespace).withName(serviceName).get() - - if (!service) { - throw new RuntimeException("Service ${serviceName} not found in namespace ${namespace}") - } - - def ports = service.spec.ports - def portIndex = ports.findIndexOf { it.name == portName } - - if (portIndex == -1) { - throw new RuntimeException("Port with name ${portName} not found in service ${serviceName}.") - } - - // Create JSON patch - def patch = [[op : "replace", - path : "/spec/ports/${portIndex}/nodePort", - value: newNodePort]] - - String patchJson = new JsonBuilder(patch).toString() - PatchContext patchContext = new PatchContext.Builder() - .withPatchType(PatchType.JSON) - .build() - - executeWithErrorHandling("patch service $serviceName") { - client.services() - .inNamespace(namespace) - .withName(serviceName) - .patch(patchContext, patchJson) - } - - log.debug("Service ${serviceName} in namespace ${namespace} successfully patched with nodePort ${newNodePort} for port ${portName}.") - } - - // ======================================== - // Public API Methods - Namespace Operations - // ======================================== - - /** - * Creates a namespace if it does not already exist (idempotent). - * - * @param name The name of the namespace to create - * @throws IllegalArgumentException if name is null or empty - * @throws RuntimeException if creation fails - */ - void createNamespace(String name) { - validateNamespaceName(name) - - if (!namespaceExists(name)) { - log.debug("Namespace ${name} does not exist, proceeding to create.") - - if (runInOpenshift()) { - OpenShiftClient osClient = client.adapt(OpenShiftClient.class) - - Project project = new ProjectBuilder() - .withNewMetadata() - .withName(name) - .endMetadata() - .build() - executeWithErrorHandling("create project ${name}") { - osClient.projects().resource(project).create() - } - log.debug("Project ${name} created successfully.") - } else { - - Namespace namespace = new NamespaceBuilder() - .withNewMetadata() - .withName(name) - .endMetadata() - .build() - - executeWithErrorHandling("create namespace ${name}") { - client.namespaces().resource(namespace).create() - } - - log.debug("Namespace ${name} created successfully.") - } - } - } - - /** - * Creates multiple namespaces. - * - * @param names List of namespace names to create - * @throws IllegalArgumentException if names is null - */ - void createNamespaces(List names) { - if (names == null) { - throw new IllegalArgumentException("Namespaces must be provided and cannot be null.") - } - names.each { name -> createNamespace(name) } - } - - /** - * Checks if a namespace exists. - * - * @param namespace The namespace name - * @return true if the namespace exists, false otherwise - */ - boolean namespaceExists(String namespace) { - try { - Namespace ns = client.namespaces().withName(namespace).get() - if (ns != null) { - log.debug("Namespace ${namespace} already exists.") - return true - } - } catch (Exception e) { - log.trace("Namespace ${namespace} does not exist: ${e.message}") - } - return false - } - - // ======================================== - // Public API Methods - Secret Operations - // ======================================== - - /** - * Creates or updates a generic secret (idempotent). - * - * @param type The type of secret - * @param name The name of the secret - * @param namespace The namespace (defaults to "default") - * @param literals Key-value pairs as Tuple2 - */ - void createSecret(String type, String name, String namespace = '', Tuple2... literals) { - log.debug("Creating secret $name of type $type in namespace $namespace") - - Map data = [:] - literals.each { tuple -> data[tuple.v1 as String] = tuple.v2 as String - } - - String resolvedType = type == 'generic' ? 'Opaque' : type - Secret secret = new SecretBuilder() - .withNewMetadata() - .withName(name) - .withNamespace(resolveNamespace(namespace)) - .endMetadata() - .withType(resolvedType) - .withStringData(data) - .build() - - executeWithErrorHandling("create secret $name") { - def secretsClient = client.secrets().inNamespace(resolveNamespace(namespace)) - if (secretsClient.withName(name).get()) { - secretsClient.withName(name).delete() - } - secretsClient.resource(secret).create() - } - - log.debug("Secret $name created/updated successfully") - } - - /** - * Creates or updates an image pull secret (idempotent). - * - * @param name The name of the secret - * @param namespace The namespace (defaults to "default") - * @param host The Docker registry host - * @param user The username - * @param password The password - */ - void createImagePullSecret(String name, String namespace = '', String host, String user, String password) { - log.debug("Creating image pull secret $name in namespace $namespace") - - String auth = Base64.encoder.encodeToString("${user}:${password}".bytes) - String dockerConfig = """{"auths":{"${host}":{"username":"${user}","password":"${password}","auth":"${auth}"}}}""" - - Secret secret = new SecretBuilder() - .withNewMetadata() - .withName(name) - .withNamespace(resolveNamespace(namespace)) - .endMetadata() - .withType(DOCKER_CONFIG_JSON_TYPE) - .addToStringData(DOCKER_CONFIG_JSON_KEY, dockerConfig) - .build() - - executeWithErrorHandling("create image pull secret $name") { - client.secrets() - .inNamespace(resolveNamespace(namespace)) - .resource(secret) - .createOrReplace() - } - - log.debug("Image pull secret $name created/updated successfully") - } - - /** - * Retrieves the 'namespaces' data from an ArgoCD secret. - * - * @param name The name of the secret - * @param namespace The namespace (defaults to "default") - * @return The base64-encoded namespaces data - * @throws RuntimeException if the secret or data cannot be retrieved - */ - String getArgoCDNamespacesSecret(String name, String namespace = '') { - log.debug("Getting Secret $name from namespace $namespace") - - String secretData = waitForResourceWithRetry("secret $name") { -> - Secret secret = client.secrets() - .inNamespace(resolveNamespace(namespace)) - .withName(name) - .get() - - return secret?.data?.containsKey('namespaces') ? secret.data['namespaces'] : null - } - - return secretData - } - - /** - * Extracts credentials from a Kubernetes secret. - * - * @param secretname The name of the secret - * @param namespace The namespace - * @param usernameKey The key for username (defaults to 'username') - * @param passwordKey The key for password (defaults to 'password') - * @return Credentials object containing username and password - * @throws RuntimeException if the secret cannot be parsed - */ - Credentials getCredentialsFromSecret(String secretname, String namespace, String usernameKey = 'username', String passwordKey = 'password') { - executeWithErrorHandling("get credentials from secret ${secretname}") { - Secret secret = client.secrets() - .inNamespace(namespace) - .withName(secretname) - .get() - - def secretData = secret.getData() - String username = new String(Base64.getDecoder().decode(secretData[usernameKey])) - String password = new String(Base64.getDecoder().decode(secretData[passwordKey])) - return new Credentials(username, password) - } - } - - /** - * Extracts credentials from a Kubernetes secret using a Credentials object as input. - * - * @param credentials Credentials object with secret location information - * @return Updated Credentials object with username and password - * @throws RuntimeException if the secret cannot be parsed - */ - Credentials getCredentialsFromSecret(Credentials credentials) { - executeWithErrorHandling("get credentials from secret ${credentials.secretName}") { - Secret secret = client.secrets() - .inNamespace(credentials.secretNamespace) - .withName(credentials.secretName) - .get() - - def secretData = secret.getData() - def usernameEncoded = secretData[credentials.usernameKey] - String username = usernameEncoded != null ? new String(Base64.decoder.decode(usernameEncoded)) : credentials.username - String password = new String(Base64.getDecoder().decode(secretData[credentials.passwordKey])) - - Credentials credentialsNew = new Credentials(credentials) - credentialsNew.username = username - credentialsNew.password = password - - return credentialsNew - } - } - - // ======================================== - // Public API Methods - ConfigMap Operations - // ======================================== - - /** - * Creates or updates a ConfigMap from a file (idempotent). - * - * @param name The name of the ConfigMap - * @param namespace The namespace (defaults to "default") - * @param filePath The path to the file - * @throws RuntimeException if the file is not found - */ - void createConfigMapFromFile(String name, String namespace = '', String filePath) { - log.debug("Creating ConfigMap $name from file $filePath in namespace $namespace") - - File file = new File(filePath) - if (!file.exists()) { - throw new RuntimeException("File not found: $filePath") - } - - Map data = [(file.name): file.text] - - ConfigMap configMap = new ConfigMapBuilder() - .withNewMetadata() - .withName(name) - .withNamespace(resolveNamespace(namespace)) - .endMetadata() - .withData(data) - .build() - - executeWithErrorHandling("create ConfigMap $name from file") { - client.configMaps() - .inNamespace(resolveNamespace(namespace)) - .resource(configMap) - .createOrReplace() - } - - log.debug("ConfigMap $name created/updated successfully") - } - - /** - * Retrieves a value from a ConfigMap. - * - * @param mapName The name of the ConfigMap - * @param key The key to retrieve - * @return The value associated with the key - * @throws RuntimeException if the ConfigMap or key is not found - */ - String getConfigMap(String mapName, String key) { - log.debug("Getting ConfigMap $mapName, key: $key") - - ConfigMap configMap = client.configMaps().inNamespace(DEFAULT_NAMESPACE).withName(mapName).get() - - if (!configMap) { - throw new RuntimeException("Could not fetch configmap $mapName") - } - - if (!configMap.data?.containsKey(key)) { - throw new RuntimeException("Could not fetch $key within config-map $mapName") - } - - return configMap.data[key] - } - - // ======================================== - // Public API Methods - Resource Management - // ======================================== - - /** - * Applies YAML resources from a file. - * - * @param yamlLocation The path to the YAML file - * @return A success message - * @throws RuntimeException if the file is not found or application fails - */ - String applyYaml(String yamlLocation) { - log.debug("Applying YAML from $yamlLocation") - - if (yamlLocation.startsWith("http://") || yamlLocation.startsWith("https://")) { - int appliedResources = applyYamlStream(new URL(yamlLocation).openStream(), yamlLocation) - return "Applied ${appliedResources} resource(s) from $yamlLocation" - } - - File location = new File(yamlLocation) - - if (!location.exists()) { - throw new RuntimeException("File or directory not found: $yamlLocation") - } - - if (location.isDirectory()) { - List yamlFiles = [] - location.traverse(type: FileType.FILES) { File file -> - if (file.name.endsWith(".yaml") || file.name.endsWith(".yml")) { - yamlFiles.add(file) - } - } - - yamlFiles = yamlFiles.sort { it.absolutePath } - - int appliedResources = 0 - yamlFiles.each { File file -> - appliedResources += applyYamlStream(file.newInputStream(), - file.absolutePath) - } - - return "Applied ${appliedResources} resource(s) from directory $yamlLocation" - } - - int appliedResources = applyYamlStream(location.newInputStream(), yamlLocation) - return "Applied ${appliedResources} resource(s) from $yamlLocation" - } - - private int applyYamlStream(InputStream stream, String sourceDescription) { - def resources = executeWithErrorHandling("load YAML from $sourceDescription") { - try { - return client.load(stream).items() - } finally { - stream.close() - } - } - - resources.each { resource -> - executeWithErrorHandling("apply resource from $sourceDescription") { - def resourceClient = client.resource(resource) - - if (resource.metadata?.namespace) { - resourceClient = resourceClient.inNamespace(resource.metadata.namespace) - } - - resourceClient.createOrReplace() - } - } - - return resources.size() - } - - /** - * Adds or updates labels on a resource. - * - * @param resource The resource type (e.g., "pod", "service") - * @param name The name of the resource - * @param namespace The namespace (defaults to "default") - * @param keyValues Label key-value pairs as Tuple2. Keys ending with '-' will be removed. - */ - @CompileStatic(TypeCheckingMode.SKIP) - void label(String resource, String name, String namespace = '', Tuple2... keyValues) { - if (!keyValues) { - throw new RuntimeException("Missing key-value-pairs") - } - - if (name == '--all') { - client.nodes().list().items.each { node -> label(resource, node.metadata.name, namespace, keyValues) - } - return - } - - log.debug("Labeling $resource/$name in namespace $namespace") - - Map labelsToAdd = [:] - List labelsToRemove = [] - - keyValues.each { tuple -> - String key = tuple.v1 as String - String value = tuple.v2 as String - - if (key.endsWith('-')) { - labelsToRemove.add(key.substring(0, key.length() - 1)) - } else { - labelsToAdd[key] = value - } - } - - executeWithErrorHandling("label $resource/$name") { - def resourceClient = getResourceClient(resource, name, namespace) - HasMetadata existingResource = resourceClient.get() as HasMetadata - - if (!existingResource) { - throw new RuntimeException("Resource $resource/$name not found") - } - - def existingLabels = existingResource.metadata?.labels ?: [:] - labelsToRemove.each { key -> existingLabels.remove(key) } - existingLabels.putAll(labelsToAdd) - - existingResource.metadata.labels = existingLabels - resourceClient.replace(existingResource) - } - - log.debug("Labels updated successfully") - } - - /** - * Removes labels from a resource. - * - * @param resource The resource type - * @param name The name of the resource - * @param namespace The namespace (defaults to "default") - * @param keys The label keys to remove - */ - void labelRemove(String resource, String name, String namespace = '', String... keys) { - Tuple2[] tuples = keys.collect { new Tuple2("${it}-", "") }.toArray(new Tuple2[0]) - label(resource, name, namespace, tuples) - } - - /** - * Patches a Kubernetes resource. - * - * @param resource The resource type - * @param name The name of the resource - * @param namespace The namespace (defaults to "default") - * @param type The patch type ('merge', 'strategic', 'json') - * @param yaml The patch content as a Map - */ - @CompileStatic(TypeCheckingMode.SKIP) - void patch(String resource, String name, String namespace = '', String type = '', Map yaml) { - log.debug("Patching $resource/$name in namespace $namespace") - - PatchContext patchContext = createPatchContext(type) - String patchJson = new JsonBuilder(yaml).toString() - log.trace("Patch JSON: $patchJson") - - executeWithErrorHandling("patch $resource/$name") { - def resourceClient = getResourceClient(resource, name, namespace) - resourceClient.patch(patchContext, patchJson) - } - - log.debug("Resource $resource/$name patched successfully") - } - - /** - * Deletes resources by label selector. - * - * @param resource The resource type - * @param namespace The namespace (defaults to "default") - * @param selectors Label selectors as Tuple2 - */ - @CompileStatic(TypeCheckingMode.SKIP) - void delete(String resource, String namespace = '', Tuple2... selectors) { - if (!selectors) { - throw new RuntimeException("Missing selectors") - } - - log.debug("Deleting $resource in namespace $namespace with selectors") - - Map labels = [:] - selectors.each { tuple -> labels[tuple.v1 as String] = tuple.v2 as String - } - - try { - deleteResourcesByType(resource, resolveNamespace(namespace), labels) - log.debug("Resources deleted successfully") - } catch (Exception e) { - log.warn("Failed to delete resources (may not exist): ${e.message}") - } - } - - /** - * Deletes a specific resource by name. - * - * @param resource The resource type - * @param namespace The namespace - * @param name The name of the resource - */ - @CompileStatic(TypeCheckingMode.SKIP) - void delete(String resource, String namespace, String name) { - log.debug("Deleting $resource/$name in namespace $namespace") - - try { - def resourceClient = getResourceClient(resource, name, namespace) - resourceClient.delete() - log.debug("Resource $resource/$name deleted successfully") - } catch (Exception e) { - log.warn("Failed to delete resource (may not exist): ${e.message}") - } - } - - /** - * Runs a pod with the specified image. - * - * @param name The name of the pod - * @param image The container image - * @param namespace The namespace (defaults to "default") - * @param overrides Additional pod overrides - * @param params Additional parameters - * @return Either a creation message or the pod logs (when -i/-it/-ti/--rm is used) - */ - String run(String name, String image, String namespace = '', Map overrides = [:], String... params) { - log.debug("Running pod $name with image $image in namespace $namespace") - String resolvedNamespace = resolveNamespace(namespace) - List runParams = params ? params.toList() : [] - - Pod pod = new PodBuilder() - .withNewMetadata() - .withName(name) - .withNamespace(resolvedNamespace) - .endMetadata() - .withNewSpec() - .addNewContainer() - .withName(name) - .withImage(image) - .endContainer() - .endSpec() - .build() - - applyRunParams(pod, runParams) - - if (overrides) { - log.debug("Applying overrides: $overrides") - pod = applyPodOverrides(pod, overrides) - } - - Pod createdPod = executeWithErrorHandling("run pod $name") { - client.pods() - .inNamespace(resolvedNamespace) - .resource(pod) - .create() - } - - log.debug("Pod $name created successfully") - if (shouldReturnPodOutput(runParams)) { - return collectPodRunOutput(createdPod.metadata.name, resolvedNamespace, shouldRemovePod(runParams)) - } - - return "pod/${createdPod.metadata.name} created" - } - - // ======================================== - // Public API Methods - Query Operations - // ======================================== - - /** - * Retrieves custom resources of a specific type across all namespaces. - * - * @param resource The custom resource type - * @return List of CustomResource objects - */ - @CompileStatic(TypeCheckingMode.SKIP) - List getCustomResource(String resource) { - log.debug("Getting custom resources of type $resource") - - try { - def apiClient = client.genericKubernetesResources(resource) - def resourceList = apiClient.inAnyNamespace().list() - - if (!resourceList || !(resourceList.hasProperty('items')) || !resourceList.items) { - return [] - } - - def items = resourceList.items as List - return items.collect { item -> - def itemMap = item as Map - def metadata = itemMap.get('metadata') as Map - new CustomResource((metadata?.get('namespace') ?: '') as String, - (metadata?.get('name') ?: '') as String) - } - } catch (Exception e) { - log.warn("Failed to get custom resources: ${e.message}") - return [] - } - } - - /** - * Retrieves the value of an annotation from a resource. - * - * @param resource The resource type - * @param name The name of the resource - * @param key The annotation key - * @param namespace The namespace (defaults to "default") - * @return The annotation value - * @throws RuntimeException if the resource or annotation is not found - */ - @CompileStatic(TypeCheckingMode.SKIP) - String getAnnotation(String resource, String name, String key, String namespace = '') { - log.debug("Getting annotation $key from $resource/$name in namespace $namespace") - - def resourceClient = getResourceClient(resource, name, namespace) - def resourceObj = resourceClient.get() - HasMetadata k8sResource = resourceObj as HasMetadata - - if (!k8sResource) { - throw new RuntimeException("Resource $resource/$name not found") - } - - def annotations = k8sResource.metadata?.annotations - if (!annotations) { - throw new RuntimeException("No annotations found on resource $resource/$name") - } - - String value = annotations[key] - log.debug("getAnnotation returns = ${value}") - return value - } - - /** - * Retrieves the current Kubernetes context. - * - * @return The name of the current context, or "(current context not set)" - */ - String getCurrentContext() { - try { - String context = client.getConfiguration().getCurrentContext()?.getName() - return context ?: '(current context not set)' - } catch (Exception e) { - log.trace("Failed to get current context: ${e.message}") - return '(current context not set)' - } - } - - // ======================================== - // Public API Methods - Wait Operations - // ======================================== - - /** - * Waits for a resource to reach a desired phase. - * - * @param resourceType The resource type (e.g., "pod", "deployment") - * @param resourceName The name of the resource - * @param namespace The namespace - * @param desiredPhase The phase to wait for (e.g., "Running", "Succeeded") - * @param timeoutSeconds Maximum wait time in seconds - * @param checkIntervalSeconds Interval between checks in seconds - * @throws IllegalArgumentException if parameters are invalid - * @throws RuntimeException if timeout is reached - */ - @CompileStatic(TypeCheckingMode.SKIP) - void waitForResourcePhase(String resourceType, String resourceName, String namespace, String desiredPhase, - int timeoutSeconds, int checkIntervalSeconds) { - validateWaitForResourcePhaseParams(resourceType, resourceName, namespace, desiredPhase, timeoutSeconds, checkIntervalSeconds) - - log.debug("Waiting for $resourceType/$resourceName to reach phase $desiredPhase") - - long startTime = System.currentTimeMillis() - long endTime = startTime + (timeoutSeconds * 1000) - - while (System.currentTimeMillis() < endTime) { - try { - def resourceClient = getResourceClient(resourceType, resourceName, namespace) - def resourceObj = resourceClient.get() - HasMetadata resource = resourceObj as HasMetadata - - if (resource) { - String phase = extractPhase(resource) - - if (phase == desiredPhase) { - log.debug("Resource ${resourceType}/${resourceName} in namespace ${namespace} reached the desired phase: ${desiredPhase}") - return - } - - log.debug("Current phase: ${phase}. Waiting for phase: ${desiredPhase}...") - } - } catch (Exception e) { - log.trace("Error checking resource phase: ${e.message}") - } - - sleep(checkIntervalSeconds * 1000) - } - - throw new RuntimeException("Timeout reached. Resource ${resourceType}/${resourceName} in namespace ${namespace} " + "did not reach the desired phase: ${desiredPhase} within ${timeoutSeconds} seconds.") - } - - @CompileStatic(TypeCheckingMode.SKIP) - private String extractPhase(def resource) { - // Typed Fabric8 resources, e.g. Pod.status.phase - if (resource.hasProperty('status') && resource.status?.hasProperty('phase')) { - return resource.status.phase as String - } - - // GenericKubernetesResource / Custom Resources - def status = resource.getAdditionalProperties()?.get('status') as Map - return status?.get('phase') as String - } - - /** - * Waits for a resource to reach a desired phase with default timeout and interval. - * - * @param resourceType The resource type - * @param resourceName The name of the resource - * @param namespace The namespace - * @param desiredPhase The phase to wait for - */ - void waitForResourcePhase(String resourceType, String resourceName, String namespace, String desiredPhase) { - waitForResourcePhase(resourceType, resourceName, namespace, desiredPhase, - DEFAULT_TIMEOUT_SECONDS, DEFAULT_CHECK_INTERVAL_SECONDS) - } - - private Pod applyPodOverrides(Pod pod, Map overrides) { - Map podAsMap = new JsonSlurper().parseText(Serialization.asJson(pod)) as Map - Map normalizedOverrides = normalizeOverrideValue(overrides) as Map - Map mergedPod = MapUtils.deepMerge(normalizedOverrides, podAsMap) - return Serialization.unmarshal(Serialization.asJson(mergedPod), Pod) as Pod - } - - private Object normalizeOverrideValue(Object value) { - if (value instanceof CharSequence) { - return value.toString() - } - - if (value instanceof Map) { - return value.collectEntries { key, mapValue -> [(key.toString()): normalizeOverrideValue(mapValue)] - } - } - - if (value instanceof Collection) { - return value.collect { entry -> normalizeOverrideValue(entry) } - } - - return value - } - - private void applyRunParams(Pod pod, List params) { - String restartPolicy = params.find { it.startsWith('--restart=') }?.substring('--restart='.length()) - if (restartPolicy) { - pod.spec.restartPolicy = restartPolicy - } - } - - private boolean shouldReturnPodOutput(List params) { - return params.any { it in ['--rm', '-i', '-it', '-ti'] } - } - - private boolean shouldRemovePod(List params) { - return params.contains('--rm') - } - - private String collectPodRunOutput(String podName, String namespace, boolean removePod) { - String phase = null - try { - phase = waitForPodCompletion(podName, namespace) - String logOutput = client.pods() - .inNamespace(namespace) - .withName(podName) - .getLog() ?: '' - - if (phase == 'Failed') { - throw new RuntimeException("Pod ${podName} failed:\n${logOutput}") - } - - return logOutput - } finally { - if (removePod) { - delete('pod', namespace, podName) - } - } - } - - private String waitForPodCompletion(String podName, String namespace) { - int tryCount = 0 - - while (tryCount < DEFAULT_RETRIES) { - Pod pod = client.pods() - .inNamespace(namespace) - .withName(podName) - .get() - - String phase = pod?.status?.phase - if (phase in ['Succeeded', 'Failed']) { - return phase - } - - tryCount++ - log.debug("Still waiting for pod/${podName} to complete... (try $tryCount/$DEFAULT_RETRIES)") - sleep(SLEEPTIME) - } - - throw new RuntimeException("Failed to retrieve completed pod/${podName} after ${DEFAULT_RETRIES} retries") - } - - // ======================================== - // Private Helper Methods - Retry Logic - // ======================================== - - /** - * Generic retry logic for waiting on resources. - * - * @param resourceDescription Description of the resource being waited on - * @param fetchClosure Closure that attempts to fetch the resource - * @return The result from the fetchClosure - * @throws RuntimeException if the resource is not available after retries - */ - private T waitForResourceWithRetry(String resourceDescription, Closure fetchClosure) { - int tryCount = 0 - T result = null - - while (!result && tryCount < DEFAULT_RETRIES) { - try { - result = fetchClosure() - } catch (Exception e) { - log.trace("Error fetching ${resourceDescription}: ${e.message}") - } - - if (!result) { - tryCount++ - log.debug("Still waiting for ${resourceDescription}... (try $tryCount/$DEFAULT_RETRIES)") - sleep(SLEEPTIME) - } - } - - if (!result) { - throw new RuntimeException("Failed to retrieve ${resourceDescription} after ${DEFAULT_RETRIES} retries") - } - - return result - } - - // ======================================== - // Private Helper Methods - Error Handling - // ======================================== - - /** - * Executes a closure with consistent error handling. - * - * @param operation Description of the operation - * @param closure The operation to execute - * @return The result of the closure - * @throws RuntimeException if the operation fails - */ - private T executeWithErrorHandling(String operation, Closure closure) { - try { - return closure() - } catch (Exception e) { - throw new RuntimeException("Failed to ${operation}: ${e.message}", e) - } - } - - // ======================================== - // Private Helper Methods - Resource Client - // ======================================== - - /** - * Gets a resource client for a specific resource type and name. - * - * @param resourceType The type of resource - * @param name The name of the resource - * @param namespace The namespace - * @return A resource client - */ - @CompileStatic(TypeCheckingMode.SKIP) - private getResourceClient(String resourceType, String name, String namespace) { - String ns = resolveNamespace(namespace) - - switch (resourceType.toLowerCase()) { - case 'pod': - case 'pods': - return client.pods().inNamespace(ns).withName(name) - - case 'service': - case 'services': - case 'svc': - return client.services().inNamespace(ns).withName(name) - - case 'deployment': - case 'deployments': - return client.apps().deployments().inNamespace(ns).withName(name) - - case 'configmap': - case 'configmaps': - case 'cm': - return client.configMaps().inNamespace(ns).withName(name) - - case 'secret': - case 'secrets': - return client.secrets().inNamespace(ns).withName(name) - - case 'namespace': - case 'namespaces': - case 'ns': - return client.namespaces().withName(name) - - case 'node': - case 'nodes': - return client.nodes().withName(name) - - case 'serviceaccount': - case 'serviceaccounts': - return client.serviceAccounts().inNamespace(ns).withName(name) - - - default: - return getCustomResourceClient(resourceType, name, ns) - } - } - - @CompileStatic(TypeCheckingMode.SKIP) - private getCustomResourceClient(String resourceType, String name, String namespace) { - String normalized = resourceType.toLowerCase() - - def crd = client.apiextensions() - .v1() - .customResourceDefinitions() - .list() - .items - .find { crd -> - crd.spec.names.kind?.equalsIgnoreCase(resourceType) || crd.spec.names.plural?.equalsIgnoreCase(normalized) || - crd.spec.names.singular?.equalsIgnoreCase(normalized) || - crd.spec.names.shortNames?.any { it.equalsIgnoreCase(normalized) } - } - - if (!crd) { - throw new RuntimeException("No CRD found for custom resource type '${resourceType}'") - } - - def version = crd.spec.versions.find { it.storage && it.served }?.name ?: crd.spec.versions.find { it.storage }?.name ?: crd.spec.versions.find { it.served }?.name - log.debug("Using CRD ${crd.metadata.name} with version ${version}, kind=${crd.spec.names.kind}, plural=${crd.spec.names.plural}") - - if (!version) { - throw new RuntimeException("No served version found for CRD '${crd.metadata.name}'") - } - - ResourceDefinitionContext context = new ResourceDefinitionContext.Builder() - .withGroup(crd.spec.group) - .withVersion(version) - .withKind(crd.spec.names.kind) - .withPlural(crd.spec.names.plural) - .withNamespaced(crd.spec.scope == "Namespaced") - .build() - - def resourceClient = client.genericKubernetesResources(context) - - if (crd.spec.scope == "Namespaced") { - return resourceClient.inNamespace(namespace).withName(name) - } - - return resourceClient.withName(name) - } - - /** - * Deletes resources by type and labels. - * - * @param resource The resource type - * @param namespace The namespace - * @param labels The label selectors - */ - @CompileStatic(TypeCheckingMode.SKIP) - private void deleteResourcesByType(String resource, String namespace, Map labels) { - switch (resource.toLowerCase()) { - case 'secret': - case 'secrets': - client.secrets().inNamespace(namespace).withLabels(labels).delete() - break - - case 'pod': - case 'pods': - client.pods().inNamespace(namespace).withLabels(labels).delete() - break - - case 'service': - case 'services': - case 'svc': - client.services().inNamespace(namespace).withLabels(labels).delete() - break - - case 'deployment': - case 'deployments': - client.apps().deployments().inNamespace(namespace).withLabels(labels).delete() - break - - case 'configmap': - case 'configmaps': - case 'cm': - client.configMaps().inNamespace(namespace).withLabels(labels).delete() - break - - default: - client.genericKubernetesResources(resource).inNamespace(namespace).withLabels(labels).delete() - } - } - - // ======================================== - // Private Helper Methods - Utilities - // ======================================== - - /** - * Resolves a namespace, defaulting to "default" if empty. - * - * @param namespace The namespace to resolve - * @return The resolved namespace - */ - private String resolveNamespace(String namespace) { - return namespace ?: DEFAULT_NAMESPACE - } - - /** - * Creates a PatchContext based on the patch type string. - * - * @param type The patch type ('merge', 'strategic', 'json', or empty for default) - * @return A configured PatchContext - */ - private PatchContext createPatchContext(String type) { - PatchType patchType - - if (!type) { - patchType = PatchType.JSON_MERGE - } else { - switch (type.toLowerCase()) { - case 'merge': - case 'json-merge': - patchType = PatchType.JSON_MERGE - break - case 'strategic': - patchType = PatchType.STRATEGIC_MERGE - break - case 'json': - patchType = PatchType.JSON - break - default: - patchType = PatchType.STRATEGIC_MERGE - } - } - - return new PatchContext.Builder().withPatchType(patchType).build() - } - - // ======================================== - // Private Helper Methods - Validation - // ======================================== - - /** - * Validates a namespace name. - * - * @param name The namespace name - * @throws IllegalArgumentException if the name is invalid - */ - private void validateNamespaceName(String name) { - if (name == null || name.trim().isEmpty()) { - throw new IllegalArgumentException("Namespace name must be provided and cannot be null or empty.") - } - } - - /** - * Validates parameters for service NodePort patching. - * - * @throws IllegalArgumentException if any parameter is invalid - */ - private void validateServiceNodePortPatch(String serviceName, String namespace, String portName, int newNodePort) { - if (!serviceName || !namespace || !portName || newNodePort <= 0) { - throw new IllegalArgumentException("Service name, namespace, port name, and valid nodePort must be provided") - } - } - - /** - * Validates parameters for waitForResourcePhase. - * - * @throws IllegalArgumentException if any parameter is invalid - */ - private void validateWaitForResourcePhaseParams(String resourceType, String resourceName, String namespace, - String desiredPhase, int timeoutSeconds, int checkIntervalSeconds) { - if (!resourceType || !resourceName || !namespace || !desiredPhase) { - throw new IllegalArgumentException("Resource type, name, namespace, and desired phase must be provided") - } - if (timeoutSeconds <= 0 || checkIntervalSeconds <= 0) { - throw new IllegalArgumentException("Timeout and check interval must be greater than zero") - } - } - - /** - * Return current namespace from running pod. - * @return - */ - String getCurrentNamespace() { - return this.client.getNamespace() - } - - private boolean runInOpenshift() { - // gopConfig can be null, in tests or at startup - return this.gopConfig?.application?.openshift ?: false - } - - // ======================================== - // Inner Classes - // ======================================== - - /** - * Represents a custom Kubernetes resource with namespace and name. */ - @Immutable - static class CustomResource { - String namespace - String name - } -} diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinition.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinition.groovy deleted file mode 100644 index 7b04a3537..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinition.groovy +++ /dev/null @@ -1,101 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.rbac - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.utils.TemplatingEngine -import groovy.util.logging.Slf4j - -import java.nio.file.Path - -@Slf4j -class RbacDefinition { - - private final Role.Variant variant - private String name - private String namespace - private List serviceAccounts = [] - private String subfolder = "rbac" - private GitRepo repo - private Config config - - private final TemplatingEngine templater = new TemplatingEngine() - - RbacDefinition(Role.Variant variant) { - this.variant = variant - } - - RbacDefinition withName(String name) { - this.name = name - return this - } - - RbacDefinition withNamespace(String namespace) { - this.namespace = namespace - return this - } - - RbacDefinition withServiceAccounts(List accounts) { - this.serviceAccounts = accounts - return this - } - - RbacDefinition withServiceAccountsFrom(String saNamespace, List saNames) { - return withServiceAccounts(ServiceAccountRef.fromNames(saNamespace, saNames)) - } - - RbacDefinition withSubfolder(String subfolder) { - this.subfolder = subfolder - return this - } - - RbacDefinition withRepo(GitRepo repo) { - this.repo = repo - return this - } - - RbacDefinition withConfig(Config config) { - this.config = config - return this - } - - void generate() { - if (!repo) { - throw new IllegalStateException("SCMM repo must be set using withRepo() before calling generate()") - } - - log.trace("Generating RBAC for name='${name}', namespace='${namespace}', subfolder='${subfolder}'") - - File outputDir = Path.of(repo.absoluteLocalRepoTmpDir, subfolder).toFile() - outputDir.mkdirs() - - generateRole(outputDir) - - generateRoleBinding(outputDir) - } - - private void generateRole(File outputDir) { - if (variant == Role.Variant.CLUSTER_ADMIN) { - log.trace("Skipping creation of ClusterRole cluster-admin") - return - } - - def role = new Role(name, namespace, variant, config) - - templater.template(role.getTemplateFile(), - role.getOutputFile(outputDir), - role.toTemplateParams()) - } - - private void generateRoleBinding(File outputDir) { - String roleName = name - if (variant == Role.Variant.CLUSTER_ADMIN) { - roleName = "cluster-admin" - } - def binding = new RoleBinding(name, namespace, roleName, serviceAccounts) - - templater.template(binding.getTemplateFile(), - binding.getOutputFile(outputDir), - binding.toTemplateParams()) - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/Role.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/Role.groovy deleted file mode 100644 index 18a97329c..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/Role.groovy +++ /dev/null @@ -1,54 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.rbac - -import com.cloudogu.gitops.config.Config - -class Role { - String name - String namespace - Variant variant - Config config - - Role(String name, String namespace, Variant variant, Config config) { - if (!name?.trim()) throw new IllegalArgumentException("Role name must not be blank") - if (!namespace?.trim()) throw new IllegalArgumentException("Role namespace must not be blank") - if (!variant) throw new IllegalArgumentException("Role variant must not be null") - if (!config) throw new IllegalArgumentException("Config must not be null") - - this.name = name - this.namespace = namespace - this.variant = variant - this.config = config - } - - enum Variant { - ARGOCD("templates/kubernetes/rbac/argocd-role.ftl.yaml"), - CLUSTER_ADMIN("") - - final String templatePath - - Variant(String templatePath) { - this.templatePath = templatePath - } - } - - Map toTemplateParams() { - return [name : name, - namespace: namespace, - config : config] - } - - File getTemplateFile() { - if (variant == Variant.CLUSTER_ADMIN) { - throw new IllegalStateException("cluster-admin role shall not be created") - } - return new File(variant.getTemplatePath()) - } - - File getOutputFile(File outputDir) { - if (variant == Variant.CLUSTER_ADMIN) { - throw new IllegalStateException("cluster-admin role shall not be created") - } - String filename = "role-${name}-${namespace}.yaml" - return new File(outputDir, filename) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RoleBinding.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RoleBinding.groovy deleted file mode 100644 index 6097a690d..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RoleBinding.groovy +++ /dev/null @@ -1,51 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.rbac - -class RoleBinding { - String name - String kind - String namespace - String roleName - String roleKind - List serviceAccounts - - RoleBinding(String name, String namespace, String roleName, List serviceAccounts) { - if (!name?.trim()) throw new IllegalArgumentException("RoleBinding name must not be blank") - if (!namespace?.trim()) throw new IllegalArgumentException("RoleBinding namespace must not be blank") - if (!roleName?.trim()) throw new IllegalArgumentException("Role name must not be blank") - if (!serviceAccounts || serviceAccounts.isEmpty()) throw new IllegalArgumentException("At least one service account is required") - - this.name = name - this.kind = "RoleBinding" - this.namespace = namespace - this.roleName = roleName - this.roleKind = "Role" - this.serviceAccounts = serviceAccounts - - if (roleName == "cluster-admin") { - this.kind = "ClusterRoleBinding" - this.roleKind = "ClusterRole" - } - } - - Map toTemplateParams() { - return [name : name, - kind : kind, - namespace : namespace, - roleName : roleName, - roleKind : roleKind, - serviceAccounts: serviceAccounts.collect { it.toMap() }] - } - - String getTemplatePath() { - return "templates/kubernetes/rbac/rolebinding.ftl.yaml" - } - - File getTemplateFile() { - return new File(getTemplatePath()) - } - - File getOutputFile(File outputDir) { - String filename = "rolebinding-${name}-${namespace}.yaml" - return new File(outputDir, filename) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/ServiceAccountRef.groovy b/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/ServiceAccountRef.groovy deleted file mode 100644 index 7188989cc..000000000 --- a/src/main/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/ServiceAccountRef.groovy +++ /dev/null @@ -1,32 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.rbac - -class ServiceAccountRef { - String name - String namespace - - ServiceAccountRef(String name, String namespace) { - if (!name?.trim()) { - throw new IllegalArgumentException("ServiceAccount name must not be blank") - } - if (!namespace?.trim()) { - throw new IllegalArgumentException("ServiceAccount namespace must not be blank") - } - this.name = name - this.namespace = namespace - } - - static List fromNames(String namespace, List names) { - if (!namespace?.trim()) { - throw new IllegalArgumentException("Namespace must not be blank for service accounts") - } - - return names - .findAll { it?.trim() } - .unique() - .collect { new ServiceAccountRef(it, namespace) } - } - - Map toMap() { - return [name: name, namespace: namespace] - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/CertManager.groovy b/src/main/groovy/com/cloudogu/gitops/tools/CertManager.groovy deleted file mode 100644 index 63fe2ea22..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/CertManager.groovy +++ /dev/null @@ -1,50 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.tools.common.ToolWithImage -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(160) -class CertManager extends Tool implements ToolWithImage { - - static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/cert-manager/templates/certManager-helm-values.ftl.yaml" - - final K8sClient k8sClient - final Config config - String namespace - - CertManager(Config config, - FileSystemUtils fileSystemUtils, - DeploymentStrategy deployer, - K8sClient k8sClient, - AirGappedUtils airGappedUtils, - GitHandler gitHandler) { - this.deployer = deployer - this.config = config - this.fileSystemUtils = fileSystemUtils - this.k8sClient = k8sClient - this.airGappedUtils = airGappedUtils - this.gitHandler = gitHandler - this.namespace = "${config.application.namePrefix}${config.features.certManager.namespace}" - } - - @Override - boolean isEnabled() { - return config.features.certManager.active - } - - @Override - void enable() { - deployHelmChart('cert-manager', 'cert-manager', namespace, config.features.certManager.helm, HELM_VALUES_PATH, config) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/ExternalSecretsOperator.groovy b/src/main/groovy/com/cloudogu/gitops/tools/ExternalSecretsOperator.groovy deleted file mode 100644 index f0181d801..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/ExternalSecretsOperator.groovy +++ /dev/null @@ -1,51 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.tools.common.ToolWithImage -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(400) -class ExternalSecretsOperator extends Tool implements ToolWithImage { - - static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/external-secrets/templates/values.ftl.yaml" - - String namespace = "${config.application.namePrefix}${config.features.secrets.namespace}" - Config config - K8sClient k8sClient - - ExternalSecretsOperator(Config config, - FileSystemUtils fileSystemUtils, - DeploymentStrategy deployer, - K8sClient k8sClient, - AirGappedUtils airGappedUtils, - GitHandler gitHandler) { - this.deployer = deployer - this.config = config - this.fileSystemUtils = fileSystemUtils - this.k8sClient = k8sClient - this.airGappedUtils = airGappedUtils - this.gitHandler = gitHandler - this.namespace = "${config.application.namePrefix}${config.features.secrets.namespace}" - } - - @Override - boolean isEnabled() { - return config.features.secrets.active - } - - @Override - void enable() { - def helmConfig = config.features.secrets.externalSecrets.helm - deployHelmChart('external-secrets-operator', 'external-secrets', namespace, helmConfig, HELM_VALUES_PATH, config) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/Ingress.groovy b/src/main/groovy/com/cloudogu/gitops/tools/Ingress.groovy deleted file mode 100644 index db3a4b4a4..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/Ingress.groovy +++ /dev/null @@ -1,50 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.tools.common.ToolWithImage -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(150) -class Ingress extends Tool implements ToolWithImage { - - static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/ingress/templates/ingress-helm-values.ftl.yaml" - - String namespace = "${config.application.namePrefix}" + config.features.ingress.ingressNamespace - Config config - K8sClient k8sClient - - Ingress(Config config, - FileSystemUtils fileSystemUtils, - DeploymentStrategy deployer, - K8sClient k8sClient, - AirGappedUtils airGappedUtils, - GitHandler gitHandler) { - this.deployer = deployer - this.config = config - this.fileSystemUtils = fileSystemUtils - this.k8sClient = k8sClient - this.airGappedUtils = airGappedUtils - this.gitHandler = gitHandler - } - - @Override - boolean isEnabled() { - return config.features.ingress.active - } - - @Override - void enable() { - def helmConfig = config.features.ingress.helm - deployHelmChart('traefik', 'traefik', namespace, helmConfig, HELM_VALUES_PATH, config) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/Monitoring.groovy b/src/main/groovy/com/cloudogu/gitops/tools/Monitoring.groovy deleted file mode 100644 index 773cd322f..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/Monitoring.groovy +++ /dev/null @@ -1,219 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.tools.common.ToolWithImage -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.TemplatingEngine -import groovy.transform.CompileStatic -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -import java.nio.file.Path - -@Slf4j -@Singleton -@Order(300) -@CompileStatic -class Monitoring extends Tool implements ToolWithImage { - - static final String HELM_VALUES_PATH = 'argocd/cluster-resources/apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml' - static final String RBAC_NAMESPACE_ISOLATION_TEMPLATE = 'argocd/cluster-resources/apps/monitoring/templates/rbac/namespace-isolation-rbac.ftl.yaml' - static final String NETWORK_POLICIES_PROMETHEUS_ALLOW_TEMPLATE = 'argocd/cluster-resources/apps/monitoring/templates/netpols/prometheus-allow-scraping.ftl.yaml' - - String namespace - Config config - K8sClient k8sClient - - private GitRepoFactory scmRepoProvider - - Monitoring(Config config, - FileSystemUtils fileSystemUtils, - DeploymentStrategy deployer, - K8sClient k8sClient, - AirGappedUtils airGappedUtils, - GitRepoFactory scmRepoProvider, - GitHandler gitHandler) { - this.config = config - this.fileSystemUtils = fileSystemUtils - this.deployer = deployer - this.k8sClient = k8sClient - this.airGappedUtils = airGappedUtils - this.scmRepoProvider = scmRepoProvider - this.gitHandler = gitHandler - this.namespace = "${config.application.namePrefix}${config.features.monitoring.namespace}" - } - - @Override - boolean isEnabled() { - return config.features.monitoring.active - } - - @Override - void enable() { - String uid = '' - if (config.application.openshift) { - uid = findValidOpenShiftUid() - } - - addHelmValuesData('monitoring', [grafana: [host: config.features.monitoring.grafanaUrl ? new URL(config.features.monitoring.grafanaUrl).host : '']]) - addHelmValuesData('namespaces', (config.application.namespaces.activeNamespaces ?: []) as LinkedHashSet) - addHelmValuesData('scm', scmConfigurationMetrics()) - addHelmValuesData('jenkins', jenkinsConfigurationMetrics()) - addHelmValuesData('uid', uid) - - // Create secrets imperatively here instead of values.yaml, because we don't want credentials to be visible in the Git repo - setupMonitoringSecrets() - createMonitoringCrd() - - GitRepo clusterResourcesRepo = scmRepoProvider.getRepo('argocd/cluster-resources', this.gitHandler.resourcesScm) - clusterResourcesRepo.cloneRepo() - - if (config.application.namespaceIsolation || config.application.netpols) { - if (config.application.namespaceIsolation) { generateNamespaceIsolationRBAC(clusterResourcesRepo) } - if (config.application.netpols) { generateNetpols(clusterResourcesRepo) } - } - - // Remove dashboards for features that are not enabled - cleanupUnusedDashboards(clusterResourcesRepo) - - clusterResourcesRepo.commitAndPush('Update Prometheus dashboards, RBAC and network policies.') - deployHelmChart('monitoring', 'kube-prometheus-stack', namespace, config.features.monitoring.helm, HELM_VALUES_PATH, config) - } - - private void setupMonitoringSecrets() { - k8sClient.createSecret('generic', - 'prometheus-metrics-creds-scmm', - namespace, - new Tuple2('password', config.application.password)) - - k8sClient.createSecret('generic', - 'prometheus-metrics-creds-jenkins', - namespace, - new Tuple2('password', config.jenkins.metricsPassword),) - - if (config.features.mail.smtpUser || config.features.mail.smtpPassword) { - k8sClient.createSecret('generic', - 'grafana-email-secret', - namespace, - new Tuple2('user', config.features.mail.smtpUser), - new Tuple2('password', config.features.mail.smtpPassword)) - } - } - - private void generateNamespaceIsolationRBAC(GitRepo repo) { - for (String currentNamespace : config.application.namespaces.activeNamespaces) { - String rbacYaml = new TemplatingEngine().template(new File(RBAC_NAMESPACE_ISOLATION_TEMPLATE), - [namespace : currentNamespace, - namePrefix: config.application.namePrefix, - config : config,]) - repo.writeFile("apps/monitoring/misc/rbac/${currentNamespace}.yaml", - rbacYaml) - } - } - - private void generateNetpols(GitRepo repo) { - for (String currentNamespace : config.application.namespaces.activeNamespaces) { - String netpolsYaml = new TemplatingEngine().template(new File(NETWORK_POLICIES_PROMETHEUS_ALLOW_TEMPLATE), - [namespace : currentNamespace, - namePrefix: config.application.namePrefix,]) - - repo.writeFile("apps/monitoring/misc/netpols/${currentNamespace}.yaml", - netpolsYaml) - } - } - - private Map scmConfigurationMetrics() { - URI uri = this.gitHandler.resourcesScm.prometheusMetricsEndpoint() - return [protocol: uri?.scheme ?: '', - host : uri?.authority ?: '', - path : uri?.path ?: '',] - } - - protected void createMonitoringCrd() { - if (!config.application.skipCrds) { - def serviceMonitorCrdYaml - if (config.application.mirrorRepos) { - serviceMonitorCrdYaml = Path.of("${config.application.localHelmChartFolder}/${config.features.monitoring.helm.chart}/charts/crds/crds/crd-servicemonitors.yaml").toString() - } else { - serviceMonitorCrdYaml = "https://raw.githubusercontent.com/prometheus-community/helm-charts/" + "kube-prometheus-stack-${config.features.monitoring.helm.version}/" + - "charts/kube-prometheus-stack/charts/crds/crds/crd-servicemonitors.yaml" - } - - log.debug("Applying ServiceMonitor CRD; Argo CD fails if it is not there. Chicken-egg-problem.\n" + "Applying from path ${serviceMonitorCrdYaml}") - k8sClient.applyYaml(serviceMonitorCrdYaml) - } - } - - private Map jenkinsConfigurationMetrics() { - URI uri = baseUriJenkins(config).resolve('prometheus') - return [metricsUsername: config.jenkins.metricsUsername ?: '', - protocol : uri.scheme ?: '', - host : uri.authority ?: '', - path : uri.path ?: '',] - } - - private static URI baseUriJenkins(Config config) { - if (config.jenkins.internal) { - return new URI("http://jenkins.${config.application.namePrefix}${config.jenkins.namespace}.svc.cluster.local/") - } - def urlString = config.jenkins?.url?.strip() ?: "" - if (!urlString) { - throw new IllegalArgumentException("config.jenkins.url must be set when config.jenkins.internal = false") - } - def url = URI.create(urlString) - return url.toString().endsWith("/") ? url : URI.create(url.toString() + "/") - } - - private String findValidOpenShiftUid() { - String uidRange = k8sClient.getAnnotation('namespace', namespace, 'openshift.io/sa.scc.uid-range') - - if (uidRange) { - log.debug("found UID=${uidRange}") - String uid = uidRange.split('/')[0] - return uid - } else { - throw new RuntimeException("Could not find a valid UID! Really running on OpenShift?") - } - } - - protected void cleanupUnusedDashboards(GitRepo clusterResourcesRepo) { - String repoRoot = clusterResourcesRepo.getAbsoluteLocalRepoTmpDir() - String dashboardRoot = "${repoRoot}/apps/prometheusstack/misc/dashboard" - - if (!config.features.ingress.active) { - fileSystemUtils.deleteFile("${dashboardRoot}/traefik-dashboard.yaml") - fileSystemUtils.deleteFile("${dashboardRoot}/traefik-dashboard-requests-handling.yaml") - } - - if (!config.jenkins.active) { - fileSystemUtils.deleteFile("${dashboardRoot}/jenkins-dashboard.yaml") - } - - if (!config.scm.scmManager?.url) { - fileSystemUtils.deleteFile("${dashboardRoot}/scmm-dashboard.yaml") - } - } - - @Override - String getNamespace() { - return namespace - } - - @Override - K8sClient getK8sClient() { - return k8sClient - } - - @Override - Config getConfig() { - return config - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/Registry.groovy b/src/main/groovy/com/cloudogu/gitops/tools/Registry.groovy deleted file mode 100644 index e12b6e179..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/Registry.groovy +++ /dev/null @@ -1,76 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(40) -class Registry extends Tool { - - /** - * Local container port of the registry within the pod*/ - public static final String CONTAINER_PORT = '5000' - - String namespace - private Config config - private K8sClient k8sClient - - Registry(Config config, - FileSystemUtils fileSystemUtils, - K8sClient k8sClient, - AirGappedUtils airGappedUtils, - // For now we deploy imperatively using helm to avoid order problems. In future we could deploy via argocd. - HelmStrategy deployer) { - this.deployer = deployer - this.config = config - this.fileSystemUtils = fileSystemUtils - this.k8sClient = k8sClient - this.airGappedUtils = airGappedUtils - - if (config.registry.internal) { - this.namespace = "${config.application.namePrefix}${config.registry.namespace}" - } - } - - @Override - boolean isEnabled() { - return config.registry.active - } - - @Override - void enable() { - - if (config.registry.internal) { - addHelmValuesData("service", [nodePort: Config.DEFAULT_REGISTRY_PORT, - type : 'NodePort']) - - def helmConfig = config.registry.helm - deployHelmChart('registry', 'docker-registry', namespace, helmConfig, "", config) - - if (config.registry.internalPort != Config.DEFAULT_REGISTRY_PORT) { - /* Add additional node port - 30000 is needed as a static by docker via port mapping of k3d, e.g. 32769 -> 30000 on server-0 container - See "-p 30000" in init-cluster.sh - e.g 32769 is needed so the kubelet can access the image inside the server-0 container - */ - - /* k8sClient.createServiceNodePort('docker-registry-internal-port', - CONTAINER_PORT, config.registry.internalPort.toString(), - namespace) */ - - k8sClient.createServiceNodePort('docker-registry-internal-port', - "${CONTAINER_PORT}:${CONTAINER_PORT}", - config.registry.internalPort.toString(), - namespace) - } - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/Vault.groovy b/src/main/groovy/com/cloudogu/gitops/tools/Vault.groovy deleted file mode 100644 index a9f54e724..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/Vault.groovy +++ /dev/null @@ -1,78 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.tools.common.ToolWithImage -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.TemplatingEngine -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(500) -class Vault extends Tool implements ToolWithImage { - static final String VAULT_START_SCRIPT_PATH = "argocd/cluster-resources/apps/vault/templates/dev-post-start.ftl.sh" - static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/vault/templates/values.ftl.yaml" - - String namespace - Config config - K8sClient k8sClient - - Vault(Config config, - FileSystemUtils fileSystemUtils, - K8sClient k8sClient, - DeploymentStrategy deployer, - AirGappedUtils airGappedUtils, - GitHandler gitHandler) { - this.deployer = deployer - this.config = config - this.fileSystemUtils = fileSystemUtils - this.k8sClient = k8sClient - this.airGappedUtils = airGappedUtils - this.gitHandler = gitHandler - this.namespace = "${config.application.namePrefix}${config.features.secrets.namespace}" - } - - @Override - boolean isEnabled() { - return config.features.secrets.active - } - - @Override - void enable() { - // Note that some specific configuration steps are implemented in ArgoCD - def helmConfig = config.features.secrets.vault.helm - - addHelmValuesData("host", config.features.secrets.vault.url ? new URL(config.features.secrets.vault.url as String).host : '') - - String vaultMode = config.features.secrets.vault.mode - if (vaultMode == 'dev') { - log.debug('WARNING! Vault dev mode is enabled! In this mode, Vault runs entirely in-memory\n' + 'and starts unsealed with a single unseal key. ') - - // Create config map from init script - // Init script creates/authorizes secrets, users, service accounts, etc. - def vaultPostStartConfigMap = 'vault-dev-post-start' - def vaultPostStartVolume = 'dev-post-start' - - def templatedFile = fileSystemUtils.copyToTempDir(fileSystemUtils.getRootDir() + "/" + VAULT_START_SCRIPT_PATH) - def postStartScript = new TemplatingEngine().replaceTemplate(templatedFile.toFile(), [namePrefix: config.application.namePrefix]) - - log.debug('Creating namespace for vault, so it can add its secrets there') - k8sClient.createNamespace(namespace) - k8sClient.createConfigMapFromFile(vaultPostStartConfigMap, namespace, postStartScript.absolutePath) - - addHelmValuesData("dev", [rootToken : UUID.randomUUID(), - vaultPostStartConfigMap: vaultPostStartConfigMap, - vaultPostStartVolume : vaultPostStartVolume, - postStartScriptName : postStartScript.name]) - } - - deployHelmChart('vault', 'vault', namespace, helmConfig, HELM_VALUES_PATH, config) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/common/CommonToolConfig.groovy b/src/main/groovy/com/cloudogu/gitops/tools/common/CommonToolConfig.groovy deleted file mode 100644 index e5379fd87..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/common/CommonToolConfig.groovy +++ /dev/null @@ -1,32 +0,0 @@ -package com.cloudogu.gitops.tools.common - -import com.cloudogu.gitops.config.Config -import groovy.util.logging.Slf4j - -@Slf4j -class CommonToolConfig extends Tool { - @Override - void preConfigInit(Config configToSet) { - validateConfig(configToSet) - } - - /** - * Make sure that config does not contain contradictory values. - * Throws RuntimeException which meaningful message, if invalid.*/ - void validateConfig(Config configToSet) { - validateMirrorReposHelmChartFolderSet(configToSet) - } - - private void validateMirrorReposHelmChartFolderSet(Config configToSet) { - if (configToSet.application.mirrorRepos && !configToSet.application.localHelmChartFolder) { - // This should only happen when run outside the image, i.e. during development - throw new RuntimeException("Missing config for localHelmChartFolder.\n" + "Either run inside the official container image or setting env var " + - "LOCAL_HELM_CHART_FOLDER='charts' after running 'scripts/downloadHelmCharts.sh' from the repo") - } - } - - @Override - boolean isEnabled() { - return false - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/common/Tool.groovy b/src/main/groovy/com/cloudogu/gitops/tools/common/Tool.groovy deleted file mode 100644 index dd16e9a15..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/common/Tool.groovy +++ /dev/null @@ -1,174 +0,0 @@ -package com.cloudogu.gitops.tools.common - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.MapUtils -import com.cloudogu.gitops.utils.TemplatingEngine -import freemarker.template.Configuration -import freemarker.template.DefaultObjectWrapperBuilder -import groovy.util.logging.Slf4j -import groovy.yaml.YamlSlurper - -import java.nio.file.Path - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType - -/** - * A single tool to be deployed by GOP. - * - * Typically, this is a helm chart (see {@link DeploymentStrategy} and - * {@code downloadHelmCharts.sh}) with its own section in the config - * (see {@link com.cloudogu.gitops.config.schema.Schema#features}).

- * - * In the config, features typically set their default helm chart coordinates and provide options to - *
    - *
  • configure images
    • - *
  • overwrite default helm values
    • - *

- * - * In addition to their own config, features react to several generic GOP config options.
- * Here are some typical examples: - *
    - *
  • Mirror the Helm Chart: {@link com.cloudogu.gitops.config.schema.Schema.ApplicationSchema#mirrorRepos} see {@link com.cloudogu.gitops.utils.AirGappedUtils#mirrorHelmRepoToGit(java.util.Map)}
    • - *
  • Create Image Pull Secrets: {@link com.cloudogu.gitops.config.schema.Schema.RegistrySchema#createImagePullSecrets} see {@link ToolWithImage}
    • - *
  • Install with Network Policies: {@link com.cloudogu.gitops.config.schema.Schema.ApplicationSchema#netpols}
    • - *
  • Install with Resource requests + limits: {@link com.cloudogu.gitops.config.schema.Schema.ApplicationSchema#podResources}
    • - *
  • Install without CRDs: {@link com.cloudogu.gitops.config.schema.Schema.ApplicationSchema#skipCrds}
    • - *
  • For apps with UI: Setting {@link com.cloudogu.gitops.config.schema.Schema.ApplicationSchema#username} and {@link com.cloudogu.gitops.config.schema.Schema.ApplicationSchema#password}
    • - *
*/ - -@Slf4j -abstract class Tool { - - protected FileSystemUtils fileSystemUtils - protected DeploymentStrategy deployer - protected AirGappedUtils airGappedUtils - protected GitHandler gitHandler - protected Map helmValuesTemplateData = [:] - - protected void addHelmValuesData(String key, Object value) { - this.helmValuesTemplateData[key] = value - } - - boolean install() { - if (isEnabled()) { - log.info("Installing Feature ${getClass().getSimpleName()}") - - if (this instanceof ToolWithImage) { - (this as ToolWithImage).createImagePullSecret() - } - - enable() - return true - } else { - log.debug("Feature ${getClass().getSimpleName()} is disabled") - disable() - return false - } - } - - String getActiveNamespaceFromFeature() { - //using reflection to get all subclasses implementing a own namespace - if (this.metaClass.hasProperty(this, 'namespace')) { - return isEnabled() ? this.getProperty('namespace') : null - } - return null - } - - static Map templateToMap(String filePath, Map parameters) { - def hydratedString = new TemplatingEngine().template(new File(filePath), parameters) - - if (hydratedString.trim().isEmpty()) { - // Otherwise YamlSlurper returns an empty array, whereas we expect a Map - return [:] - } - return new YamlSlurper().parseText(hydratedString) as Map - } - - protected void deployHelmChart(String featureName, - String releaseName, - String namespace, - Config.HelmConfigWithValues helmConfig, - String helmValuesTemplatePath, - Config config) { - String repoURL = helmConfig.repoURL - String chartOrPath = helmConfig.chart - String version = helmConfig.version - RepoType repoType = RepoType.HELM - - this.addHelmValuesData("config", config) - this.addHelmValuesData("statics", new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build().getStaticModels()) - - /* If we get a helmValuesTemplatePath we render the Template with the given Data. - * Some Features might not use a values template and thus passing no helmValuesTemplatePath, in that - * case we simply treat helmValuesTemplateData directly as helmValuesData */ - Map helmValuesData = this.helmValuesTemplateData - if (helmValuesTemplatePath) { - def helmValuesPath = helmValuesTemplatePath.toString() - if (helmValuesPath.contains(".ftl")) { - log.debug("Rendering helm values template from ${helmValuesTemplatePath}") - helmValuesData = templateToMap(helmValuesTemplatePath, this.helmValuesTemplateData) - } else { - log.debug("Reading plain helm values YAML from ${helmValuesTemplatePath}") - helmValuesData = fileSystemUtils.readYaml(Path.of(helmValuesTemplatePath)) as Map - } - } - - helmValuesData = MapUtils.deepMerge(helmConfig.values, helmValuesData) - Path tempValuesPath = this.fileSystemUtils.writeTempFile(helmValuesData) - - if (config.application.mirrorRepos) { - log.debug("Using a local, mirrored git repo as deployment source for feature ${featureName}") - - String repoNamespaceAndName = this.airGappedUtils.mirrorHelmRepoToGit(helmConfig) - repoURL = this.gitHandler.resourcesScm.repoUrl(repoNamespaceAndName) - chartOrPath = '.' - repoType = RepoType.GIT - version = new YamlSlurper() - .parse(Path.of("${config.application.localHelmChartFolder}/${helmConfig.chart}", - 'Chart.yaml'))['version'] - } - - log.debug("Starting deployment of feature ${featureName} from ${repoURL}.") - log.debug("helm values used: ${helmValuesData}") - - this.deployer.deployFeature(repoURL, - featureName, - chartOrPath, - version, - namespace, - releaseName, - tempValuesPath, - repoType) - } - - abstract boolean isEnabled() - - /* - * Hooks for enabling or disabling a feature. Both optional, because not always needed. - */ - - protected void enable() {} - - protected void disable() {} - - /* - * Hook for special feature validation. Optional. - * Feature should throw RuntimeException to stop immediately. - */ - - void validate() {} - - /** - * Hook for preConfigInit. Optional. - * Feature should throw RuntimeException to stop immediately.*/ - void preConfigInit(Config configToSet) {} - - /** - * Hook for postConfigInit. Optional. - * Feature should throw RuntimeException to stop immediately.*/ - void postConfigInit(Config configToSet) {} -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/common/ToolWithImage.groovy b/src/main/groovy/com/cloudogu/gitops/tools/common/ToolWithImage.groovy deleted file mode 100644 index 5bdb2dc06..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/common/ToolWithImage.groovy +++ /dev/null @@ -1,32 +0,0 @@ -package com.cloudogu.gitops.tools.common - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import org.slf4j.Logger -import org.slf4j.LoggerFactory - -/** - * A feature that relies on container images running inside the kubernetes cluster.*/ -trait ToolWithImage { - - final Logger log = LoggerFactory.getLogger(this.class) - - void createImagePullSecret() { - if (config.registry.createImagePullSecrets) { - - log.trace("Creating image pull secret 'proxy-registry' in namespace ${this.namespace}") - String url = config.registry.proxyUrl ?: config.registry.url - String user = config.registry.proxyUsername ?: config.registry.readOnlyUsername ?: config.registry.username - String password = config.registry.proxyPassword ?: config.registry.readOnlyPassword ?: config.registry.password - - k8sClient.createNamespace(this.namespace) - k8sClient.createImagePullSecret('proxy-registry', namespace, url, user, password) - } - } - - abstract String getNamespace() - - abstract K8sClient getK8sClient() - - abstract Config getConfig() -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/core/Jenkins.groovy b/src/main/groovy/com/cloudogu/gitops/tools/core/Jenkins.groovy deleted file mode 100644 index 910e95058..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/core/Jenkins.groovy +++ /dev/null @@ -1,254 +0,0 @@ -package com.cloudogu.gitops.tools.core - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.util.ScmProviderType -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.jenkins.GlobalPropertyManager -import com.cloudogu.gitops.infrastructure.jenkins.JobManager -import com.cloudogu.gitops.infrastructure.jenkins.PrometheusConfigurator -import com.cloudogu.gitops.infrastructure.jenkins.UserManager -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.CommandExecutor -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.NetworkingUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton - -@Slf4j -@Singleton -@Order(70) -class Jenkins extends Tool { - - static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/jenkins/values.ftl.yaml" - - String namespace - private Config config - private CommandExecutor commandExecutor - private GlobalPropertyManager globalPropertyManager - private JobManager jobManager - private UserManager userManager - private PrometheusConfigurator prometheusConfigurator - private K8sClient k8sClient - private NetworkingUtils networkingUtils - - Jenkins(Config config, - CommandExecutor commandExecutor, - FileSystemUtils fileSystemUtils, - GlobalPropertyManager globalPropertyManager, - JobManager jobManager, - UserManager userManager, - PrometheusConfigurator prometheusConfigurator, - HelmStrategy deployer, - K8sClient k8sClient, - NetworkingUtils networkingUtils, - AirGappedUtils airGappedUtils, - GitHandler gitHandler) { - this.config = config - this.commandExecutor = commandExecutor - this.fileSystemUtils = fileSystemUtils - this.globalPropertyManager = globalPropertyManager - this.jobManager = jobManager - this.userManager = userManager - this.prometheusConfigurator = prometheusConfigurator - this.deployer = deployer - this.k8sClient = k8sClient - this.networkingUtils = networkingUtils - this.airGappedUtils = airGappedUtils - this.gitHandler = gitHandler - - if (config.jenkins.internal) { - this.namespace = "${config.application.namePrefix}${config.jenkins.namespace}" - } - } - - @Override - boolean isEnabled() { - return config.jenkins.active - } - - @Override - void enable() { - - if (config.jenkins.internal) { - - k8sClient.createNamespace(namespace) - - // Mark the first node for Jenkins and agents. See jenkins/values.ftl.yaml "agent.workingDir" for details. - // Remove first (in case new nodes were added) - k8sClient.labelRemove('node', '--all', '', 'node') - def nodeName = k8sClient.waitForNode().replace('node/', '') - k8sClient.label('node', nodeName, new Tuple2('node', 'jenkins')) - - k8sClient.createSecret('generic', 'jenkins-credentials', namespace, - new Tuple2('jenkins-admin-user', config.jenkins.username), - new Tuple2('jenkins-admin-password', config.jenkins.password)) - - def helmConfig = config.jenkins.helm - String releaseName = "jenkins" - addHelmValuesData("dockerGid", findDockerGid()) - - deployHelmChart('jenkins', releaseName, namespace, helmConfig, HELM_VALUES_PATH, config) - - // Defined here: https://github.com/jenkinsci/helm-charts/blob/jenkins-5.8.1/charts/jenkins/templates/_helpers.tpl#L46-L57 - String serviceName = releaseName - // Update jenkins.url after it is deployed (and ports are known) - if (config.application.runningInsideK8s) { - log.debug("Setting jenkins url to k8s service, since installation is running inside k8s") - config.jenkins.url = networkingUtils.createUrl("${serviceName}.${namespace}.svc.cluster.local", "80") - } else { - log.debug("Setting jenkins configs for local single node cluster with internal jenkins. Waiting for NodePort...") - def port = k8sClient.waitForNodePort(serviceName, namespace) - String clusterBindAddress = networkingUtils.findClusterBindAddress() - config.jenkins.url = networkingUtils.createUrl(clusterBindAddress, port) - } - } - - commandExecutor.execute("${fileSystemUtils.rootDir}/scripts/jenkins/init-jenkins.sh", [TRACE : config.application.trace, - INTERNAL_JENKINS : config.jenkins.internal, - JENKINS_HELM_CHART_VERSION: config.jenkins.helm.version, - JENKINS_URL : config.jenkins.url, - JENKINS_USERNAME : config.jenkins.username, - JENKINS_PASSWORD : config.jenkins.password, - SCM_URL : this.gitHandler.tenant.url, - PREFIXED_SCM_URL : this.gitHandler.tenant.repoPrefix(), - SCM_PASSWORD : this.gitHandler.tenant.credentials.password, - SCM_PROVIDER : config.scm.scmProviderType, - INSTALL_ARGOCD : config.features.argocd.active, - NAME_PREFIX : config.application.namePrefix, - INSECURE : config.application.insecure, - SKIP_RESTART : config.jenkins.skipRestart, - SKIP_PLUGINS : config.jenkins.skipPlugins]) - - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}SCM_URL", this.gitHandler.tenant.url) - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}PREFIXED_SCM_URL", this.gitHandler.tenant.repoPrefix()) - - if (config.jenkins.additionalEnvs) { - for (entry in (config.jenkins.additionalEnvs as Map).entrySet()) { - globalPropertyManager.setGlobalProperty(entry.key.toString(), entry.value.toString()) - } - } - - if (config.registry.url) { - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}REGISTRY_URL", config.registry.url) - } - - if (config.registry.path) { - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}REGISTRY_PATH", config.registry.path) - } - - if (config.registry.twoRegistries) { - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}REGISTRY_PROXY_URL", config.registry.proxyUrl) - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}REGISTRY_PROXY_PATH", config.registry.proxyPath) - } - - if (config.jenkins.mavenCentralMirror) { - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}MAVEN_CENTRAL_MIRROR", config.jenkins.mavenCentralMirror) - } - - globalPropertyManager.setGlobalProperty("${config.application.namePrefixForEnvVars}K8S_VERSION", Config.K8S_VERSION) - - if (userManager.isUsingCasSecurityRealm()) { - log.trace("Using CAS Security Realm. Must not create user.") - } else { - userManager.createUser(config.jenkins.metricsUsername, config.jenkins.metricsPassword) - } - - userManager.grantPermission(config.jenkins.metricsUsername, UserManager.Permissions.METRICS_VIEW) - - if (config.features.monitoring.active && config.jenkins.internal) { - // And external Jenkins can likely not be monitored - prometheusConfigurator.enableAuthentication() - } - - } - - void createJenkinsjob(String namespace, String repoName) { - def credentialId = "scm-user" - String prefixedNamespace = "${config.application.namePrefix}${namespace}" - String jobName = "${config.application.namePrefix}${repoName}" - - jobManager.createJob(jobName, - this.gitHandler.tenant.url, - prefixedNamespace, - credentialId) - - if (config.scm.scmProviderType == ScmProviderType.SCM_MANAGER) { - jobManager.createCredential(jobName, - credentialId, - "${config.application.namePrefix}gitops", - "${config.scm.getScmManager().password}", - 'credentials for accessing scm-manager') - } - - if (config.scm.scmProviderType == ScmProviderType.GITLAB) { - jobManager.createCredential(jobName, - credentialId, - "${config.scm.getGitlab().username}", - "${config.scm.getGitlab().password}", - 'credentials for accessing gitlab') - } - - jobManager.createCredential(jobName, - "registry-user", - "${config.registry.username}", - "${config.registry.password}", - 'credentials for accessing the docker-registry for writing images built on jenkins') - - if (config.registry.twoRegistries) { - jobManager.createCredential(jobName, - "registry-proxy-user", - "${config.registry.proxyUsername}", - "${config.registry.proxyPassword}", - 'credentials for accessing the docker-registry that contains 3rd party or base images') - } - - jobManager.startJob(jobName) - } - - protected String findDockerGid() { - String gid = '' - def etcGroup = k8sClient.run("tmp-docker-gid-grepper-${new Random().nextInt(10000)}", - 'irrelevant' /* Redundant, but mandatory param */, namespace, createGidGrepperOverrides(), - '--restart=Never', '-ti', '--rm', '--quiet') - // --quiet is necessary to avoid 'pod deleted' output - - def lines = etcGroup?.split('\n') - for (String it : lines) { - def parts = it.split(":") - if (parts[0] == 'docker') { - gid = parts[2] - break - } - } - - if (!gid) { - log.warn('Unable to determine Docker Group ID (GID). Jenkins Agent pods will run as root user (UID 0)!\n' + "Group docker not found in /etc/group:\n${etcGroup}") - return '' - } else { - log.debug("Using Docker Group ID (GID) ${gid} for Jenkins Agent pods") - return gid - } - } - - Map createGidGrepperOverrides() { - ['spec': ['containers' : [['name' : 'tmp-docker-gid-grepper', - // We use the same image for several tasks for performance and maintenance reasons - 'image' : "${config.jenkins.internalBashImage}", - 'args' : ['cat', '/etc/group'], - 'volumeMounts': [['name' : 'group', - 'mountPath': '/etc/group', - 'readOnly' : true]]]], - 'nodeSelector': ['node': 'jenkins'], - 'volumes' : [['name' : 'group', - 'hostPath': ['path': '/etc/group']]]]] - } - - @Override - String getActiveNamespaceFromFeature() { - return isEnabled() && config?.jenkins?.internal ? getNamespace() : null - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/core/ScmManagerSetup.groovy b/src/main/groovy/com/cloudogu/gitops/tools/core/ScmManagerSetup.groovy deleted file mode 100644 index adb541d22..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/core/ScmManagerSetup.groovy +++ /dev/null @@ -1,175 +0,0 @@ -package com.cloudogu.gitops.tools.core - -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManager -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerUser -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.MapUtils -import com.cloudogu.gitops.utils.TemplatingEngine -import groovy.util.logging.Slf4j - -@Slf4j -class ScmManagerSetup { - - private ScmManager scmManager - - static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/scm-manager/templates/values.ftl.yaml" - - ScmManagerSetup(ScmManager scmManager) { - this.scmManager = scmManager - } - - void waitForScmmAvailable(int timeoutSeconds = 180, int intervalMillis = 5000, int startDelay = 0) { - long startTime = System.currentTimeMillis() - long timeoutMillis = timeoutSeconds * 1000L - sleep(startDelay) - while (System.currentTimeMillis() - startTime < timeoutMillis) { - try { - def call = scmManager.apiClient.generalApi().checkScmmAvailable() - def response = call.execute() - - if (response.successful) { - return - } - } catch (Exception e) { - log.debug("Waiting for SCM-Manager... Error: ${e.message}") - } - - sleep(intervalMillis) - } - throw new RuntimeException("Timeout: SCM-Manager did not respond with 200 OK within ${timeoutSeconds} seconds") - } - - void configure() { - installScmmPlugins() - setSetupConfigs() - if (this.scmManager.config.jenkins.active) { - configureJenkinsPlugin() - } - addDefaultUsers() - log.info("ScmManager Setup finished!") - } - - void setupHelm() { - def releaseName = 'scmm' - - def templatedMap = TemplatingEngine.templateToMap(HELM_VALUES_PATH, [config : this.scmManager.config, - host : this.scmManager.scmmConfig.ingress, - username : this.scmManager.scmmConfig.credentials.username, - password : this.scmManager.scmmConfig.credentials.password, - helm : this.scmManager.scmmConfig.helm, - releaseName: releaseName]) - - def helmConfig = this.scmManager.scmmConfig.helm - def mergedMap = MapUtils.deepMerge(helmConfig.values, templatedMap) - def tempValuesPath = new FileSystemUtils().writeTempFile(mergedMap) - this.scmManager.helmStrategy.deployFeature(helmConfig.repoURL, - 'scm-manager', - helmConfig.chart, - helmConfig.version, - this.scmManager.scmmConfig.namespace, - releaseName, - tempValuesPath) - } - - def installScmmPlugins() { - - if (this.scmManager.config.scm.scmManager.skipPlugins) { - log.debug("Skipping SCM plugin installation") - return - } - - def pluginNames = ["scm-mail-plugin", - "scm-review-plugin", - "scm-code-editor-plugin", - "scm-editor-plugin", - "scm-landingpage-plugin", - "scm-el-plugin", - "scm-readme-plugin", - "scm-webhook-plugin", - "scm-ci-plugin", - "scm-metrics-prometheus-plugin"] - - if (this.scmManager.config.jenkins.active) { - pluginNames.add("scm-jenkins-plugin") - } - Boolean restartForThisPlugin = false - pluginNames.each { String pluginName -> - log.debug("Installing Plugin ${pluginName} ...") - restartForThisPlugin = !this.scmManager.config.scm.scmManager.skipRestart && pluginName == pluginNames.last() - ScmManagerApiClient.handleApiResponse(scmManager.apiClient.pluginApi().install(pluginName, restartForThisPlugin)) - } - - log.debug("SCM-Manager plugin installation finished successfully!") - if (restartForThisPlugin) { - waitForScmmAvailable(180, 2000, 100) - } - } - - void setSetupConfigs() { - def setupConfigs = [enableProxy : false, - proxyPort : 8080, - proxyServer : "proxy.mydomain.com", - proxyUser : null, - proxyPassword : null, - realmDescription : "SONIA :: SCM Manager", - disableGroupingGrid : false, - dateFormat : "YYYY-MM-DD HH:mm:ss", - anonymousAccessEnabled : false, - anonymousMode : "OFF", - baseUrl : this.scmManager.url, - forceBaseUrl : false, - loginAttemptLimit : -1, - proxyExcludes : [], - skipFailedAuthenticators: false, - pluginUrl : "https://plugin-center-api.scm-manager.org/api/v1/plugins/{version}?os={os}&arch={arch}", - loginAttemptLimitTimeout: 300, - enabledXsrfProtection : true, - namespaceStrategy : "CustomNamespaceStrategy", - loginInfoUrl : "https://login-info.scm-manager.org/api/v1/login-info", - releaseFeedUrl : "https://scm-manager.org/download/rss.xml", - mailDomainName : "scm-manager.local", - adminGroups : [], - adminUsers : []] - - ScmManagerApiClient.handleApiResponse(scmManager.apiClient.generalApi().setConfig(setupConfigs)) - log.debug("Successfully added SCMM Setup Configs") - } - - void configureJenkinsPlugin() { - - def jenkinsPluginConfig = [disableRepositoryConfiguration: false, - disableMercurialTrigger : false, - disableGitTrigger : false, - disableEventTrigger : false, - url : this.scmManager.config.jenkins.urlForScm] as Map - - ScmManagerApiClient.handleApiResponse(this.scmManager.apiClient.pluginApi().configureJenkinsPlugin(jenkinsPluginConfig)) - log.debug("Successfully configured JenkinsPlugin in SCM-Manager.") - } - - void addDefaultUsers() { - def metricsUsername = "${this.scmManager.config.application.namePrefix}metrics" - addUser(this.scmManager.scmmConfig.gitOpsUsername, this.scmManager.scmmConfig.password) - addUser(metricsUsername, this.scmManager.scmmConfig.password) - grantUserPermissions(metricsUsername, ["metrics:read"]) - } - - void addUser(String username, String password, String email = 'changeme@test.local') { - ScmManagerUser userRequest = [name : username, - displayName: username, - mail : email, - external : false, - password : password, - active : true, - _links : [:]] - ScmManagerApiClient.handleApiResponse(scmManager.apiClient.usersApi().addUser(userRequest)) - log.debug("Successfully created SCM-Manager User.") - } - - void grantUserPermissions(String username, List permissions) { - def permissionBody = [permissions: permissions] - ScmManagerApiClient.handleApiResponse(scmManager.apiClient.usersApi().setPermissionForUser(username, permissionBody)) - log.debug("Granted permissions ${permissions} to user ${username}.") - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCD.groovy b/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCD.groovy deleted file mode 100644 index 104364387..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCD.groovy +++ /dev/null @@ -1,323 +0,0 @@ -package com.cloudogu.gitops.tools.core.argocd - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.infrastructure.kubernetes.rbac.RbacDefinition -import com.cloudogu.gitops.infrastructure.kubernetes.rbac.Role -import com.cloudogu.gitops.tools.common.Tool -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.MapUtils -import groovy.util.logging.Slf4j -import io.micronaut.core.annotation.Order -import jakarta.inject.Singleton -import org.springframework.security.crypto.bcrypt.BCrypt - -import java.nio.file.Path - -@Slf4j -@Singleton -@Order(100) -class ArgoCD extends Tool { - - private final String namespace - private final Config config - private final K8sClient k8sClient - private final HelmClient helmClient - private final FileSystemUtils fileSystemUtils - private final GitRepoFactory repoProvider - private final GitHandler gitHandler - private final String password - - private ArgoCDRepoSetup repoSetup - private RepoLayout clusterResourcesRepo - - ArgoCD(Config config, - K8sClient k8sClient, - HelmClient helmClient, - FileSystemUtils fileSystemUtils, - GitRepoFactory repoProvider, - GitHandler gitHandler) { - this.repoProvider = repoProvider - this.config = config - this.k8sClient = k8sClient - this.helmClient = helmClient - this.fileSystemUtils = fileSystemUtils - this.gitHandler = gitHandler - this.password = config.application.password - this.namespace = "${config.application.namePrefix}${config.features.argocd.namespace}" - } - - @Override - boolean isEnabled() { - config.features.argocd.active - } - - @Override - void postConfigInit(Config configToSet) { - // Exit early if not in operator mode or if env list is empty - if (!configToSet.features.argocd.operator || !configToSet.features.argocd.env) { - log.debug("Skipping features.argocd.env validation: operator mode is disabled or env list is empty.") - return - } - - List env = configToSet.features.argocd.env as List> - - log.info("Validating env list in features.argocd.env with {} entries.", env.size()) - - env.each { map -> - if (!(map instanceof Map) || !map.containsKey('name') || !map.containsKey('value')) { - throw new IllegalArgumentException("Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: $map") - } - } - - log.info("Env list validation for features.argocd.env completed successfully.") - } - - @Override - void enable() { - this.repoSetup = ArgoCDRepoSetup.create(config, fileSystemUtils, repoProvider, gitHandler) - this.clusterResourcesRepo = repoSetup.clusterRepoLayout() - - log.debug('Cloning Repositories') - repoSetup.initLocalRepos() - repoSetup.prepareClusterResourcesRepo() - repoSetup.commitAndPushAll('Initial Commit') - - log.debug('Installing Argo CD') - installArgoCd() - } - - private void installArgoCd() { - - log.debug("Creating namespaces") - k8sClient.createNamespaces(config.application.namespaces.activeNamespaces.toList()) - - createSCMCredentialsSecret() - - if (config.features.mail.smtpUser || config.features.mail.smtpPassword) { - k8sClient.createSecret('generic', - 'argocd-notifications-secret', - namespace, - new Tuple2('email-username', config.features.mail.smtpUser), - new Tuple2('email-password', config.features.mail.smtpPassword)) - } - - if (config.features.argocd.operator) { - generateRBAC() - deployWithOperator() - } else { - if (this.config.features.argocd?.values) { - String argocdConfigPath = clusterResourcesRepo.helmValuesFile() - log.debug("extend Argocd values.yaml with ${this.config.features.argocd.values}") - def argocdYaml = fileSystemUtils.readYaml(Path.of(argocdConfigPath)) - - def result = MapUtils.deepMerge(this.config.features.argocd.values, argocdYaml) - fileSystemUtils.writeYaml(result, new File(argocdConfigPath)) - log.debug("Argocd values.yaml contains ${result}") - } - deployWithHelm() - } - - if (config.multiTenant.useDedicatedInstance) { - //Bootstrapping dedicated instance - k8sClient.applyYaml(Path.of(clusterResourcesRepo.projectsDir(), "tenant.yaml").toString()) - k8sClient.applyYaml(Path.of(clusterResourcesRepo.applicationsDir(), "bootstrap.yaml").toString()) - - //Bootstrapping tenant Argocd projects - RepoLayout tenantRepoLayout = repoSetup.tenantRepoLayout() - k8sClient.applyYaml(Path.of(tenantRepoLayout.projectsDir(), "argocd.yaml").toString()) - k8sClient.applyYaml(Path.of(tenantRepoLayout.applicationsDir(), "bootstrap.yaml").toString()) - } else { - // Bootstrap root application - k8sClient.applyYaml(Path.of(clusterResourcesRepo.projectsDir(), "argocd.yaml").toString()) - k8sClient.applyYaml(Path.of(clusterResourcesRepo.applicationsDir(), "bootstrap.yaml").toString()) - } - - // Delete helm-argo secrets to decouple from helm. - // This does not delete Argo from the cluster, but you can no longer modify argo directly with helm - // For development keeping it in helm makes it easier (e.g. for helm uninstall). - k8sClient.delete('secret', namespace, - new Tuple2('owner', 'helm'), new Tuple2('name', 'argocd')) - } - - private void deployWithOperator() { - // Apply argocd yaml from operator folder - String argocdConfigPath = clusterResourcesRepo.operatorConfigFile() - if (this.config.features.argocd?.values) { - log.debug("extend Argocd.yaml with ${this.config.features.argocd.values}") - def argocdYaml = fileSystemUtils.readYaml(Path.of(clusterResourcesRepo.operatorConfigFile())) - - def result = MapUtils.deepMerge(this.config.features.argocd.values, argocdYaml) - fileSystemUtils.writeYaml(result, new File(argocdConfigPath)) - log.debug("Argocd.yaml for operator contains ${result}") - // reload file - argocdConfigPath = clusterResourcesRepo.operatorConfigFile() - } - k8sClient.applyYaml(argocdConfigPath) - - // ArgoCD is not installed until the ArgoCD-Operator did his job. - // This can take some time, so we wait for the status of the custom resource to become "Available" - k8sClient.waitForResourcePhase("argocd", "argocd", namespace, "Available") - - log.debug("Setting new argocd admin password") - // Set admin password imperatively here instead of operator/argocd.yaml, because we don't want it to show in git repo - // The Operator uses an extra secret to store the admin Password, which is not bcrypted - k8sClient.patch('secret', 'argocd-cluster', namespace, - [stringData: ['admin.password': password]]) - - // In newer Versions ArgoCD Operator uses the password in argocd-cluster secret only as generated initial password - // but we want to set our own admin password so we set the password in both Secrets for consistency - String bcryptArgoCDPassword = BCrypt.hashpw(password, BCrypt.gensalt(4)) - k8sClient.patch('secret', 'argocd-secret', namespace, - [stringData: ['admin.password': bcryptArgoCDPassword]]) - - updatingArgoCDManagedNamespaces() - - log.debug("Apply RBAC permissions for ArgoCD in all managed namespaces imperatively") - // Apply rbac yamls from operator/rbac folder - String argocdRbacPath = clusterResourcesRepo.operatorRbacDir() - k8sClient.applyYaml(argocdRbacPath) - } - - private void deployWithHelm() { - - // Install umbrella chart from argocd/argocd - String umbrellaChartPath = clusterResourcesRepo.helmDir() - // Even if the Chart.lock already contains the repo, we need to add it before resolving it - // See https://github.com/helm/helm/issues/8036#issuecomment-872502901 - List helmDependencies = fileSystemUtils.readYaml(Path.of(clusterResourcesRepo.chartYaml()))['dependencies'].collect { it } - helmClient.addRepo('argo', helmDependencies[0]['repository'] as String) - helmClient.dependencyBuild(umbrellaChartPath) - helmClient.upgrade('argocd', umbrellaChartPath, [namespace: namespace]) - - log.debug("Setting new argocd admin password") - // Set admin password imperatively here instead of values.yaml, because we don't want it to show in git repo - String bcryptArgoCDPassword = BCrypt.hashpw(password, BCrypt.gensalt(4)) - k8sClient.patch('secret', 'argocd-secret', namespace, - [stringData: ['admin.password': bcryptArgoCDPassword]]) - - } - - // The ArgoCD instance installed via an operator only manages its deployment namespace. - // To manage additional namespaces, we need to update the 'argocd-default-cluster-config' secret with all managed namespaces. - void updatingArgoCDManagedNamespaces() { - - log.debug("Updating managed namespaces in ArgoCD configuration secret.") - def namespaceList = !config.multiTenant.useDedicatedInstance ? config.application.namespaces.activeNamespaces : config.application.namespaces.tenantNamespaces - - k8sClient.patch('secret', 'argocd-default-cluster-config', namespace, - [stringData: ['namespaces': namespaceList.join(',')]]) - - if (config.multiTenant.useDedicatedInstance) { - // Append new namespaces to existing ones from the secret. - // `kubectl patch` can't merge list subfields, so we read, decode, merge, and update the secret. - // This ensures all centrally managed namespaces are preserved. - String base64Namespaces = k8sClient.getArgoCDNamespacesSecret('argocd-default-cluster-config', config.multiTenant.centralArgocdNamespace) - byte[] decodedBytes = Base64.decoder.decode(base64Namespaces) - String decoded = new String(decodedBytes, "UTF-8") - def decodedList = decoded?.split(',') as List ?: [] - def activeList = config.application.namespaces.activeNamespaces?.flatten() as List ?: [] - def merged = (decodedList + activeList).unique().join(',') - log.debug("Updating Central Argocd 'argocd-default-cluster-config' secret") - k8sClient.patch('secret', 'argocd-default-cluster-config', config.multiTenant.centralArgocdNamespace, - [stringData: ['namespaces': merged]]) - } - } - - private void generateRBAC() { - - log.debug("Generate RBAC permissions for ArgoCD in all managed namespaces") - - if (config.multiTenant.useDedicatedInstance) { - //Generating Tenant Namespace RBACs for Tenant Argocd - for (String ns : config.application.namespaces.tenantNamespaces) { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("argocd") - .withNamespace(ns) - .withServiceAccountsFrom(namespace, - ["argocd-argocd-server", "argocd-argocd-application-controller", "argocd-applicationset-controller"]) - .withConfig(config) - .withRepo(repoSetup.clusterResources.repo) - .withSubfolder(clusterResourcesRepo.operatorRbacTenantSubfolder()) - .generate() - } - - //Generating Central ArgoCD RBACs for managed namespaces - for (String ns : config.application.namespaces.activeNamespaces) { - log.debug("Generate RBAC permissions for centralized ArgoCD to access tenant ArgoCDs") - new RbacDefinition(Role.Variant.ARGOCD) - .withName('argocd-central') - .withNamespace(ns) - .withServiceAccountsFrom(config.multiTenant.centralArgocdNamespace, - ["argocd-argocd-server", "argocd-argocd-application-controller", "argocd-applicationset-controller"]) - .withConfig(config) - .withRepo(repoSetup.clusterResources.repo) - .withSubfolder(clusterResourcesRepo.operatorRbacSubfolder()) - .generate() - } - } else { - for (String ns : config.application.namespaces.activeNamespaces) { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("argocd") - .withNamespace(ns) - .withServiceAccountsFrom(namespace, - ["argocd-argocd-server", "argocd-argocd-application-controller", "argocd-applicationset-controller"]) - .withConfig(config) - .withRepo(repoSetup.clusterResources.repo) - .withSubfolder(clusterResourcesRepo.operatorRbacSubfolder()) - .generate() - } - - if (config.application.clusterAdmin) { - new RbacDefinition(Role.Variant.CLUSTER_ADMIN) - .withName("argocd-cluster-admin") - .withNamespace(namespace) - .withServiceAccountsFrom(namespace, - ["argocd-argocd-server", "argocd-argocd-application-controller", "argocd-applicationset-controller"]) - .withConfig(config) - .withRepo(repoSetup.clusterResources.repo) - .withSubfolder(clusterResourcesRepo.operatorRbacSubfolder()) - .generate() - } - } - } - - protected void createSCMCredentialsSecret() { - log.debug("Creating repo credential secret that is used by argocd to access repos in ${config.scm.scmProviderType.toString()}") - - // Create secret imperatively here instead of values.yaml, because we don't want it to show in git repo - createRepoCredentialsSecret('argocd-repo-creds-scm', - namespace, - gitHandler.tenant.url, - gitHandler.tenant.credentials.username, - gitHandler.tenant.credentials.password) - - if (config.multiTenant.useDedicatedInstance) { - log.debug("Creating central repo credential secret that is used by argocd to access repos in ${config.scm.scmProviderType.toString()}") - - // Create secret imperatively here instead of values.yaml, because we don't want it to show in git repo - createRepoCredentialsSecret('argocd-repo-creds-central-scm', - config.multiTenant.centralArgocdNamespace, - gitHandler.central.url, - gitHandler.central.credentials.username, - gitHandler.central.credentials.password) - } - } - - private void createRepoCredentialsSecret(String secretName, String ns, String url, String username, String password) { - k8sClient.createSecret('generic', secretName, ns, - new Tuple2('url', url), - new Tuple2('username', username), - new Tuple2('password', password)) - k8sClient.label('secret', secretName, ns, - new Tuple2('argocd.argoproj.io/secret-type', 'repo-creds')) - } - - protected ArgoCDRepoSetup getRepoSetup() { - return this.repoSetup - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetup.groovy b/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetup.groovy deleted file mode 100644 index f8d700eae..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetup.groovy +++ /dev/null @@ -1,154 +0,0 @@ -package com.cloudogu.gitops.tools.core.argocd - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.util.logging.Slf4j - -import java.nio.file.Path - -/** - * Holds ArgoCD-related repo initialization actions (cluster-resources + optional tenant bootstrap) - * and encapsulates the initialization logic (single-instance vs. dedicated instance).*/ -@Slf4j -class ArgoCDRepoSetup { - - final RepoInitializationAction clusterResources - final RepoInitializationAction tenantBootstrap - // may be null - final List allRepos - - private final Config config - private final FileSystemUtils fileSystemUtils - - private ArgoCDRepoSetup(Config config, - FileSystemUtils fileSystemUtils, - RepoInitializationAction clusterResources, - RepoInitializationAction tenantBootstrap, - List allRepos) { - this.config = config - this.fileSystemUtils = fileSystemUtils - this.clusterResources = clusterResources - this.tenantBootstrap = tenantBootstrap - this.allRepos = allRepos - } - - static ArgoCDRepoSetup create(Config config, FileSystemUtils fileSystemUtils, GitRepoFactory repoFactory, GitHandler gitHandler) { - RepoInitializationAction cluster - RepoInitializationAction tenant - List all = [] - - if (config.multiTenant.useDedicatedInstance) { - // Dedicated instance: tenant bootstrap (tenant provider) + cluster-resources (central provider) - tenant = createRepoInitializationAction(config, repoFactory, gitHandler, - 'argocd/cluster-resources/apps/argocd/multiTenant/tenant', - 'argocd/cluster-resources', - gitHandler.tenant) - all.add(tenant) - - cluster = createRepoInitializationAction(config, repoFactory, gitHandler, - 'argocd/cluster-resources', - 'argocd/cluster-resources', - gitHandler.central) - all.add(cluster) - - } else { - // Single instance: only cluster-resources (tenant provider) - cluster = createRepoInitializationAction(config, repoFactory, gitHandler, - 'argocd/cluster-resources', - 'argocd/cluster-resources', - gitHandler.tenant) - all.add(cluster) - } - - // Configure which subdirectories should be copied into the cluster-resources repo - cluster.subDirsToCopy = determineClusterResourceSubDirs(config) - - return new ArgoCDRepoSetup(config, fileSystemUtils, cluster, tenant, all) - } - - RepoLayout clusterRepoLayout() { - new RepoLayout(clusterResources.repo.getAbsoluteLocalRepoTmpDir()) - } - - RepoLayout tenantRepoLayout() { - if (tenantBootstrap == null) { - throw new IllegalStateException("tenantBootstrap repo is not initialized (single-instance mode).") - } - new RepoLayout(tenantBootstrap.repo.getAbsoluteLocalRepoTmpDir()) - } - - void initLocalRepos() { - allRepos.each { it.initLocalRepo() } - } - - void prepareClusterResourcesRepo() { - RepoLayout layout = clusterRepoLayout() - - if (config.features.argocd.operator) { - fileSystemUtils.deleteDir(layout.helmDir()) - } else { - fileSystemUtils.deleteDir(layout.operatorDir()) - } - - if (config.multiTenant.useDedicatedInstance) { - log.debug("Deleting unnecessary non dedicated instances folders from argocd repo: applications=${clusterRepoLayout().applicationsDir()}, projects=${clusterRepoLayout().projectsDir()}, tenant=${clusterRepoLayout().multiTenantDir()}/tenant") - FileSystemUtils.deleteDir clusterRepoLayout().applicationsDir() - FileSystemUtils.deleteDir clusterRepoLayout().projectsDir() - fileSystemUtils.moveDirectoryMergeOverwrite(Path.of(clusterRepoLayout().multiTenantDir() + "/central"), Path.of(clusterRepoLayout().argocdRoot())) - FileSystemUtils.deleteDir clusterRepoLayout().multiTenantDir() - } else { - fileSystemUtils.deleteDir(layout.multiTenantDir()) - } - - if (!config.application.netpols) { - fileSystemUtils.deleteFile(layout.netpolFile()) - } - } - - void commitAndPushAll(String message) { - allRepos.each { it.repo.commitAndPush(message) } - } - - private static Set determineClusterResourceSubDirs(Config config) { - Set clusterResourceSubDirs = new LinkedHashSet<>() - - clusterResourceSubDirs.add(RepoLayout.argocdSubdirRel()) - - if (config.features.certManager.active) { - clusterResourceSubDirs.add(RepoLayout.certManagerSubdirRel()) - } - if (config.features.ingress.active) { - clusterResourceSubDirs.add(RepoLayout.ingressSubdirRel()) - } - if (config.jenkins.active) { - clusterResourceSubDirs.add(RepoLayout.jenkinsSubdirRel()) - } - if (config.features.monitoring.active) { - clusterResourceSubDirs.add(RepoLayout.monitoringSubdirRel()) - } - if (config.scm.scmManager?.url) { - clusterResourceSubDirs.add(RepoLayout.scmManagerSubdirRel()) - } - if (config.features.secrets.active) { - clusterResourceSubDirs.add(RepoLayout.secretsSubdirRel()) - clusterResourceSubDirs.add(RepoLayout.vaultSubdirRel()) - } - - return clusterResourceSubDirs - } - - private static RepoInitializationAction createRepoInitializationAction(Config config, - GitRepoFactory repoFactory, - GitHandler gitHandler, - String localSrcDir, - String scmRepoTarget, - GitProvider gitProvider) { - new RepoInitializationAction(config, - repoFactory.getRepo(scmRepoTarget, gitProvider), - gitHandler, - localSrcDir) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/RepoInitializationAction.groovy b/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/RepoInitializationAction.groovy deleted file mode 100644 index 6da919904..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/RepoInitializationAction.groovy +++ /dev/null @@ -1,125 +0,0 @@ -package com.cloudogu.gitops.tools.core.argocd - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import freemarker.template.DefaultObjectWrapperBuilder -import groovy.util.logging.Slf4j - -@Slf4j -class RepoInitializationAction { - private GitRepo repo - private String copyFromDirectory - Set subDirsToCopy = [] as Set - private Config config - private GitHandler gitHandler - - RepoInitializationAction(Config config, GitRepo repo, GitHandler gitHandler, String copyFromDirectory) { - this.config = config - this.repo = repo - this.copyFromDirectory = copyFromDirectory - this.gitHandler = gitHandler - } - - /** - * Clone repo from SCM and initialize it by copying only the configured subdirectories. - * Afterwards we can edit these files.*/ - void initLocalRepo() { - repo.cloneRepo() - - log.debug("Initializing repo ${repo.repoTarget} from ${copyFromDirectory} with subdirs: ${subDirsToCopy}") - repo.copyDirectoryContents(copyFromDirectory, createSubdirFilter()) - replaceTemplates() - } - - void replaceTemplates() { - Map templateModel = buildTemplateValues(config) - repo.replaceTemplates(templateModel) - } - - GitRepo getRepo() { - return repo - } - - private Map buildTemplateValues(Config config) { - def model = [tenantName: config.application.tenantName, - argocd : [host: config.features.argocd.url ? new URL(config.features.argocd.url).host : ""], - scm : [baseUrl : this.repo.gitProvider.url, - host : this.repo.gitProvider.host, - protocol : this.repo.gitProvider.protocol, - repoUrl : this.repo.gitProvider.repoPrefix(), - centralScmUrl: this.gitHandler.central?.repoPrefix() ?: ''], - config : config, - // Allow for using static classes inside the templates - statics : new DefaultObjectWrapperBuilder(freemarker.template.Configuration.VERSION_2_3_32).build().getStaticModels()] as Map - - return model - } - - private FileFilter createSubdirFilter() { - if (!subDirsToCopy || subDirsToCopy.isEmpty()) { - return { File f -> true } as FileFilter - } - - File srcRoot = new File(copyFromDirectory).canonicalFile - - // Normalize entries like "argocd", "apps/monitoring" to "argocd/" or "apps/monitoring/" - Set prefixes = subDirsToCopy.collect { String s -> - def norm = s.replace('\\', '/') - norm = norm.replaceAll('^/+', '').replaceAll('/+$', '') - return norm + '/' - } as Set - - boolean hasPrefixes = !prefixes.isEmpty() - - // Templates that MUST be copied (chart templates), even though they match the global templates-exclude - Set templateIncludePrefixes = ['apps/argocd/argocd/templates/'] as Set - - return { File f -> - File canon = f.canonicalFile - String rel = srcRoot.toURI().relativize(canon.toURI()).toString() - rel = rel.replace('\\', '/') - - // Always copy the root (copyFromDirectory itself), otherwise we can't build up the directory structure - if (rel == '' || rel == '.') { - return true - } - - boolean isDir = f.isDirectory() - // For directories, always compare using a trailing slash - String relDir = rel.endsWith('/') ? rel : rel + '/' - - // --- Exception: keep required chart templates (e.g., ArgoCD chart templates) --- - // If the current path is inside an explicitly allowed templates subtree, always allow it. - if (templateIncludePrefixes.any { String p -> (isDir ? relDir : rel).startsWith(p) - }) { - return true - } - - // --- Global excludes for feature templates --- - // do NOT copy anything under apps/**/templates/** into the SCM repo - if (rel.startsWith('apps/') && relDir.contains('/templates/')) { - return false - } - - // If no prefixes are configured, copy everything (except templates) - if (!hasPrefixes) { - return true - } - - if (isDir) { - // Allow a directory if it is: - // - exactly one of the requested subdirs, or - // - inside one of them, or - // - a parent of one of them (needed to keep the tree structure). - return prefixes.any { String p -> relDir == p || relDir.startsWith(p) || p.startsWith(relDir) - } - } else { - // Only copy files that are directly under one of the allowed subtrees - return prefixes.any { String p -> rel.startsWith(p) - } - } - } as FileFilter - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/RepoLayout.groovy b/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/RepoLayout.groovy deleted file mode 100644 index 2ef2c3bde..000000000 --- a/src/main/groovy/com/cloudogu/gitops/tools/core/argocd/RepoLayout.groovy +++ /dev/null @@ -1,132 +0,0 @@ -package com.cloudogu.gitops.tools.core.argocd - -import java.nio.file.Path - -class RepoLayout { - private static final String APPS_MONITORING_DIR = 'apps/monitoring' - private static final String APPS_SECRETS_DIR = 'apps/external-secrets' - private static final String APPS_VAULT_DIR = 'apps/vault' - private static final String APPS_CERTMANAGER_DIR = 'apps/cert-manager' - private static final String APPS_JENKINS_DIR = 'apps/jenkins' - private static final String APPS_INGRESS_DIR = 'apps/ingress' - private static final String APPS_SCMMANAGER_DIR = 'apps/scm-manager' - private static final String APPS_ARGOCD_DIR = 'apps/argocd' - - private static final String OPERATOR_DIR = 'operator' - private static final String MULTITENANT_DIR = 'multiTenant' - private static final String APPLICATIONS_DIR = 'applications' - private static final String PROJECTS_DIR = 'projects' - private static final String HELM_DIR = 'argocd' - // argocd/argocd - private static final String NETPOL_YAML = 'templates/allow-namespaces.yaml' - - private final String repoRootDir - - RepoLayout(String repoRootDir) { - this.repoRootDir = repoRootDir - } - - String rootDir() { - repoRootDir - } - - String argocdRoot() { - Path.of(repoRootDir, APPS_ARGOCD_DIR).toString() - } - - // --- folder --- - - String operatorDir() { - Path.of(argocdRoot(), OPERATOR_DIR).toString() - } - - String operatorRbacDir() { - // "cluster-resources/apps/argocd/operator/rbac" - Path.of(operatorDir(), "rbac").toString() - } - - String operatorConfigFile() { - // "cluster-resources/apps/argocd/operator/argocd.yaml" - Path.of(operatorDir(), "argocd.yaml").toString() - } - - String multiTenantDir() { - Path.of(argocdRoot(), MULTITENANT_DIR).toString() - } - - String applicationsDir() { - Path.of(argocdRoot(), APPLICATIONS_DIR).toString() - } - - String projectsDir() { - Path.of(argocdRoot(), PROJECTS_DIR).toString() - } - - String helmDir() { - Path.of(argocdRoot(), HELM_DIR).toString() - } - - String helmValuesFile() { - // "cluster-resources/apps/argocd/argocd/values.yaml" - Path.of(helmDir(), "values.yaml").toString() - } - - String chartYaml() { - Path.of(helmDir(), "Chart.yaml").toString() - } - - String netpolFile() { - Path.of(helmDir(), NETPOL_YAML).toString() - } - - String monitoringDir() { - Path.of(repoRootDir, APPS_MONITORING_DIR).toString() - } - - String vaultDir() { - Path.of(repoRootDir, APPS_VAULT_DIR).toString() - } - - static String monitoringSubdirRel() { - APPS_MONITORING_DIR - } - - static String secretsSubdirRel() { - APPS_SECRETS_DIR - } - - static String vaultSubdirRel() { - APPS_VAULT_DIR - } - - static String certManagerSubdirRel() { - APPS_CERTMANAGER_DIR - } - - static String jenkinsSubdirRel() { - APPS_JENKINS_DIR - } - - static String ingressSubdirRel() { - APPS_INGRESS_DIR - } - - static String scmManagerSubdirRel() { - APPS_SCMMANAGER_DIR - } - - static String argocdSubdirRel() { - APPS_ARGOCD_DIR - } - - // --- relative subfolders for RBAC (passed to RbacDefinition.withSubfolder) --- - static String operatorRbacSubfolder() { - // "argocd/operator/rbac" - "${APPS_ARGOCD_DIR}/${OPERATOR_DIR}/rbac" - } - - static String operatorRbacTenantSubfolder() { - // "argocd/operator/rbac/tenant" - "${operatorRbacSubfolder()}/tenant" - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/AirGappedUtils.groovy b/src/main/groovy/com/cloudogu/gitops/utils/AirGappedUtils.groovy deleted file mode 100644 index 11f6a27e1..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/AirGappedUtils.groovy +++ /dev/null @@ -1,123 +0,0 @@ -package com.cloudogu.gitops.utils - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Config.HelmConfig -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import groovy.util.logging.Slf4j -import groovy.yaml.YamlSlurper -import jakarta.inject.Singleton - -import java.nio.file.Path - -@Slf4j -@Singleton -class AirGappedUtils { - - private Config config - private GitRepoFactory repoProvider - private FileSystemUtils fileSystemUtils - private HelmClient helmClient - private GitHandler gitHandler - - AirGappedUtils(Config config, GitRepoFactory repoProvider, - FileSystemUtils fileSystemUtils, HelmClient helmClient, GitHandler gitHandler) { - this.config = config - this.repoProvider = repoProvider - this.fileSystemUtils = fileSystemUtils - this.helmClient = helmClient - this.gitHandler = gitHandler - } - - /** - * In air-gapped mode, the chart's dependencies can't be resolved. - * As helm does not provide an option for changing them interactively, we push the charts into a separate repo. - * We alter these repos to resolve dependencies locally from SCM. - * - * @return the repo namespace and name - */ - String mirrorHelmRepoToGit(HelmConfig helmConfig) { - String repoName = helmConfig.chart - String namespace = GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES - String repoNamespaceAndName = "${namespace}/${repoName}" - String localHelmChartFolder = "${config.application.localHelmChartFolder}/${repoName}" - - validateChart(repoNamespaceAndName, localHelmChartFolder, repoName) - - GitRepo repo = repoProvider.getRepo(repoNamespaceAndName, gitHandler.tenant) - - repo.createRepositoryAndSetPermission("Mirror of Helm chart $repoName from ${helmConfig.repoURL}", false) - - repo.cloneRepo() - - repo.copyDirectoryContents(localHelmChartFolder) - - def chartYaml = localizeChartYaml(repo) - - // Chart.lock contains pinned dependencies and digest. - // We either have to update or remove them. Take the easier approach. - new File(repo.absoluteLocalRepoTmpDir, 'Chart.lock').delete() - - repo.commitAndPush("Chart ${chartYaml.name}, version: ${chartYaml.version}\n\n" + "Source: ${helmConfig.repoURL}\n" + - "Dependencies localized to run in air-gapped environments", chartYaml.version as String) - return repoNamespaceAndName - } - - private void validateChart(repoNamespaceAndName, String localHelmChartFolder, String repoName) { - log.debug("Validating helm chart before pushing it to SCM, by running helm template.\n" + "Potential repo: ${repoNamespaceAndName}, chart folder: ${localHelmChartFolder}") - try { - helmClient.template(repoName, localHelmChartFolder) - } catch (RuntimeException e) { - throw new RuntimeException("Helm chart in folder ${localHelmChartFolder} seems invalid.", e) - } - } - - private Map localizeChartYaml(GitRepo gitRepo) { - log.debug("Preparing repo ${gitRepo.repoTarget} for air-gapped use: Changing Chart.yaml to resolve depencies locally") - - def chartYamlPath = Path.of(gitRepo.absoluteLocalRepoTmpDir, 'Chart.yaml') - - Map chartYaml = new YamlSlurper().parse(chartYamlPath) as Map - Map chartLock = parseChartLockIfExists(gitRepo) - - List dependencies = chartYaml.dependencies as List ?: [] - for (Map chartYamlDep : dependencies) { - resolveDependencyVersion(chartLock, chartYamlDep, gitRepo) - - // Remove link to external repo, to force using local one - chartYamlDep.repository = '' - } - fileSystemUtils.writeYaml(chartYaml, chartYamlPath.toFile()) - return chartYaml - } - - private static Map parseChartLockIfExists(GitRepo scmmRepo) { - def chartLock = Path.of(scmmRepo.absoluteLocalRepoTmpDir, 'Chart.lock') - if (!chartLock.toFile().exists()) { - return [:] - } - new YamlSlurper().parse(chartLock) as Map - } - - /** - * Resolve proper dependency version from Chart.lock, e.g. 5.18.* -> 5.18.1*/ - private void resolveDependencyVersion(Map chartLock, Map chartYamlDep, GitRepo gitRepo) { - def chartLockDep = findByName(chartLock.dependencies as List, chartYamlDep.name as String) - if (chartLockDep) { - chartYamlDep.version = chartLockDep.version - } else if ((chartYamlDep.version as String).contains('*')) { - throw new RuntimeException("Unable to determine proper version for dependency " + "${chartYamlDep.name} (version: ${chartYamlDep.version}) from repo ${gitRepo.repoTarget}") - } - } - - Map findByName(List list, String name) { - if (!list) return [:] - // Note that list.find{} does not work in GraalVM native image: - // UnsupportedFeatureError: Runtime reflection is not supported - list.stream() - .filter(map -> map.name == name) - .findFirst().orElse([:]) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/AllowListFreemarkerObjectWrapper.groovy b/src/main/groovy/com/cloudogu/gitops/utils/AllowListFreemarkerObjectWrapper.groovy deleted file mode 100644 index 2d354389a..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/AllowListFreemarkerObjectWrapper.groovy +++ /dev/null @@ -1,33 +0,0 @@ -package com.cloudogu.gitops.utils - -import freemarker.template.* - -class AllowListFreemarkerObjectWrapper extends DefaultObjectWrapper { - - Set allowlist - - AllowListFreemarkerObjectWrapper(Version freemarkerVersion, Set allowlist) { - super(freemarkerVersion) - this.allowlist = allowlist - } - - TemplateHashModel getStaticModels() { - final TemplateHashModel originalStaticModels = super.getStaticModels() - final Set allowlistCopy = this.allowlist - - return new TemplateHashModel() { - @Override - TemplateModel get(String key) throws TemplateModelException { - if (allowlistCopy.contains(key)) { - return originalStaticModels.get(key) - } - return null - } - - @Override - boolean isEmpty() throws TemplateModelException { - return allowlistCopy.isEmpty() - } - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/CommandExecutor.groovy b/src/main/groovy/com/cloudogu/gitops/utils/CommandExecutor.groovy deleted file mode 100644 index 5db864f51..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/CommandExecutor.groovy +++ /dev/null @@ -1,139 +0,0 @@ -package com.cloudogu.gitops.utils - -import java.util.concurrent.TimeUnit -import jakarta.inject.Singleton -import groovy.util.logging.Slf4j - -import org.apache.commons.io.output.TeeOutputStream - -@Slf4j -@Singleton -class CommandExecutor { - - /* This timeout is mainly here to not freeze forever the apply process in the worst case scenario. - - Calls to init-scmm.sh and init-jenkins.sh take several minutes at best and might be slower with poor connections - to the internet. - Once they are migrated to groovy we can reduce this timeout.*/ - public static final int PROCESS_TIMEOUT_MINUTES = 15 - - Output execute(String[] command, boolean failOnError = true) { - Process proc = doExecute(command) - return getOutput(proc, command.join(" "), failOnError) - } - - /** - * Please prefer using {@link #execute(java.lang.String [ ], boolean)}, because - * it avoids quoting issues when passing arguments containing whitespaces.*/ - @Deprecated - Output execute(String command, boolean failOnError = true) { - Process proc = doExecute(command) - return getOutput(proc, command, failOnError) - } - - /** - * @param envp a List of Objects (converted to Strings using toString), each member of which has environment - * variable settings in the format name=value, or null if the subprocess should inherit - * the environment of the current process. - */ - Output execute(String command, Map additionalEnv, boolean failOnError = true) { - Map newEnv = [:] - newEnv.putAll(System.getenv()) // Copy existing environment variables - newEnv.putAll(additionalEnv) - - Process proc = doExecute(command, newEnv.collect { key, value -> "${key}=${value}" }) - return getOutput(proc, command, failOnError) - } - - Output execute(String[] command1, String[] command2, boolean failOnError = true) { - String pipedCommand = "${command1.join(' ')} | ${command2.join(' ')}" - def process1 = doExecute(command1) - def process2 = doExecute(command2) - - def finalOutput = getOutput(process1.pipeTo(process2), pipedCommand, false) - // Proc1 should have finished when proc2 has. - // Still, there is the occasional "IllegalThreadStateException: process hasn't exited"... concurrency 🤷 - // Avoid the exceptions, by explicitly waiting for the process to end - waitForOrKill(process1, command1.join(' ')) - - if (process1.exitValue() > 0) { - log.error("Pipefail! First process of command failed ${pipedCommand}.") - log.error("Stderr: ${process1.err.text.trim()}") - } - if (process2.exitValue() > 0) { - log.error("Executing command failed: ${pipedCommand}") - log.error("Stderr: ${finalOutput.stdErr}") - log.error("StdOut: ${finalOutput.stdOut}") - } - - boolean success = process1.exitValue() == 0 && process2.exitValue() == 0 - if (!success && failOnError) { - throw new RuntimeException("Executing command failed: ${pipedCommand}") - } - - return finalOutput - } - - protected Process doExecute(String command, List envp = null) { - log.trace("Executing command: '${command}'") - command.execute(envp, null) - } - - protected Process doExecute(String[] command) { - log.trace("Executing command: '${command}'") - command.execute() - } - - protected Output getOutput(Process proc, String command, boolean failOnError = true) { - ByteArrayOutputStream stdOut = new ByteArrayOutputStream() - ByteArrayOutputStream stdErr = new ByteArrayOutputStream() - TeeOutputStream teeOut, teeErr - - if (log.isTraceEnabled()) { - // While waiting for the process to finish, also print stdout and stderr streams through to the main process - teeOut = new TeeOutputStream(stdOut, System.out) - teeErr = new TeeOutputStream(stdErr, System.err) - proc.consumeProcessOutput(teeOut, teeErr) - } else { - proc.consumeProcessOutput(stdOut, stdErr) - } - - waitForOrKill(proc, command) - - // Make sure all bytes have been written, before returning output - if (teeOut) teeOut.flush() - if (teeErr) teeErr.flush() - def output = new Output(stdErr.toString().trim(), stdOut.toString().trim(), proc.exitValue()) - - if (failOnError && proc.exitValue() > 0) { - log.error("Executing command failed: ${command}") - log.error("Stderr: ${output.stdErr}") - log.error("StdOut: ${output.stdOut}") - if (failOnError) { - throw new RuntimeException("Executing command failed: ${command}") - } - } - - return output - } - - protected void waitForOrKill(Process proc, String command) { - def processFinished = proc.waitFor(PROCESS_TIMEOUT_MINUTES, TimeUnit.MINUTES) - if (!processFinished) { - log.error("Timeout waiting for command ${command}. Killing process.") - proc.waitForOrKill(1) - } - } - - static class Output { - String stdErr - String stdOut - int exitCode - - Output(String stdErr, String stdOut, int exitCode) { - this.stdErr = stdErr - this.stdOut = stdOut - this.exitCode = exitCode - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/DockerImageParser.groovy b/src/main/groovy/com/cloudogu/gitops/utils/DockerImageParser.groovy deleted file mode 100644 index 77d2de436..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/DockerImageParser.groovy +++ /dev/null @@ -1,70 +0,0 @@ -package com.cloudogu.gitops.utils - -class DockerImageParser { - static class Image { - public String registry - public String repository - public String tag - - Image(String registry, String repository, String tag) { - this.registry = registry - this.repository = repository - this.tag = tag - } - - String getRegistryAndRepositoryAsString() { - if (registry == "") { - return repository - } - - return "$registry/$repository" - } - - String getRegistry() { - return registry - } - - String getRepository() { - return repository - } - - String getTag() { - return tag - } - - @Override - String toString() { - return getRegistryAndRepositoryAsString() + ":$tag" - } - } - - static Image parse(String image) { - if (!image.contains(":")) { - // Most helm charts expect an explicit image tag, otherwise they use the version set by the app. - // This will likely be unexpected so force using a tag - throw new RuntimeException("Cannot set image '$image' due to missing tag. Must be the format '\$repository:\$tag'") - } - - // docker.io / foo/bar : latest - // ^ registry ^ repository ^ tag - // ^ ------------- image ----------------- - def tuple = splitTag(image) - def imageWithoutTag = tuple.v1 - def tag = tuple.v2 - - def parts = imageWithoutTag.split("/") - def repository = parts.takeRight(2).join("/") - parts = parts.dropRight(2) - def registry = parts.join("/") - - return new Image(registry, repository, tag) - } - - private static Tuple2 splitTag(String image) { - String[] imageParts = image.split(":") - String tag = imageParts.last() - def imageWithoutTag = imageParts.dropRight(1).join(":") - - return new Tuple2(imageWithoutTag, tag) - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/FileSystemUtils.groovy b/src/main/groovy/com/cloudogu/gitops/utils/FileSystemUtils.groovy deleted file mode 100644 index baef522b2..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/FileSystemUtils.groovy +++ /dev/null @@ -1,316 +0,0 @@ -//file:noinspection GrMethodMayBeStatic - it's not static to be able to hook in for testing -package com.cloudogu.gitops.utils - -import java.nio.file.Files -import java.nio.file.Path -import java.nio.file.StandardCopyOption -import java.util.regex.Pattern -import jakarta.inject.Singleton -import groovy.io.FileType -import groovy.util.logging.Slf4j -import groovy.yaml.YamlBuilder -import groovy.yaml.YamlSlurper - -import org.apache.commons.io.FileUtils - -@Slf4j -@Singleton -class FileSystemUtils { - - /** - * Replaces text in files. If you want to change a YAML field, better use - * {@link #readYaml(java.nio.file.Path)} and - * {@link #writeYaml(java.util.Map, java.io.File)} */ - File replaceFileContent(String folder, String fileToChange, String from, String to) { - File file = new File(folder + "/" + fileToChange) - String newConfig = file.text.replace(from, to) - file.setText(newConfig) - return file - } - - String replaceFileContent(String fileToChange, String from, String to) { - File file = new File(fileToChange) - String newConfig = file.text.replaceAll(from, to) - file.setText(newConfig) - return file - } - - String getSubstringOfFile(String fileLocation, CharSequence pattern, int from, int to) { - File file = new File(fileLocation) - String found = "" - file.readLines().forEach(line -> { - if (line.contains(pattern)) { - found = line.substring(from, to) - } - }) - return found - } - - String getSubstringOfFile(String fileLocation, CharSequence pattern, int from) { - File file = new File(fileLocation) - String found = "" - file.readLines().forEach(line -> { - if (line.contains(pattern)) { - found = line.substring(from) - } - }) - return found - } - - String getLineFromFile(String fileLocation, CharSequence pattern) { - File file = new File(fileLocation) - String found = "" - String fileText = file.getText() - String[] lines = fileText.split("\n") - for (int i = 0; i < lines.size(); i++) { - if (lines[i].contains(pattern)) { - found = lines[i] - } - } - return found - } - - List getAllLinesFromFile(String fileLocation, CharSequence pattern) { - File file = new File(fileLocation) - List foundLines = new ArrayList<>() - file.readLines().forEach(line -> { - if (line.contains(pattern)) { - foundLines.add(line) - } - }) - return foundLines - } - - static void deleteFile(String path) { - boolean successfullyDeleted = new File(path).delete() - if (!successfullyDeleted) { - log.warn("Faild to delete file ${path}") - } - } - - static void deleteDir(String path) { - boolean successfullyDeleted = new File(path).deleteDir() - if (!successfullyDeleted) { - log.warn("Faild to delete dir ${path}") - } - } - - String goBackToDir(String filePath, String directory) { - return filePath.substring(0, filePath.indexOf(directory) + directory.length()) - } - - String getRootDir() { - return System.getProperty("user.dir") - } - - List getAllFilesFromDirectoryWithEnding(String directory, String ending) { - List foundFiles = new ArrayList<>() - new File(directory).eachFileRecurse(FileType.FILES) { - if (it.name.endsWith(ending)) { - foundFiles.add(it) - } - } - return foundFiles - } - - void listDirectories(String parentDir) { - List list = [] - - File dir = new File(parentDir) - dir.eachFileRecurse(FileType.FILES) { file -> list << file - } - list.each { - println it.path - } - } - - static void makeWritable(File directory) { - if (!directory.exists()) { - return - } - directory.eachFileRecurse { file -> - if (!file.canWrite()) { - file.setWritable(true) - } - } - } - - void copyDirectory(String source, String destination) { - copyDirectory(source, destination, null) - } - - void copyDirectory(String source, String destination, FileFilter fileFilter) { - - log.debug("Copying directory " + source + " to " + destination) - File sourceDir = new File(source) - File destinationDir = new File(destination) - - try { - FileUtils.copyDirectory(sourceDir, destinationDir, fileFilter) - } catch (IOException e) { - log.error("An error occured while copying directories: ", e) - } - } - - void copyFile(String sourcePath, String destinationPath) { - File sourceFile = new File(sourcePath) - File destinationFile = new File(destinationPath) - - log.debug("Copying file from ${sourcePath} to ${destinationPath}") - - try { - File parentDir = destinationFile.getParentFile() - if (!parentDir.exists()) { - log.debug("Creating missing destination directories: ${parentDir}") - parentDir.mkdirs() - } - - FileUtils.copyFile(sourceFile, destinationFile) - log.debug("File copy completed successfully.") - } catch (IOException e) { - log.error("An error occurred while copying the file: ", e) - } - } - - void createDirectory(String directory) { - log.trace("Creating folder: " + directory) - new File(directory).mkdirs() - } - - Path copyToTempDir(String filePath) { - def sourcePath = Path.of(filePath) - def destDir = File.createTempDir("gitops-playground-").toPath() - def destPath = destDir.resolve(sourcePath.fileName) - return Files.copy(sourcePath, destPath) - } - - void deleteEmptyFiles(Path path, Pattern pathPattern) { - Files.walk(path).filter { it.size() == 0 && it.toString() =~ pathPattern }.each { Path it -> - log.trace("Deleting empty file $it") - it.toFile().delete() - } - } - - Path createTempDir() { - File.createTempDir("gitops-playground-").toPath() - } - - Path createTempFile() { - def file = File.createTempFile("gitops-playground-", '') - file.deleteOnExit() - - return file.toPath() - } - - Map readYaml(Path path) { - def ys = new YamlSlurper() - if (Files.exists(path)) { - return (ys.parse path) as Map - } - - // Fallback to classpath - String resourceName = path.toString() - // Ensure it starts with / for getResourceAsStream from root - if (!resourceName.startsWith("/")) { - resourceName = "/" + resourceName - } - - // Remove src/main/resources if present, as it's not part of the classpath in the JAR - resourceName = resourceName.replace("/src/main/resources", "") - - log.debug("Path ${path} not found on filesystem, trying classpath: ${resourceName}") - def inputStream = FileSystemUtils.class.getResourceAsStream(resourceName) - if (inputStream != null) { - return (ys.parseText(inputStream.text)) as Map - } - - log.warn("Could not find YAML at ${path} or on classpath ${resourceName}") - return [:] - } - - Path writeTempFile(Map mapValues) { - def tmpHelmValues = createTempFile() - writeYaml(mapValues, tmpHelmValues.toFile()) - return tmpHelmValues - } - - // Note that YAML builder seems to use double quotes to escape strings. So for example: - // This: log-format-upstream: '..."$request"...' - // Becomes: log-format-upstream: "...\"$request\"..." - // Harder to read but same payload. Not sure if we can do something about it. - void writeYaml(Map yaml, File file) { - def builder = new YamlBuilder() - builder yaml - file.setText(builder.toString()) - } - - void deleteFilesExcept(File parentPath, String... fileOrFolderNamesToKeep) { - for (File file : parentPath.listFiles()) { - if (file.name in fileOrFolderNamesToKeep) { - continue - } - if (!file.isDirectory()) { - file.delete() - } else { - file.deleteDir() - } - } - } - - /** - * Moves all direct children of sourceDir into an existing targetDir. - * Conflicts are overwritten. - * Directories are merged recursively.*/ - void moveDirectoryMergeOverwrite(Path sourceDir, Path targetDir) { - if (!Files.exists(targetDir)) { - Files.createDirectories(targetDir.parent) - // fast path: try moving the whole directory - try { - Files.move(sourceDir, targetDir) - return - } catch (IOException ignored) { - // fallback to merge logic - Files.createDirectories(targetDir) - } - } else if (!Files.isDirectory(targetDir)) { - // target exists as file -> overwrite it with directory - Files.delete(targetDir) - Files.createDirectories(targetDir) - } - - Files.list(sourceDir).forEach { Path child -> - Path dest = targetDir.resolve(child.fileName.toString()) - if (Files.isDirectory(child)) { - moveDirectoryMergeOverwrite(child, dest) - } else { - moveFileOverwrite(child, dest) - } - } - - // remove empty source dir - try { - Files.deleteIfExists(sourceDir) - } catch (IOException ignored) {} - } - - private void moveFileOverwrite(Path sourceFile, Path targetFile) { - Files.createDirectories(targetFile.parent) - - try { - Files.move(sourceFile, targetFile, StandardCopyOption.REPLACE_EXISTING) - } catch (IOException moveFailed) { - // cross-device fallback - Files.copy(sourceFile, targetFile, StandardCopyOption.REPLACE_EXISTING) - Files.delete(sourceFile) - } - } - - /** - * This filter can be used to copy whole directories without .git folder.*/ - static class IgnoreDotGitFolderFilter implements FileFilter { - @Override - boolean accept(File file) { - return !file.absolutePath.contains(File.separator + ".git") - } - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/MapUtils.groovy b/src/main/groovy/com/cloudogu/gitops/utils/MapUtils.groovy deleted file mode 100644 index 2e7579afb..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/MapUtils.groovy +++ /dev/null @@ -1,32 +0,0 @@ -package com.cloudogu.gitops.utils - -class MapUtils { - - static Map deepMerge(Map src, Map target) { - src.each { key, value -> - def oldVal = target.containsKey(key) ? target[key] : null - if (oldVal instanceof Map && value instanceof Map) { - target[key] = deepMerge((Map) value, (Map) oldVal) - } else { - target[key] = value - } - } - return target - } - - static Map deepMergeDefaults(Map src, Map target) { - src.each { key, value -> - if (value == null && target.containsKey(key)) { - return - } - - def oldVal = target.containsKey(key) ? target[key] : null - if (oldVal instanceof Map && value instanceof Map) { - target[key] = deepMergeDefaults((Map) value, (Map) oldVal) - } else { - target[key] = value - } - } - return target - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/NetworkingUtils.groovy b/src/main/groovy/com/cloudogu/gitops/utils/NetworkingUtils.groovy deleted file mode 100644 index 998ca6897..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/NetworkingUtils.groovy +++ /dev/null @@ -1,102 +0,0 @@ -package com.cloudogu.gitops.utils - -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import groovy.util.logging.Slf4j -import jakarta.inject.Singleton - -@Slf4j -@Singleton -class NetworkingUtils { - - private K8sClient k8sClient - private CommandExecutor commandExecutor - - NetworkingUtils(K8sClient k8sClient = new K8sClient(), - CommandExecutor commandExecutor = new CommandExecutor()) { - this.k8sClient = k8sClient - this.commandExecutor = commandExecutor - } - - String createUrl(String hostname, String port, String postfix = "") { - // argo forwards to HTTPS so symply us HTTP here - String url = "http://" + hostname + ":" + port + postfix - log.debug("Creating url: " + url) - return url - } - - String findClusterBindAddress() { - log.debug("Figuring out the address of the k8s cluster") - - String potentialClusterBindAddress = k8sClient.waitForInternalNodeIp() - potentialClusterBindAddress = potentialClusterBindAddress.replaceAll("'", "") - - String localAddress = localAddress - - log.debug("Local address: " + localAddress) - log.debug("Cluster address: " + potentialClusterBindAddress) - - if (!potentialClusterBindAddress) { - throw new RuntimeException("Could not connect to kubernetes cluster: no cluster bind address") - } - - if (localAddress == potentialClusterBindAddress) { - // This happens, when running on local cluster that runs in the host network. - // The reasons for introducing this might not be valid anymore: - // https://github.com/cloudogu/gitops-playground/commit/ea805d - // We no longer use jenkins notifications and have removed the address part from the welcome screen. - // So in the future, we might consider removing this and the whole localAdresse part to reduce complexity. - log.debug("Local address and cluster bind address are equal, so returning localhost") - return "localhost" - } else { - log.debug("Installing on external cluster, so returning cluster ip address") - return potentialClusterBindAddress - } - } - - /** - * Try to emulate the command "ip route get 1" by iterating the interfaces by index and returning first local address*/ - String getLocalAddress() { - try { - List sortedInterfaces = - Collections.list(NetworkInterface.getNetworkInterfaces()).sort { it.index } - - for (NetworkInterface anInterface : sortedInterfaces) { - for (InetAddress address : Collections.list(anInterface.inetAddresses)) { - if (!address.isLoopbackAddress() && address.isSiteLocalAddress()) { - return address.getHostAddress() - } - } - } - return '' - } catch (SocketException e) { - throw new RuntimeException("Could not determine local ip address", e) - } - } - - /** - * Legacy function with misleading name. Returns the part after the protocol of an URL. - * e.g. - * http://host:42/path returns host:42/path - * - * @return the part after http:///https://. Otherwise returns the input url. Works for urls without protocol, - * but not for outer protocols like ftp:// 😬 Good enough for here, but should be removed anyway. - */ - @Deprecated - static String getHost(String url) { - if (url.contains("https://")) return url.substring(8) - if (url.contains("http://")) return url.substring(7) - return url - } - - /** - * Extracts the protocol from an URL string. - * - * @return http or https. Defensively empty string in all other cases. - */ - @Deprecated - static String getProtocol(String url) { - if (url.contains("https://")) return "https" - if (url.contains("http://")) return "http" - return '' - } -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/TemplatingEngine.groovy b/src/main/groovy/com/cloudogu/gitops/utils/TemplatingEngine.groovy deleted file mode 100644 index da92ce65e..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/TemplatingEngine.groovy +++ /dev/null @@ -1,95 +0,0 @@ -package com.cloudogu.gitops.utils - -import java.nio.file.Files -import java.nio.file.Path -import java.util.regex.Pattern -import groovy.yaml.YamlSlurper - -import freemarker.template.Configuration -import freemarker.template.Template -import freemarker.template.Version - -class TemplatingEngine { - private Configuration engine - - TemplatingEngine(Configuration engine = null) { - def configuration = new Configuration(new Version("2.3.32")) - this.engine = engine ?: configuration - this.engine.setSharedVariable("nullToEmpty", ''); - } - - /** - * Executes template with parameters and replaces the .ftl in the file name.*/ - File replaceTemplate(File templateFile, Map parameters) { - def targetFile = new File(templateFile.toString().replace(".ftl", "")) - def rendered = template(templateFile, parameters) - - // Only write file if template has non-empty output. - // This avoids creating empty files when the entire template is skipped via <#if>. - if (rendered?.trim()) { - targetFile.text = rendered - } else { - targetFile.delete() - } - - templateFile.delete() - return targetFile - } - - /** - * Recursively templates all .ftl files in path. - * - * That is, apply {@link #replaceTemplate(java.io.File, java.util.Map)} to all files matching filepathMatches. */ - void replaceTemplates(File path, Map parameters, Pattern filepathMatches = ~/\.ftl/) { - Files.walk(path.toPath()) - .filter { filepathMatches.matcher(it.toString()).find() } - .each { Path it -> replaceTemplate(it.toFile(), parameters) } - } - - static Map templateToMap(String filePath, Map parameters) { - def hydratedString = new TemplatingEngine().template(new File(filePath), parameters) - - if (hydratedString.trim().isEmpty()) { - // Otherwise YamlSlurper returns an empty array, whereas we expect a Map - return [:] - } - return new YamlSlurper().parseText(hydratedString) as Map - } - - /** - * Executes template and writes to targetFile, keeping the template file.*/ - File template(File templateFile, File targetFile, Map parameters) { - Template template = prepareTemplate(templateFile) - template.process(parameters, targetFile.newWriter()) - - return targetFile - } - - String template(File templateFile, Map parameters) { - Template template = prepareTemplate(templateFile) - - StringWriter writer = new StringWriter() - template.process(parameters, writer) - - return writer.toString() - } - - String template(String template, Map parameters) { - StringWriter writer = new StringWriter() - Template templateObj = new Template("template", new StringReader(template), engine) - templateObj.process(parameters, writer) - return writer.toString() - } - - protected Template prepareTemplate(File templateFile) { - if (!templateFile.name.contains(".ftl")) { - throw new RuntimeException("File must contain .ftl to be a template") - } - - engine.setDirectoryForTemplateLoading(templateFile.parentFile) - - def template = engine.getTemplate(templateFile.name) - template - } - -} \ No newline at end of file diff --git a/src/main/groovy/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProvider.groovy b/src/main/groovy/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProvider.groovy deleted file mode 100644 index 63d27db0b..000000000 --- a/src/main/groovy/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProvider.groovy +++ /dev/null @@ -1,49 +0,0 @@ -package com.cloudogu.gitops.utils.jgit.helpers - -import org.eclipse.jgit.errors.UnsupportedCredentialItem -import org.eclipse.jgit.transport.CredentialItem -import org.eclipse.jgit.transport.CredentialsProvider -import org.eclipse.jgit.transport.URIish - -/** - * JGit, a project used within eclipse, is developed with an interactive UI in mind. - * The documentation for the CredentialsProvider says - * > CredentialItems are usually presented in bulk, allowing implementors to combine them into a single UI widget and streamline the authentication process for an end-user. - * This highlights the focus on the UI for an end-user. - * - * As a result, checking for SSL verification is a little clunky as we need to check for messages intended for end-users. - * - * Other options would have included overwriting the HttpConnection or saving the git configuration on disk. - * - * @link https://archive.eclipse.org/jgit/site/4.10.0.201712302008-r/apidocs/org/eclipse/jgit/transport/CredentialsProvider.html - */ -class InsecureCredentialProvider extends CredentialsProvider { - @Override - boolean isInteractive() { - return false - } - - @Override - boolean supports(CredentialItem... items) { - def message = items.find { it instanceof CredentialItem.InformationalMessage } - if (message == null) { - return false - } - - return message.promptText =~ /^A secure connection to .* could not be established/ - } - - @Override - boolean get(URIish uri, CredentialItem... items) throws UnsupportedCredentialItem { - items.findAll { it instanceof CredentialItem.YesNoType }.each { - if (it.promptText == "Skip SSL verification for this single git operation" || it.promptText =~ /^Skip SSL verification for git operations for repository/) { - (it as CredentialItem.YesNoType).setValue(true) - } else if (it.promptText == "Always skip SSL verification for this server from now on") { - // otherwise we would persistently overwrite our $HOME/.gitconfig - (it as CredentialItem.YesNoType).setValue(false) - } - } - - return true - } -} \ No newline at end of file diff --git a/src/main/java/com/cloudogu/gitops/application/Application.java b/src/main/java/com/cloudogu/gitops/application/Application.java new file mode 100644 index 000000000..54940b130 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/Application.java @@ -0,0 +1,140 @@ +package com.cloudogu.gitops.application; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.DeploymentOrchestrator; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryProvisioning; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractTool; +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.cloudogu.gitops.utils.Tuple; +import freemarker.template.Configuration; +import freemarker.template.DefaultObjectWrapperBuilder; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.extern.slf4j.Slf4j; + +import java.util.ArrayList; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; + +@Singleton +@Slf4j +public class Application { + + private static final String DEFAULT_GOP_NAMESPACE = "gop-job"; + + @Getter + private final List tools; + private final Config config; + private final ContextBuilder contextBuilder; + private final K8sClient k8sClient; + private final CredentialsResolver credentialsResolver; + private final GitHandler gitHandler; + private final RepositoryProvisioning repositoryProvisioning; + private final DeploymentOrchestrator deploymentOrchestrator; + + public Application( + Config config, + ContextBuilder contextBuilder, + K8sClient k8sClient, + CredentialsResolver credentialsResolver, + GitHandler gitHandler, + RepositoryProvisioning repositoryProvisioning, + DeploymentOrchestrator deploymentOrchestrator) { + this.config = config; + this.contextBuilder = contextBuilder; + this.k8sClient = k8sClient; + this.credentialsResolver = credentialsResolver; + this.gitHandler = gitHandler; + this.repositoryProvisioning = repositoryProvisioning; + this.deploymentOrchestrator = deploymentOrchestrator; + this.tools = deploymentOrchestrator.getTools(); + } + + public void start() { + log.debug("Starting Application"); + + gitHandler.validate(); + + DeploymentContext context = contextBuilder.build(); + + setNamespaceListToConfig(context); + storeGopInformationInSecret(); + gitHandler.prepareProviders(context); + repositoryProvisioning.prepare(context); + try (RepositoryWorkspace workspace = repositoryProvisioning.provideWorkspace(context)) { + deploymentOrchestrator.deployTools(context, workspace); + } + + log.debug("Application finished"); + } + + private void storeGopInformationInSecret() { + String namespace = DEFAULT_GOP_NAMESPACE; + if (config.getApplication().getGopNamespace() != null && !config.getApplication().getGopNamespace().isEmpty()) { + namespace = config.getApplication().getNamePrefix() + config.getApplication().getGopNamespace(); + } else if (this.k8sClient.getCurrentNamespace() != null) { + namespace = this.k8sClient.getCurrentNamespace(); + } else { + // keep default namespace + } + ResolvedCredentials applicationCredentials = credentialsResolver.resolve( + config.getApplication().getCredentials(), + config.getApplication().getUsername(), + config.getApplication().getPassword() + ); + + log.debug("Storing GOP configuration in secret 'gop-configuration' in namespace '{}'", namespace); + k8sClient.createNamespace(namespace); + k8sClient.createSecret( + "generic", + "gop-configuration", + namespace, + new Tuple<>("gop-initial-password", applicationCredentials.password()), + new Tuple<>("gop-config", config.toYaml(true)) + ); + } + + public void setNamespaceListToConfig(DeploymentContext context) { + LinkedHashSet tenantNamespaces = new LinkedHashSet<>(); + TemplatingEngine engine = new TemplatingEngine(); + + if (config.getContent() != null && config.getContent().getNamespaces() != null) { + for (String ns : config.getContent().getNamespaces()) { + try { + tenantNamespaces.add(engine.template( + ns, Map.of( + "config", + config, + "statics", + new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build() + .getStaticModels() + ) + )); + } catch (Exception e) { + throw new RuntimeException("Failed to render namespace template: " + ns, e); + } + } + config.getContent().setNamespaces(new ArrayList<>(tenantNamespaces)); + } + + LinkedHashSet dedicatedNamespaces = new LinkedHashSet<>(); + for (AbstractTool tool : this.tools) { + String activeNs = tool.getActiveNamespaceFromFeature(context); + if (activeNs != null && !activeNs.isEmpty()) { + dedicatedNamespaces.add(activeNs); + } + } + + config.getApplication().getNamespaces().setDedicatedNamespaces(dedicatedNamespaces); + config.getApplication().getNamespaces().setTenantNamespaces(tenantNamespaces); + log.debug("Active namespaces retrieved: {}", config.getApplication().getNamespaces().getActiveNamespaces()); + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/content/ContentLoader.java b/src/main/java/com/cloudogu/gitops/application/content/ContentLoader.java new file mode 100644 index 000000000..cc338c781 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/content/ContentLoader.java @@ -0,0 +1,907 @@ +package com.cloudogu.gitops.application.content; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Config.OverwriteMode; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractTool; +import com.cloudogu.gitops.tools.common.ConfigLifecycleHook; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.core.Jenkins; +import com.cloudogu.gitops.utils.AllowListFreemarkerObjectWrapper; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.MapUtils; +import com.cloudogu.gitops.utils.TemplatingEngine; +import freemarker.template.Configuration; +import freemarker.template.DefaultObjectWrapperBuilder; +import io.micronaut.core.annotation.Order; +import io.micronaut.core.order.Ordered; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.io.FileUtils; +import org.eclipse.jgit.api.CloneCommand; +import org.eclipse.jgit.api.FetchCommand; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.LsRemoteCommand; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static com.cloudogu.gitops.config.Config.ContentRepoType; +import static com.cloudogu.gitops.config.Config.ContentSchema.ContentRepositorySchema; + +@Singleton +@Slf4j +@Order(Ordered.LOWEST_PRECEDENCE) +public class ContentLoader extends AbstractTool implements ConfigLifecycleHook { + + private static final String CONTENT_REPOS_TYPE_PREFIX = "content.repos.type "; + private static final String REFS_HEADS_PREFIX = "refs/heads/"; + private static final String REFS_TAGS_PREFIX = "refs/tags/"; + private static final String OVERWRITE_MODE_PREFIX = "OverwriteMode "; + private static final String SET_FOR_REPO_SUFFIX = " set for repo '"; + + private final Config config; + private final K8sClient k8sClient; + private final CredentialsResolver credentialsResolver; + private final GitRepoFactory repoProvider; + private final Jenkins jenkins; + + private TemplatingEngine templatingEngine; + private List cachedRepoCoordinates = new ArrayList<>(); + protected File mergedReposFolder; + + public ContentLoader( + Config config, + K8sClient k8sClient, + CredentialsResolver credentialsResolver, + GitRepoFactory repoProvider, + Jenkins jenkins, + GitHandler gitHandler, + FileSystemUtils fileSystemUtils, + Deployer deployer) { + this.config = config; + this.k8sClient = k8sClient; + this.credentialsResolver = credentialsResolver; + this.repoProvider = repoProvider; + this.jenkins = jenkins; + this.gitHandler = gitHandler; + this.fileSystemUtils = fileSystemUtils; + this.deployer = deployer; + } + + @Override + public boolean isEnabled(DeploymentContext context) { + return true; // for now always on + } + + @Override + protected void deploy() { + try { + clearCache(); + cachedRepoCoordinates = cloneContentRepos(); + createImagePullSecrets(); + createContentRepos(); + deployHelmReleasesFromContent(); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException("Failed to load and deploy content", e); + } + } + + @Override + public void validate() { + // No additional validation needed beyond preConfigInit + } + + @Override + public void preConfigInit(Config configToSet) { + if (configToSet.getContent() == null || configToSet.getContent().getRepos() == null) { + return; + } + + for (ContentRepositorySchema repo : configToSet.getContent().getRepos()) { + validateRepo(repo); + } + } + + private static void validateRepo(ContentRepositorySchema repo) { + if (repo.getUrl() == null || repo.getUrl().isEmpty()) { + throw new IllegalArgumentException("content.repos requires a url parameter."); + } + if (repo.getTarget() != null && !repo.getTarget().isEmpty() && !repo.getTarget().contains("/")) { + throw new IllegalArgumentException( + "content.target needs / to separate namespace/group from repo name. Repo: " + repo.getUrl()); + } + + switch (repo.getType()) { + case COPY: + validateCopyRepo(repo); + break; + case FOLDER_BASED: + validateFolderBasedRepo(repo); + break; + case MIRROR: + validateMirrorRepo(repo); + break; + } + } + + private static void validateCopyRepo(ContentRepositorySchema repo) { + if (repo.getTarget() == null || repo.getTarget().isEmpty()) { + throw new IllegalArgumentException(CONTENT_REPOS_TYPE_PREFIX + ContentRepoType.COPY + " requires content.repos.target to be set. Repo: " + repo.getUrl()); + } + } + + private static void validateFolderBasedRepo(ContentRepositorySchema repo) { + if (repo.getTarget() != null && !repo.getTarget().isEmpty()) { + throw new IllegalArgumentException(CONTENT_REPOS_TYPE_PREFIX + ContentRepoType.FOLDER_BASED + " does not support target parameter. Repo: " + repo.getUrl()); + } + if (repo.getTargetRef() != null && !repo.getTargetRef().isEmpty()) { + throw new IllegalArgumentException(CONTENT_REPOS_TYPE_PREFIX + ContentRepoType.FOLDER_BASED + " does not support targetRef parameter. Repo: " + repo.getUrl()); + } + } + + private static void validateMirrorRepo(ContentRepositorySchema repo) { + if (repo.getTarget() == null || repo.getTarget().isEmpty()) { + throw new IllegalArgumentException(CONTENT_REPOS_TYPE_PREFIX + ContentRepoType.MIRROR + " requires content.repos.target to be set. Repo: " + repo.getUrl()); + } + if (!ContentRepositorySchema.DEFAULT_PATH.equals(repo.getPath())) { + throw new IllegalArgumentException(CONTENT_REPOS_TYPE_PREFIX + ContentRepoType.MIRROR + " does not support path. Current path: " + repo.getPath() + ". Repo: " + repo.getUrl()); + } + if (repo.getTemplating()) { + throw new IllegalArgumentException(CONTENT_REPOS_TYPE_PREFIX + ContentRepoType.MIRROR + " does not support templating. Repo: " + repo.getUrl()); + } + } + + protected void deployHelmReleasesFromContent() throws GitAPIException { + if (getConfig().getContent() == null || getConfig().getContent() + .getHelmReleases() == null || getConfig().getContent() + .getHelmReleases() + .isEmpty()) { + log.debug("No content.helmReleases configured - skipping."); + return; + } + + for (Config.ContentSchema.HelmReleaseSchema helmRelease : getConfig().getContent().getHelmReleases()) { + deployHelmReleaseFromContent(helmRelease); + } + } + + private void deployHelmReleaseFromContent(Config.ContentSchema.HelmReleaseSchema helmRelease) throws GitAPIException { + String version = helmRelease.getVersion() != null ? helmRelease.getVersion().trim() : ""; + if (version.isEmpty()) { + version = "*"; + } + + HelmChartConfig helmConfig = HelmChartConfig.builder() + .repoURL(helmRelease.getRepoURL()) + .chart(helmRelease.getChart()) + .version(version) + .values(new HashMap<>()) + .localHelmChartFolder(getConfig().getApplication().getLocalHelmChartFolder()) + .build(); + + Map fileValues = new HashMap<>(); + if (helmRelease.getValuesPath() != null && !helmRelease.getValuesPath().trim().isEmpty()) { + Map readValues = fileSystemUtils.readYaml(Path.of(helmRelease.getValuesPath())); + if (readValues != null) { + fileValues = readValues; + } + } + + Map inlineValues = helmRelease.getValues() != null ? helmRelease.getValues() : Collections.emptyMap(); + + Map mergedValues = MapUtils.deepMerge(inlineValues, fileValues); + + Path mergedValuesFile = fileSystemUtils.writeTempFile(mergedValues); + String mergedValuesFilePath = mergedValuesFile.toString(); + + String releaseName = (helmRelease.getReleaseName() != null && !helmRelease.getReleaseName() + .isEmpty()) ? helmRelease.getReleaseName() : helmRelease.getName(); + + deployHelmChart( + helmRelease.getName(), + releaseName, + helmRelease.getNamespace(), + helmConfig, + mergedValuesFilePath, + context, + false + ); + + repositoryWorkspace.commitAndPushClusterResourcesChanges("Update " + releaseName + " GitOps resources"); + } + + void createImagePullSecrets() { + if (!getConfig().getRegistry().getCreateImagePullSecrets()) { + return; + } + + ResolvedCredentials registryCredentials = resolveRegistryPullCredentials(); + ResolvedCredentials proxyCredentials = null; + if (getConfig().getRegistry().getTwoRegistries()) { + proxyCredentials = credentialsResolver.resolve( + getConfig().getRegistry().getProxyCredentials(), + getConfig().getRegistry().getProxyUsername(), + getConfig().getRegistry().getProxyPassword() + ); + } + + for (String namespace : getConfig().getContent().getNamespaces()) { + k8sClient.createNamespace(namespace); + + k8sClient.createImagePullSecret( + "registry", + namespace, + getConfig().getRegistry().getUrl(), + registryCredentials.username(), + registryCredentials.password() + ); + + k8sClient.patch( + "serviceaccount", + "default", + namespace, + Map.of("imagePullSecrets", List.of(Map.of("name", "registry"))) + ); + + if (proxyCredentials != null) { + k8sClient.createImagePullSecret( + "proxy-registry", + namespace, + getConfig().getRegistry().getProxyUrl(), + proxyCredentials.username(), + proxyCredentials.password() + ); + } + } + } + + private ResolvedCredentials resolveRegistryPullCredentials() { + Config.RegistrySchema registry = getConfig().getRegistry(); + if (referenceHasSecretLocation(registry.getReadOnlyCredentials())) { + return credentialsResolver.resolve( + registry.getReadOnlyCredentials(), + registry.getReadOnlyUsername(), + registry.getReadOnlyPassword() + ); + } + if (referenceHasSecretLocation(registry.getCredentials())) { + return credentialsResolver.resolve( + registry.getCredentials(), + registry.getUsername(), + registry.getPassword() + ); + } + + return new ResolvedCredentials( + firstNonBlank(registry.getReadOnlyUsername(), registry.getUsername()), + firstNonBlank(registry.getReadOnlyPassword(), registry.getPassword()) + ); + } + + private static String firstNonBlank(String preferred, String fallback) { + return preferred != null && !preferred.isEmpty() ? preferred : fallback; + } + + private static boolean referenceHasSecretLocation(Credentials reference) { + return reference != null + && ((reference.getSecretName() != null && !reference.getSecretName().isEmpty()) + || (reference.getSecretNamespace() != null && !reference.getSecretNamespace().isEmpty())); + } + + void createContentRepos() throws Exception { + if (cachedRepoCoordinates.isEmpty()) { + cachedRepoCoordinates = cloneContentRepos(); + } + pushTargetRepos(cachedRepoCoordinates); + clearCache(); + } + + protected List cloneContentRepos() throws Exception { + try { + mergedReposFolder = Files.createTempDirectory("gitops-playground-based-content-repos-").toFile(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to create temporary directory", e); + } + List repoCoordinates = new ArrayList<>(); + + log.debug("Aggregating structure for all {} repos.", getConfig().getContent().getRepos().size()); + for (ContentRepositorySchema repoConfig : getConfig().getContent().getRepos()) { + createRepoCoordinates(repoConfig, mergedReposFolder, repoCoordinates); + } + return repoCoordinates; + } + + private TemplatingEngine getTemplatingEngine() { + if (templatingEngine == null) { + templatingEngine = new TemplatingEngine(); + } + return templatingEngine; + } + + private void createRepoCoordinates( + ContentRepositorySchema repoConfig, + File mergedReposFolder, + List repoCoordinates) { + File repoTmpDir; + try { + repoTmpDir = Files.createTempDirectory("gitops-playground-content-repo-").toFile(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to create temporary directory", e); + } + log.debug( + "Cloning content repo, {}, revision {}, path {}, overwriteMode {}", + repoConfig.getUrl(), + repoConfig.getRef(), + repoConfig.getPath(), + repoConfig.getOverwriteMode() + ); + + UsernamePasswordCredentialsProvider credentialsProvider = null; + if (repoConfig.getCredentials() != null && repoConfig.getCredentials() + .getUsername() != null && repoConfig.getCredentials() + .getPassword() != null) { + credentialsProvider = new UsernamePasswordCredentialsProvider( + repoConfig.getCredentials() + .getUsername(), repoConfig.getCredentials() + .getPassword() + ); + } else if (repoConfig.getCredentials() != null && repoConfig.getCredentials() + .getSecretName() != null && repoConfig.getCredentials() + .getSecretNamespace() != null) { + Credentials credentials = this.k8sClient.getCredentialsFromSecret(repoConfig.getCredentials()); + credentialsProvider = new UsernamePasswordCredentialsProvider( + credentials.getUsername(), + credentials.getPassword() + ); + } else { + // no credentials configured for this repo; clone anonymously + } + + cloneToLocalFolder(repoConfig, repoTmpDir, credentialsProvider); + + File contentRepoDir = new File(repoTmpDir, repoConfig.getPath()); + applyTemplatingIfApplicable(repoConfig, contentRepoDir); + + switch (repoConfig.getType()) { + case FOLDER_BASED: + createRepoCoordinatesForTypeFolderBased( + repoConfig, + repoTmpDir, + contentRepoDir, + mergedReposFolder, + repoCoordinates + ); + try { + FileUtils.deleteDirectory(repoTmpDir); + } catch (IOException e) { + log.debug("Failed to delete temporary directory {}", repoTmpDir, e); + } + break; + case COPY: + createRepoCoordinatesForTypeCopy( + repoConfig, + contentRepoDir, + mergedReposFolder, + repoTmpDir, + repoCoordinates + ); + try { + FileUtils.deleteDirectory(repoTmpDir); + } catch (IOException e) { + log.debug("Failed to delete temporary directory {}", repoTmpDir, e); + } + break; + case MIRROR: + createRepoCoordinateForTypeMirror(repoConfig, repoTmpDir, repoCoordinates); + break; + } + log.debug("Finished cloning content repos. repoCoordinates={}", repoCoordinates); + } + + private static void createRepoCoordinatesForTypeCopy( + ContentRepositorySchema repoConfig, + File contentRepoDir, + File mergedRepoFolder, + File repoTmpDir, + List repoCoordinates) { + String namespace = repoConfig.getTarget().split("/")[0]; + String repoName = repoConfig.getTarget().split("/")[1]; + + RepoCoordinate repoCoordinate = mergeRepoDirs( + contentRepoDir, + namespace, + repoName, + mergedRepoFolder, + repoConfig + ); + repoCoordinate.refIsTag = GitRepo.isTag(repoTmpDir, repoConfig.getRef()); + addRepoCoordinates(repoCoordinates, repoCoordinate); + } + + private static void createRepoCoordinatesForTypeFolderBased( + ContentRepositorySchema repoConfig, + File repoTmpDir, + File contentRepoDir, + File mergedRepoFolder, + List repoCoordinates) { + boolean refIsTag = GitRepo.isTag(repoTmpDir, repoConfig.getRef()); + for (File contentRepoNamespaceDir : findRepoDirectories(contentRepoDir)) { + for (File contentRepoFolder : findRepoDirectories(contentRepoNamespaceDir)) { + String namespace = contentRepoNamespaceDir.getName(); + String repoName = contentRepoFolder.getName(); + RepoCoordinate repoCoordinate = mergeRepoDirs( + contentRepoFolder, + namespace, + repoName, + mergedRepoFolder, + repoConfig + ); + repoCoordinate.refIsTag = refIsTag; + addRepoCoordinates(repoCoordinates, repoCoordinate); + } + } + } + + private static void createRepoCoordinateForTypeMirror( + ContentRepositorySchema repoConfig, + File repoTmpDir, + List repoCoordinates) { + String namespace = repoConfig.getTarget().split("/")[0]; + String repoName = repoConfig.getTarget().split("/")[1]; + RepoCoordinate repoCoordinate = new RepoCoordinate(); + repoCoordinate.namespace = namespace; + repoCoordinate.repoName = repoName; + repoCoordinate.clonedContentRepo = repoTmpDir; + repoCoordinate.repoConfig = repoConfig; + repoCoordinate.refIsTag = GitRepo.isTag(repoTmpDir, repoConfig.getRef()); + addRepoCoordinates(repoCoordinates, repoCoordinate); + } + + private static RepoCoordinate mergeRepoDirs( + File src, + String namespace, + String repoName, + File mergedRepoFolder, + ContentRepositorySchema repoConfig) { + File target = new File(new File(mergedRepoFolder, namespace), repoName); + log.debug("Merging content repo, namespace {}, repoName {} from {} to {}", namespace, repoName, src, target); + try { + FileUtils.copyDirectory(src, target, new FileSystemUtils.IgnoreDotGitFolderFilter()); + } catch (IOException e) { + throw new UncheckedIOException("Failed to copy directory from " + src + " to " + target, e); + } + + RepoCoordinate repoCoordinate = new RepoCoordinate(); + repoCoordinate.namespace = namespace; + repoCoordinate.repoName = repoName; + repoCoordinate.clonedContentRepo = target; + repoCoordinate.repoConfig = repoConfig; + return repoCoordinate; + } + + private static Collection findRepoDirectories(File srcRepo) { + File[] files = srcRepo.listFiles(); + if (files == null) { + return Collections.emptyList(); + } + return Arrays.stream(files).filter(file -> file.isDirectory() && !file.getName().startsWith(".")).toList(); + } + + private void applyTemplatingIfApplicable(ContentRepositorySchema repoConfig, File srcPath) { + if (!repoConfig.getTemplating()) { + return; + } + + TemplatingEngine engine = getTemplatingEngine(); + + try (GitRepo repo = this.repoProvider.create(repoConfig.getTarget(), this.gitHandler.getTenant())) { + engine.replaceTemplates( + srcPath, Map.of( + "config", getConfig(), "scm", Map.of( + "baseUrl", + repo.getGitProvider() + .getUrl(), + "host", + repo.getGitProvider() + .getHost(), + "protocol", + repo.getGitProvider() + .getProtocol(), + "repoUrl", + repo.getGitProvider() + .repoPrefix() + ), "statics", !getConfig().getContent() + .getUseWhitelist() ? new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build() + .getStaticModels() : new AllowListFreemarkerObjectWrapper( + Configuration.VERSION_2_3_32, getConfig().getContent() + .getAllowedStaticsWhitelist() + ).getStaticModels() + ) + ); + } catch (Exception e) { + throw new RuntimeException("Failed to replace templates in " + srcPath, e); + } + } + + private void cloneToLocalFolder( + ContentRepositorySchema repoConfig, + File repoTmpDir, + UsernamePasswordCredentialsProvider credentialsProvider) { + CloneCommand cloneCommand = gitClone().setURI(repoConfig.getUrl()) + .setDirectory(repoTmpDir) + .setNoCheckout(false); + + if (credentialsProvider != null) { + cloneCommand.setCredentialsProvider(credentialsProvider); + } + + try (Git git = cloneCommand.call()) { + if (ContentRepoType.MIRROR == repoConfig.getType()) { + FetchCommand fetch = git.fetch(); + + if (credentialsProvider != null) { + fetch.setCredentialsProvider(credentialsProvider); + } + fetch.setRefSpecs("+refs/*:refs/*").call(); // Fetch all branches and tags + } + + if (repoConfig.getRef() != null && !repoConfig.getRef().isEmpty()) { + String actualRef = findRef(repoConfig, git.getRepository()); + git.checkout().setName(actualRef).call(); + } + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException("Failed to clone content repo " + repoConfig.getUrl(), e); + } + } + + private static String findRef(ContentRepositorySchema repoConfig, Repository gitRepo) { + try { + if (gitRepo.resolve(repoConfig.getRef()) != null) { + return repoConfig.getRef(); + } + + LsRemoteCommand remoteCommand = Git.lsRemoteRepository() + .setRemote(repoConfig.getUrl()) + .setHeads(true) + .setTags(true); + + Collection refs = remoteCommand.call(); + String potentialRef = null; + for (Ref ref : refs) { + if (ref.getName().equals(REFS_HEADS_PREFIX + repoConfig.getRef()) || ref.getName() + .equals(REFS_TAGS_PREFIX + repoConfig.getRef())) { + potentialRef = ref.getName(); + break; + } + } + + if (potentialRef == null) { + throw new IllegalStateException("Reference '" + repoConfig.getRef() + "' not found in content repository '" + repoConfig.getUrl() + "'"); + } + + return potentialRef.replace(REFS_HEADS_PREFIX, "origin/"); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException( + "Failed to find ref " + repoConfig.getRef() + " in repo " + repoConfig.getUrl(), + e + ); + } + } + + private void pushTargetRepos(List repoCoordinates) throws Exception { + for (RepoCoordinate repoCoordinate : repoCoordinates) { + pushTargetRepo(repoCoordinate); + } + } + + private void pushTargetRepo(RepoCoordinate repoCoordinate) throws Exception { + log.trace( + "Preparing ContentLoader target repo '{}'. type='{}', overwriteMode='{}', targetRef='{}', refIsTag='{}', source='{}'", + repoCoordinate.getFullRepoName(), + repoCoordinate.repoConfig.getType(), + repoCoordinate.repoConfig.getOverwriteMode(), + repoCoordinate.repoConfig.getTargetRef(), + repoCoordinate.refIsTag, + repoCoordinate.clonedContentRepo != null ? repoCoordinate.clonedContentRepo.getAbsolutePath() : null + ); + + try (GitRepo targetRepo = repoProvider.create(repoCoordinate.getFullRepoName(), this.gitHandler.getTenant())) { + boolean isNewRepo = targetRepo.createRepositoryAndSetPermission("", false); + log.trace( + "ContentLoader target repo '{}'. isNewRepo='{}', localTargetRepo='{}'", + repoCoordinate.getFullRepoName(), + isNewRepo, + targetRepo.getAbsoluteLocalRepoTmpDir() + ); + + if (!isValidForPush(isNewRepo, repoCoordinate)) { + log.debug( + "Skipping ContentLoader push for repo '{}'. isNewRepo='{}', overwriteMode='{}'", + repoCoordinate.getFullRepoName(), + isNewRepo, + repoCoordinate.repoConfig.getOverwriteMode() + ); + return; + } + + targetRepo.cloneRepo(); + + if (repoCoordinate.repoConfig.getType() == ContentRepoType.MIRROR) { + handleRepoMirroring(repoCoordinate, targetRepo); + } else { + copyContentAndPushTargetRepo(repoCoordinate, targetRepo, isNewRepo); + } + + createJenkinsJobIfApplicable(repoCoordinate, targetRepo); + cleanUpTargetRepoTempFolders(repoCoordinate, targetRepo); + } + } + + private static void cleanUpTargetRepoTempFolders(RepoCoordinate repoCoordinate, GitRepo targetRepo) { + log.trace( + "Cleaning ContentLoader temp folders for repo '{}'. source='{}', target='{}'", + repoCoordinate.getFullRepoName(), + repoCoordinate.clonedContentRepo != null ? repoCoordinate.clonedContentRepo.getAbsolutePath() : null, + targetRepo.getAbsoluteLocalRepoTmpDir() + ); + + try { + if (repoCoordinate.clonedContentRepo != null) { + FileUtils.deleteDirectory(repoCoordinate.clonedContentRepo); + } + FileUtils.deleteDirectory(new File(targetRepo.getAbsoluteLocalRepoTmpDir())); + } catch (IOException e) { + log.debug("Failed to clean up temp folders for repo '{}'", repoCoordinate.getFullRepoName(), e); + } + } + + private static void copyContentAndPushTargetRepo( + RepoCoordinate repoCoordinate, + GitRepo targetRepo, + boolean isNewRepo) throws Exception { + log.trace( + "Copying ContentLoader content into repo '{}'. isNewRepo='{}', overwriteMode='{}', source='{}', target='{}'", + repoCoordinate.getFullRepoName(), + isNewRepo, + repoCoordinate.repoConfig.getOverwriteMode(), + repoCoordinate.clonedContentRepo != null ? repoCoordinate.clonedContentRepo.getAbsolutePath() : null, + targetRepo.getAbsoluteLocalRepoTmpDir() + ); + + if (!isNewRepo) { + clearTargetRepoIfApplicable(repoCoordinate, targetRepo); + } + + try { + targetRepo.copyDirectoryContents( + repoCoordinate.clonedContentRepo.getAbsolutePath(), + new FileSystemUtils.IgnoreDotGitFolderFilter() + ); + } catch (Exception e) { + throw new RuntimeException("Failed to copy directory contents", e); + } + + String commitMessage = "Initialize content repo " + repoCoordinate.namespace + "/" + repoCoordinate.repoName; + String targetRefShort = repoCoordinate.repoConfig.getTargetRef() + .replace(REFS_HEADS_PREFIX, "") + .replace(REFS_TAGS_PREFIX, ""); + + if (!targetRefShort.isEmpty()) { + String refSpec = setRefSpec(repoCoordinate, targetRefShort); + log.trace( + "Committing ContentLoader repo '{}'. targetRefShort='{}', refSpec='{}'", + repoCoordinate.getFullRepoName(), + targetRefShort, + refSpec + ); + targetRepo.commitAndPush(commitMessage, targetRefShort, refSpec); + } else { + log.trace("Committing ContentLoader repo '{}' to default main branch.", repoCoordinate.getFullRepoName()); + targetRepo.commitAndPush(commitMessage); + } + } + + private static String setRefSpec(RepoCoordinate repoCoordinate, String targetRefShort) { + String refSpec; + if ((repoCoordinate.refIsTag && !repoCoordinate.repoConfig.getTargetRef() + .startsWith(REFS_HEADS_PREFIX)) || repoCoordinate.repoConfig.getTargetRef() + .startsWith( + REFS_TAGS_PREFIX)) { + refSpec = REFS_TAGS_PREFIX + targetRefShort + ":" + REFS_TAGS_PREFIX + targetRefShort; + } else { + refSpec = "HEAD:" + REFS_HEADS_PREFIX + targetRefShort; + } + return refSpec; + } + + private static void clearTargetRepoIfApplicable(RepoCoordinate repoCoordinate, GitRepo targetRepo) { + if (OverwriteMode.INIT != repoCoordinate.repoConfig.getOverwriteMode()) { + if (OverwriteMode.RESET == repoCoordinate.repoConfig.getOverwriteMode()) { + log.info( + "OverwriteMode {} set for repo '{}': Deleting existing files in repo and replacing them with new content.", + OverwriteMode.RESET, + repoCoordinate.getFullRepoName() + ); + targetRepo.clearRepo(); + } else { + log.debug( + "OverwriteMode {} set for repo '{}': Merging new content into existing repo.", + OverwriteMode.UPGRADE, + repoCoordinate.getFullRepoName() + ); + } + } + } + + private static void handleRepoMirroring(RepoCoordinate repoCoordinate, GitRepo targetRepo) throws Exception { + try (Git targetGit = Git.open(new File(targetRepo.getAbsoluteLocalRepoTmpDir()))) { + String remoteUrl = targetGit.getRepository().getConfig().getString("remote", "origin", "url"); + + FileSystemUtils.makeWritable(new File(targetRepo.getAbsoluteLocalRepoTmpDir(), ".git")); + + targetRepo.copyDirectoryContents(repoCoordinate.clonedContentRepo.getAbsolutePath()); + + targetGit.getRepository().getConfig().setString("remote", "origin", "url", remoteUrl); + targetGit.getRepository().getConfig().save(); + } catch (Exception e) { + throw new RuntimeException("Failed to open or configure mirrored Git repo", e); + } + + if (repoCoordinate.repoConfig.getRef() != null && !repoCoordinate.repoConfig.getRef().isEmpty()) { + validateCommitReferences(repoCoordinate); + if (repoCoordinate.repoConfig.getTargetRef() != null && !repoCoordinate.repoConfig.getTargetRef() + .isEmpty()) { + log.debug( + "Mirroring repo '{}' ref '{}' to target repo {}, targetRef: '{}'", + repoCoordinate.repoConfig.getUrl(), + repoCoordinate.repoConfig.getRef(), + repoCoordinate.getFullRepoName(), + repoCoordinate.repoConfig.getTargetRef() + ); + targetRepo.pushRef(repoCoordinate.repoConfig.getRef(), repoCoordinate.repoConfig.getTargetRef(), true); + } else { + log.debug( + "Mirroring repo '{}' ref '{}' to target repo {}", + repoCoordinate.repoConfig.getUrl(), + repoCoordinate.repoConfig.getRef(), + repoCoordinate.getFullRepoName() + ); + targetRepo.pushRef(repoCoordinate.repoConfig.getRef(), true); + } + } else { + log.debug( + "Mirroring whole repo '{}' to target repo {}", + repoCoordinate.repoConfig.getUrl(), + repoCoordinate.getFullRepoName() + ); + targetRepo.pushAll(true); + } + } + + private static void validateCommitReferences(RepoCoordinate repoCoordinate) { + if (GitRepo.isCommit(repoCoordinate.clonedContentRepo, repoCoordinate.repoConfig.getRef())) { + throw new IllegalArgumentException( + "Mirroring commit references is not supported for content repos at the moment. content repository '" + repoCoordinate.repoConfig.getUrl() + "', ref: " + repoCoordinate.repoConfig.getRef()); + } + } + + private void createJenkinsJobIfApplicable(RepoCoordinate repoCoordinate, GitRepo repo) { + if (repoCoordinate.repoConfig.getCreateJenkinsJob() && jenkins.isEnabled(context) && GitRepo.existFileInSomeBranch( + repo.getAbsoluteLocalRepoTmpDir(), + "Jenkinsfile" + )) { + jenkins.createJenkinsjob(repoCoordinate.namespace, repoCoordinate.namespace); + } + } + + protected CloneCommand gitClone() { + return Git.cloneRepository(); + } + + static void addRepoCoordinates(List repoCoordinates, RepoCoordinate newRepoCoordinate) { + List existingRepoCoordinates = newRepoCoordinate.findSame(repoCoordinates); + + if (!existingRepoCoordinates.isEmpty()) { + log.debug("Found existing repo coordinates for {}: {}", newRepoCoordinate, existingRepoCoordinates); + + RepoCoordinate repoCoordinateToOverwrite = newRepoCoordinate.findSameNotMirror(existingRepoCoordinates); + if (repoCoordinateToOverwrite != null) { + repoCoordinates.remove(repoCoordinateToOverwrite); + log.debug( + "Replacing existing repo coordinate {} with new one: {}", + existingRepoCoordinates, + newRepoCoordinate + ); + } + } + repoCoordinates.add(newRepoCoordinate); + } + + static boolean isValidForPush(boolean isNewRepo, RepoCoordinate repoCoordinate) { + if (!isNewRepo && OverwriteMode.INIT == repoCoordinate.repoConfig.getOverwriteMode()) { + log.warn(OVERWRITE_MODE_PREFIX + OverwriteMode.INIT + SET_FOR_REPO_SUFFIX + repoCoordinate.getFullRepoName() + "' and repo already exists in target: Not pushing content!" + "If you want to override, set " + OverwriteMode.UPGRADE + " or " + OverwriteMode.RESET + " ."); + return false; + } + return true; + } + + private Config getConfig() { + return config; + } + + private void clearCache() { + if (mergedReposFolder != null) { + try { + FileUtils.deleteDirectory(mergedReposFolder); + } catch (IOException e) { + log.debug("Failed to delete merged repos folder {}", mergedReposFolder, e); + } + } + cachedRepoCoordinates.clear(); + mergedReposFolder = null; + } + + @Getter + @Setter + @NoArgsConstructor + public static class RepoCoordinate { + private String namespace; + private String repoName; + private File clonedContentRepo; + private ContentRepositorySchema repoConfig; + private boolean refIsTag; + + @Override + public String toString() { + return "RepoCoordinates{ namespace='" + namespace + "', repoName='" + repoName + "', repoConfig.type='" + repoConfig.getType() + "', repoConfig.overwriteMode='" + repoConfig.getOverwriteMode() + "', clonedContentRepo=" + clonedContentRepo + "', refIsTag='" + refIsTag + "' }"; + } + + public String getFullRepoName() { + return namespace + "/" + repoName; + } + + public List findSame(Collection repoCoordinates) { + return repoCoordinates.stream().filter(coordinate -> coordinate.getFullRepoName().equals(getFullRepoName())).toList(); + } + + public RepoCoordinate findSameNotMirror(Collection repoCoordinates) { + return repoCoordinates.stream() + .filter(coordinate -> coordinate.getFullRepoName() + .equals(getFullRepoName()) && ContentRepoType.MIRROR != coordinate.repoConfig.getType()) + .findFirst() + .orElse(null); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/context/ContextBuilder.java b/src/main/java/com/cloudogu/gitops/application/context/ContextBuilder.java new file mode 100644 index 000000000..341bd39e5 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/context/ContextBuilder.java @@ -0,0 +1,45 @@ +package com.cloudogu.gitops.application.context; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class ContextBuilder { + + private final Config config; + + public DeploymentContext build() { + return new DeploymentContext( + tenantMode(), + scmManagerDeploymentMode(), + config.getApplication().getMirrorRepos(), + clusterDistribution() + ); + } + + private DeploymentContext.TenantMode tenantMode() { + return config.getMultiTenant() + .getUseDedicatedInstance() ? DeploymentContext.TenantMode.MULTI_TENANT : DeploymentContext.TenantMode.SINGLE_TENANT; + } + + private DeploymentContext.ScmManagerDeploymentMode scmManagerDeploymentMode() { + if (config.getScm() == null || config.getScm().getScmProviderType() != ScmProviderType.SCM_MANAGER) { + return DeploymentContext.ScmManagerDeploymentMode.DISABLED; + } + + boolean internal = config.getScm().getScmManager() != null + && Boolean.TRUE.equals(config.getScm().getScmManager().getInternal()); + + return internal + ? DeploymentContext.ScmManagerDeploymentMode.INTERNAL + : DeploymentContext.ScmManagerDeploymentMode.EXTERNAL; + } + + private DeploymentContext.ClusterDistribution clusterDistribution() { + return config.getApplication() + .getOpenshift() ? DeploymentContext.ClusterDistribution.OPENSHIFT : DeploymentContext.ClusterDistribution.KUBERNETES; + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/context/DeploymentContext.java b/src/main/java/com/cloudogu/gitops/application/context/DeploymentContext.java new file mode 100644 index 000000000..6cc27273c --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/context/DeploymentContext.java @@ -0,0 +1,50 @@ +package com.cloudogu.gitops.application.context; + +import lombok.Getter; +import lombok.RequiredArgsConstructor; + +@Getter +@RequiredArgsConstructor +public class DeploymentContext { + + private final TenantMode tenantMode; + private final ScmManagerDeploymentMode scmManagerDeploymentMode; + private final boolean airgapped; + private final ClusterDistribution clusterDistribution; + + public boolean isMultiTenant() { + return tenantMode == TenantMode.MULTI_TENANT; + } + + public boolean isSingleTenant() { + return tenantMode == TenantMode.SINGLE_TENANT; + } + + public boolean isInternalScmManager() { + return scmManagerDeploymentMode == ScmManagerDeploymentMode.INTERNAL; + } + + public boolean isExternalScmManager() { + return scmManagerDeploymentMode == ScmManagerDeploymentMode.EXTERNAL; + } + + public boolean isOpenshift() { + return clusterDistribution == ClusterDistribution.OPENSHIFT; + } + + public enum TenantMode { + SINGLE_TENANT, + MULTI_TENANT + } + + public enum ScmManagerDeploymentMode { + INTERNAL, + EXTERNAL, + DISABLED + } + + public enum ClusterDistribution { + KUBERNETES, + OPENSHIFT + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/credentials/CredentialsReference.java b/src/main/java/com/cloudogu/gitops/application/credentials/CredentialsReference.java new file mode 100644 index 000000000..6703d55b1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/credentials/CredentialsReference.java @@ -0,0 +1,24 @@ +package com.cloudogu.gitops.application.credentials; + +import com.cloudogu.gitops.config.Credentials; + +public record CredentialsReference( + String secretName, + String secretNamespace, + String usernameKey, + String passwordKey +) { + + public static CredentialsReference from(Credentials credentials) { + if (credentials == null) { + return null; + } + + return new CredentialsReference( + credentials.getSecretName(), + credentials.getSecretNamespace(), + credentials.getUsernameKey(), + credentials.getPasswordKey() + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/credentials/CredentialsResolver.java b/src/main/java/com/cloudogu/gitops/application/credentials/CredentialsResolver.java new file mode 100644 index 000000000..b1b0c2d28 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/credentials/CredentialsResolver.java @@ -0,0 +1,58 @@ +package com.cloudogu.gitops.application.credentials; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class CredentialsResolver { + + private static final String INCOMPLETE_SECRET_REFERENCE = + "Kubernetes Secret credentials require both secretName and secretNamespace"; + + private final K8sClient k8sClient; + + public ResolvedCredentials resolve( + Credentials reference, + String fallbackUsername, + String fallbackPassword) { + return resolveReference(CredentialsReference.from(reference), fallbackUsername, fallbackPassword); + } + + public ResolvedCredentials resolveReference( + CredentialsReference reference, + String fallbackUsername, + String fallbackPassword) { + if (reference == null) { + return new ResolvedCredentials(fallbackUsername, fallbackPassword); + } + + boolean secretNameConfigured = hasText(reference.secretName()); + boolean secretNamespaceConfigured = hasText(reference.secretNamespace()); + + if (!secretNameConfigured && !secretNamespaceConfigured) { + return new ResolvedCredentials(fallbackUsername, fallbackPassword); + } + if (secretNameConfigured != secretNamespaceConfigured) { + throw new IllegalArgumentException(INCOMPLETE_SECRET_REFERENCE); + } + + Credentials referenceWithFallback = new Credentials( + fallbackUsername, + null, + reference.secretName(), + reference.secretNamespace(), + reference.usernameKey(), + reference.passwordKey() + ); + + Credentials resolved = k8sClient.getCredentialsFromSecret(referenceWithFallback); + return new ResolvedCredentials(resolved.getUsername(), resolved.getPassword()); + } + + private static boolean hasText(String value) { + return value != null && !value.isBlank(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/credentials/ResolvedCredentials.java b/src/main/java/com/cloudogu/gitops/application/credentials/ResolvedCredentials.java new file mode 100644 index 000000000..8d3f62e2a --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/credentials/ResolvedCredentials.java @@ -0,0 +1,9 @@ +package com.cloudogu.gitops.application.credentials; + +public record ResolvedCredentials(String username, String password) { + + @Override + public String toString() { + return "ResolvedCredentials[username=" + username + ", password=]"; + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/orchestration/DeploymentOrchestrator.java b/src/main/java/com/cloudogu/gitops/application/orchestration/DeploymentOrchestrator.java new file mode 100644 index 000000000..1726c1e94 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/orchestration/DeploymentOrchestrator.java @@ -0,0 +1,35 @@ +package com.cloudogu.gitops.application.orchestration; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.tools.common.AbstractTool; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.util.List; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class DeploymentOrchestrator { + @Getter + private final List tools; + + public void deployTools(DeploymentContext context, RepositoryWorkspace workspace) { + log.debug("Starting tool orchestration. "); + + for (AbstractTool tool : tools) { + if (!tool.isEnabled(context)) { + log.debug("Skipping disabled tool {}", tool.getClass().getSimpleName()); + continue; + } + + log.debug("Deploying tool {}", tool.getClass().getSimpleName()); + tool.execute(context, workspace); + } + + log.debug("Tool orchestration finished."); + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/orchestration/GitHandler.java b/src/main/java/com/cloudogu/gitops/application/orchestration/GitHandler.java new file mode 100644 index 000000000..b4d160213 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/orchestration/GitHandler.java @@ -0,0 +1,197 @@ +package com.cloudogu.gitops.application.orchestration; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.gitlab.GitlabProvider; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManagerProvider; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.NetworkingUtils; +import io.micronaut.core.util.StringUtils; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class GitHandler { + + @Getter + private final K8sClient k8sClient; + + @Getter + private final NetworkingUtils networkingUtils; + + private final Config config; + private final CredentialsResolver credentialsResolver; + + @Getter + @Setter + private GitProvider tenant; + + @Getter + @Setter + private GitProvider central; + + public void validate() { + boolean gitlabRequested = config.getScm().getScmProviderType() == ScmProviderType.GITLAB; + boolean gitlabUrlConfigured = config.getScm().getGitlab() != null && !StringUtils.isEmpty(config.getScm() + .getGitlab() + .getUrl()); + if (gitlabRequested || gitlabUrlConfigured) { + config.getScm().setScmProviderType(ScmProviderType.GITLAB); + config.getScm().setScmManager(null); + + var gitlab = config.getScm().getGitlab(); + if (gitlab == null || StringUtils.isEmpty(gitlab.getUrl()) + || !credentialsConfigured(gitlab.getCredentials(), gitlab.getPassword()) + || StringUtils.isEmpty(gitlab.getParentGroupId())) { + throw new IllegalArgumentException( + "GitLab configuration incomplete: please provide url, credentials and parentGroupId"); + } + return; + } + + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + if (config.getScm().getScmManager() != null) { + String prefix = config.getApplication().getNamePrefix(); + if (prefix == null) { + prefix = ""; + } + config.getScm().getScmManager().setGitOpsUsername(prefix + "gitops"); + } + } + + public void prepareProviders(DeploymentContext context) { + this.tenant = createTenantScmProvider(); + + if (context.isMultiTenant()) { + this.central = createCentralScmProvider(); + } + } + + public GitProvider getResourcesScm() { + if (central != null) { + return central; + } + + if (tenant != null) { + return tenant; + } + + throw new IllegalStateException("No SCM provider found."); + } + + private GitProvider createTenantScmProvider() { + return switch (config.getScm().getScmProviderType()) { + case GITLAB -> { + var gitlab = config.getScm().getGitlab(); + yield new GitlabProvider( + gitlab, + resolveRuntimeCredentials( + gitlab.getCredentials(), gitlab.getUsername(), gitlab.getPassword() + ), + config.getApplication().getNamePrefix() + ); + } + case SCM_MANAGER -> { + String prefix = config.getApplication().getNamePrefix(); + if (prefix == null) { + prefix = ""; + } + var scmManager = config.getScm().getScmManager(); + yield new ScmManagerProvider( + scmManager, + resolveRuntimeCredentials( + scmManager.getCredentials(), scmManager.getUsername(), scmManager.getPassword() + ), + k8sClient, + networkingUtils, + config.getApplication().getNamePrefix(), + config.getApplication().getRunningInsideK8s(), + config.getApplication().getInsecure(), + prefix + ); + } + default -> + throw new IllegalArgumentException("Unsupported SCM provider found in TenantSCM: " + config.getScm() + .getScmProviderType()); + }; + } + + private GitProvider createCentralScmProvider() { + return switch (config.getMultiTenant().getScmProviderType()) { + case GITLAB -> { + var gitlab = config.getMultiTenant().getGitlab(); + yield new GitlabProvider( + gitlab, + resolveRuntimeCredentials( + gitlab.getCredentials(), gitlab.getUsername(), gitlab.getPassword() + ), + config.getApplication().getNamePrefix() + ); + } + case SCM_MANAGER -> { + var scmManager = config.getMultiTenant().getScmManager(); + yield new ScmManagerProvider( + scmManager, + resolveRuntimeCredentials( + scmManager.getCredentials(), scmManager.getUsername(), scmManager.getPassword() + ), + k8sClient, + networkingUtils, + config.getApplication().getNamePrefix(), + config.getApplication().getRunningInsideK8s(), + config.getApplication().getInsecure(), + centralScmManagerServicePrefix(config) + ); + } + default -> throw new IllegalArgumentException("Unsupported SCM-Central provider: " + config.getMultiTenant() + .getScmProviderType()); + }; + } + + private Credentials resolveRuntimeCredentials( + Credentials reference, + String fallbackUsername, + String fallbackPassword) { + ResolvedCredentials resolved = credentialsResolver.resolve(reference, fallbackUsername, fallbackPassword); + return new Credentials(resolved.username(), resolved.password()); + } + + private static boolean credentialsConfigured(Credentials reference, String fallbackPassword) { + if (hasText(fallbackPassword)) { + return true; + } + + return reference != null + && hasText(reference.getSecretName()) + && hasText(reference.getSecretNamespace()); + } + + private static boolean hasText(String value) { + return value != null && !value.isBlank(); + } + + private static String centralScmManagerServicePrefix(Config config) { + String namespace = config.getMultiTenant().getScmManager().getNamespace(); + if (namespace == null) { + namespace = ""; + } + namespace = namespace.strip(); + String baseNamespace = "scm-manager"; + + if (namespace.equals(baseNamespace) || !namespace.endsWith(baseNamespace)) { + return ""; + } + + return namespace.substring(0, namespace.length() - baseNamespace.length()); + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/repository/RepositoryProvisioning.java b/src/main/java/com/cloudogu/gitops/application/repository/RepositoryProvisioning.java new file mode 100644 index 000000000..78e5b9621 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/repository/RepositoryProvisioning.java @@ -0,0 +1,187 @@ +package com.cloudogu.gitops.application.repository; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; + +/** + * Prepares and makes the required GitOps repositories available during a GOP deployment. + * + *

This class is responsible for creating the shared {@link RepositoryWorkspace}, ensuring that + * the required remote repositories exist, and cloning those repositories when they are already + * available. + * + *

The main repository managed here is the {@code cluster-resources} repository. It contains the + * generated GitOps resources that are consumed by ArgoCD, for example applications and projects. + * + *

In dedicated multi-tenant setups, two repository workspaces are required: + * + *

    + *
  • the cluster-resources repository in the central SCM-Manager, used by the central ArgoCD + * instance + *
  • the tenant bootstrap repository in the tenant SCM-Manager, used to bootstrap the tenant + * ArgoCD instance + *
+ * + *

Both repositories can have the same logical repository target, but they must use separate + * local workspaces because their templates may contain overlapping paths. + * + *

This class does not generate tool-specific resources. Tools write their files into the + * prepared {@link RepositoryWorkspace}. Repository provisioning only coordinates repository + * availability, local workspace preparation, and commit/push entry points. + */ +@Singleton +@Slf4j +public class RepositoryProvisioning { + + public static final String CLUSTER_RESOURCES_REPO_TARGET = "argocd/cluster-resources"; + + private final GitRepoFactory gitRepoFactory; + private final GitHandler gitHandler; + + @Getter + @Setter + private RepositoryWorkspace workspace; + + @Getter + @Setter + private boolean repositoriesCloned; + + public RepositoryProvisioning(GitRepoFactory gitRepoFactory, GitHandler gitHandler) { + this.gitRepoFactory = gitRepoFactory; + this.gitHandler = gitHandler; + } + + public void prepare(DeploymentContext context) { + provideWorkspace(context); + + if (mustWaitForInternalScmManagerDeployment(context)) { + log.debug("Preparing local repository workspace only because internal SCM-Manager is not deployed yet."); + workspace.createLocalDirectories(); + return; + } + + ensureRemoteRepositoriesExist(); + cloneRepositories(); + } + + public RepositoryWorkspace provideWorkspace(DeploymentContext context) { + if (workspace != null) { + return workspace; + } + + if (context.isMultiTenant()) { + workspace = createDedicatedInstanceWorkspace(context); + } else { + workspace = createSingleInstanceWorkspace(context); + } + + return workspace; + } + + public void ensureRemoteRepositoriesExist() { + assertWorkspacePrepared(); + workspace.ensureRemoteRepositoriesExist(); + } + + public void cloneRepositories() { + if (repositoriesCloned) { + log.debug("Repositories already cloned. Skipping."); + return; + } + + assertWorkspacePrepared(); + try { + workspace.cloneRepositories(); + } catch (Exception e) { + throw new RuntimeException("Failed to clone repositories", e); + } + repositoriesCloned = true; + } + + public void publishClusterResourcesRepositoryChanges(String toolName) { + publishClusterResourcesRepositoryChanges(toolName, null); + } + + public void publishClusterResourcesRepositoryChanges(String toolName, String message) { + assertWorkspacePrepared(); + String actualMessage = message != null ? message : ("Update " + toolName + " resources"); + try { + workspace.commitAndPushClusterResourcesChanges(actualMessage); + } catch (Exception e) { + throw new RuntimeException("Failed to publish cluster resources repository changes", e); + } + } + + public String clusterResourcesRepoTarget() { + return CLUSTER_RESOURCES_REPO_TARGET; + } + + // Ownership of clusterResourcesRepository is handed off to the returned RepositoryWorkspace, + // which closes it in RepositoryWorkspace#close(). Sonar can't trace that across the boundary. + private RepositoryWorkspace createSingleInstanceWorkspace(DeploymentContext context) { + log.debug("Creating single-instance repository workspace."); + + GitRepo clusterResourcesRepository = gitRepoFactory.create( + clusterResourcesRepoTarget(), + gitHandler.getResourcesScm() + ); + + return new RepositoryWorkspace(clusterResourcesRepository); + } + + // Ownership of both GitRepo instances is handed off to the returned RepositoryWorkspace, + // which closes them in RepositoryWorkspace#close(). Sonar can't trace that across the boundary. + private RepositoryWorkspace createDedicatedInstanceWorkspace(DeploymentContext context) { + log.debug("Creating dedicated-instance repository workspace."); + + GitRepo clusterResourcesRepository = gitRepoFactory.create( + clusterResourcesRepoTarget(), + gitHandler.getResourcesScm() + ); + + GitRepo tenantBootstrapRepository = gitRepoFactory.create(clusterResourcesRepoTarget(), gitHandler.getTenant()); + + RepositoryWorkspace dedicatedWorkspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + validateDedicatedWorkspace(dedicatedWorkspace); + + return dedicatedWorkspace; + } + + private static void validateDedicatedWorkspace(RepositoryWorkspace workspace) { + try { + String clusterRoot = new File(workspace.clusterResourcesRootDir()).getCanonicalPath(); + String tenantRoot = new File(workspace.tenantBootstrapRootDir()).getCanonicalPath(); + + if (clusterRoot.equals(tenantRoot)) { + throw new IllegalStateException("Dedicated Multi-Tenant mode requires separate local workspaces for " + "central cluster-resources and tenant bootstrap repositories. Both resolved to: " + clusterRoot); + } + } catch (IOException e) { + throw new UncheckedIOException("Failed to resolve canonical path", e); + } + } + + private void assertWorkspacePrepared() { + if (workspace == null) { + throw new IllegalStateException( + "Repository workspace must be prepared before repository changes can be published."); + } + } + + private static boolean mustWaitForInternalScmManagerDeployment(DeploymentContext context) { + return context.isInternalScmManager(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/application/repository/RepositoryWorkspace.java b/src/main/java/com/cloudogu/gitops/application/repository/RepositoryWorkspace.java new file mode 100644 index 000000000..53c9ea921 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/application/repository/RepositoryWorkspace.java @@ -0,0 +1,244 @@ +package com.cloudogu.gitops.application.repository; + +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import lombok.Getter; +import lombok.extern.slf4j.Slf4j; +import org.eclipse.jgit.api.errors.GitAPIException; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.stream.Stream; + +/** + * Represents the prepared local GitOps repository workspace used during a GOP deployment. + * + *

The workspace provides access to the local checkout of the {@code cluster-resources} + * repository. This repository contains the generated GitOps resources that are consumed by ArgoCD, + * for example applications and projects. + * + *

In single-instance setups only the {@code cluster-resources} repository is required. In + * dedicated multi-tenant setups an additional tenant bootstrap repository is required. This second + * repository contains the bootstrap resources for the tenant ArgoCD instance, while the regular + * {@code cluster-resources} repository is used by the central ArgoCD instance to bootstrap/manage + * tenant resources. + * + *

This class does not decide which repositories are needed. That decision belongs to {@link + * RepositoryProvisioning}. This class only exposes the prepared repositories and the directory + * structure that tools can write to. + */ +@Slf4j +public class RepositoryWorkspace implements AutoCloseable { + + @Getter + private final GitRepo clusterResourcesRepository; + + @Getter + private final GitRepo tenantBootstrapRepository; + + private boolean remoteRepositoriesEnsured; + + public RepositoryWorkspace(GitRepo clusterResourcesRepository) { + this(clusterResourcesRepository, null); + } + + public RepositoryWorkspace(GitRepo clusterResourcesRepository, GitRepo tenantBootstrapRepository) { + this.clusterResourcesRepository = clusterResourcesRepository; + this.tenantBootstrapRepository = tenantBootstrapRepository; + } + + public boolean hasTenantBootstrapRepository() { + return tenantBootstrapRepository != null; + } + + /** + * Returns the tenant bootstrap repository or fails if this workspace was created for a + * single-instance setup. + */ + public GitRepo tenantBootstrapRepositoryOrFail() { + if (tenantBootstrapRepository == null) { + throw new IllegalStateException("Tenant bootstrap repository is not available in single-instance mode."); + } + + return tenantBootstrapRepository; + } + + /** + * Ensures that all remote repositories represented by this workspace exist. + * + *

The decision which repositories are part of this workspace still belongs to {@link + * RepositoryProvisioning}. This method only ensures the already prepared repository handles. + */ + public void ensureRemoteRepositoriesExist() { + if (remoteRepositoriesEnsured) { + log.debug("Remote repositories already ensured. Skipping."); + return; + } + + log.debug("Ensuring cluster resources repository. repoTarget='{}'", clusterResourcesRepository.getRepoTarget()); + + ensureRepositoryExists( + clusterResourcesRepository.getGitProvider(), + clusterResourcesRepository.getRepoTarget(), + "GitOps repo for basic cluster-resources" + ); + + if (hasTenantBootstrapRepository()) { + log.debug( + "Ensuring tenant bootstrap repository. repoTarget='{}'", + tenantBootstrapRepositoryOrFail().getRepoTarget() + ); + + ensureRepositoryExists( + tenantBootstrapRepositoryOrFail().getGitProvider(), + tenantBootstrapRepositoryOrFail().getRepoTarget(), + "GitOps repo for tenant bootstrap resources" + ); + } + + remoteRepositoriesEnsured = true; + } + + public void createLocalDirectories() { + Stream.of( + clusterResourcesRootDir(), + clusterResourcesAppsDir(), + clusterResourcesArgoCdDir(), + clusterResourcesApplicationsDir(), + clusterResourcesProjectsDir() + ).forEach(this::createDirectorySafely); + + if (hasTenantBootstrapRepository()) { + Stream.of( + tenantBootstrapRootDir(), + tenantBootstrapAppsDir(), + tenantBootstrapArgoCdDir(), + tenantBootstrapApplicationsDir(), + tenantBootstrapProjectsDir() + ).forEach(this::createDirectorySafely); + } + } + + private void createDirectorySafely(String directory) { + try { + Files.createDirectories(Path.of(directory)); + } catch (IOException e) { + throw new UncheckedIOException("Failed to create directory: " + directory, e); + } + } + + public void cloneRepositories() throws GitAPIException { + clusterResourcesRepository.cloneRepo(); + + if (hasTenantBootstrapRepository()) { + tenantBootstrapRepositoryOrFail().cloneRepo(); + } + } + + /** + * Initializes local repositories when they cannot be cloned yet. + * + *

This is needed when GOP deploys an internal SCM-Manager first. In that case, the remote + * repositories are not available at the beginning of the deployment, but tools still need local + * directories to write their generated resources. + */ + public void initLocalRepositoriesIfNeeded() throws GitAPIException { + clusterResourcesRepository.initLocalRepoIfNeeded(); + + if (hasTenantBootstrapRepository()) { + tenantBootstrapRepositoryOrFail().initLocalRepoIfNeeded(); + } + } + + public String clusterResourcesRootDir() { + return clusterResourcesRepository.getAbsoluteLocalRepoTmpDir(); + } + + public String clusterResourcesAppsDir() { + return Path.of(clusterResourcesRootDir(), "apps").toString(); + } + + public String clusterResourcesArgoCdDir() { + return Path.of(clusterResourcesAppsDir(), "argocd").toString(); + } + + public String clusterResourcesApplicationsDir() { + return Path.of(clusterResourcesArgoCdDir(), "applications").toString(); + } + + public String clusterResourcesProjectsDir() { + return Path.of(clusterResourcesArgoCdDir(), "projects").toString(); + } + + public String tenantBootstrapRootDir() { + return tenantBootstrapRepositoryOrFail().getAbsoluteLocalRepoTmpDir(); + } + + public String tenantBootstrapAppsDir() { + return Path.of(tenantBootstrapRootDir(), "apps").toString(); + } + + public String tenantBootstrapArgoCdDir() { + return Path.of(tenantBootstrapAppsDir(), "argocd").toString(); + } + + public String tenantBootstrapApplicationsDir() { + return Path.of(tenantBootstrapArgoCdDir(), "applications").toString(); + } + + public String tenantBootstrapProjectsDir() { + return Path.of(tenantBootstrapArgoCdDir(), "projects").toString(); + } + + public void commitAndPushClusterResourcesAndTenantBootstrapChanges(String message) throws GitAPIException { + commitAndPushClusterResourcesChanges(message); + + if (hasTenantBootstrapRepository()) { + commitAndPushTenantBootstrapChanges(message); + } + } + + public void commitAndPushTenantBootstrapChanges(String message) throws GitAPIException { + tenantBootstrapRepositoryOrFail().commitAndPush(message); + } + + public void commitAndPushClusterResourcesChanges(String message) throws GitAPIException { + log.debug("Committing cluster resources: {}", message); + clusterResourcesRepository.commitAndPush(message); + } + + /** + * Aligns locally initialized repositories with the remote main branch if it already exists. + */ + public void alignWithRemoteMainIfPresent() throws GitAPIException, IOException { + clusterResourcesRepository.checkoutRemoteMainIfLocalMainMissing(); + + if (hasTenantBootstrapRepository()) { + tenantBootstrapRepositoryOrFail().checkoutRemoteMainIfLocalMainMissing(); + } + } + + private static void ensureRepositoryExists(GitProvider gitProvider, String repoTarget, String description) { + gitProvider.createRepository(repoTarget, description, false); + } + + @Override + public void close() { + try { + if (clusterResourcesRepository != null) { + clusterResourcesRepository.close(); + } + } catch (Exception e) { + log.warn("Error closing cluster resources repository", e); + } + try { + if (tenantBootstrapRepository != null) { + tenantBootstrapRepository.close(); + } + } catch (Exception e) { + log.warn("Error closing tenant bootstrap repository", e); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/cli/ApplicationConfigurator.java b/src/main/java/com/cloudogu/gitops/cli/ApplicationConfigurator.java new file mode 100644 index 000000000..8d46f79c3 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/cli/ApplicationConfigurator.java @@ -0,0 +1,403 @@ +package com.cloudogu.gitops.cli; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import lombok.extern.slf4j.Slf4j; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.MalformedURLException; +import java.net.URI; +import java.util.function.Function; + +@Slf4j +public class ApplicationConfigurator { + private final Function environment; + + public ApplicationConfigurator() { + this(System::getenv); + } + + ApplicationConfigurator(Function environment) { + this.environment = environment; + } + + private static boolean hasText(String value) { + return value != null && !value.isEmpty(); + } + + private static String firstNonBlank(String preferred, String fallback) { + return hasText(preferred) ? preferred : fallback; + } + + private static boolean hasCredentials(Credentials reference, String username, String password) { + return hasCompleteSecretReference(reference) || (hasText(username) && hasText(password)); + } + + private static boolean hasCompleteSecretReference(Credentials reference) { + return reference != null + && hasText(reference.getSecretName()) + && hasText(reference.getSecretNamespace()); + } + + /** + * Sets dynamic fields and validates params + */ + public Config initConfig(Config newConfig) { + addAdditionalApplicationConfig(newConfig); + addNamePrefix(newConfig); + checkAndSetNamespaces(newConfig); + addScmConfig(newConfig); + addRegistryConfig(newConfig); + addJenkinsConfig(newConfig); + addFeatureConfig(newConfig); + evaluateBaseUrl(newConfig); + setResourceInclusionsCluster(newConfig); + setMultiTenantModeConfig(newConfig); + + return newConfig; + } + + private void addFeatureConfig(Config newConfig) { + if (newConfig.getFeatures().getSecrets().getVault().getMode() != null) { + newConfig.getFeatures().getSecrets().setActive(true); + } + + if (hasText(newConfig.getFeatures().getMail().getSmtpAddress())) { + newConfig.getFeatures().getMail().setActive(true); + } + + if (newConfig.getFeatures().getIngress().getActive() && !hasText(newConfig.getApplication().getBaseUrl())) { + log.warn( + "Ingress-controller is activated without baseUrl parameter. Services will not be accessible by hostnames. To avoid this use baseUrl with ingress. "); + } + } + + private static void addNamePrefix(Config newConfig) { + String namePrefix = newConfig.getApplication().getNamePrefix(); + if (hasText(namePrefix)) { + if (!namePrefix.endsWith("-")) { + newConfig.getApplication().setNamePrefix(namePrefix + "-"); + } + newConfig.getApplication() + .setNamePrefixForEnvVars(newConfig.getApplication() + .getNamePrefix() + .toUpperCase() + .replace('-', '_')); + } + } + + private static void addRegistryConfig(Config newConfig) { + // Process image pull secrets first, they might even be relevant if no registry is set + if (newConfig.getRegistry().getCreateImagePullSecrets()) { + boolean hasSecretCredentials = hasCompleteSecretReference(newConfig.getRegistry().getCredentials()) + || hasCompleteSecretReference(newConfig.getRegistry().getReadOnlyCredentials()); + String username = firstNonBlank( + newConfig.getRegistry().getReadOnlyUsername(), newConfig.getRegistry().getUsername() + ); + String password = firstNonBlank( + newConfig.getRegistry().getReadOnlyPassword(), newConfig.getRegistry().getPassword() + ); + + if (!hasSecretCredentials && (!hasText(username) || !hasText(password))) { + throw new IllegalArgumentException( + "createImagePullSecrets needs to be used with either registry username and password or the readOnly variants"); + } + } + + + if (hasText(newConfig.getRegistry().getUrl())) { + newConfig.getRegistry().setInternal(false); + newConfig.getRegistry().setActive(true); + } else if (newConfig.getRegistry().getActive()) { + /* Internal Docker registry must be on localhost. Otherwise docker will use HTTPS, leading to errors on + docker push in the example application's Jenkins Jobs. + Both setting up HTTPS or allowing insecure registry via daemon.json makes the playground difficult to use. + So, always use localhost. + Allow overriding the port, in case multiple playground instance run on a single host in different + k3d clusters. */ + newConfig.getRegistry().setInternal(true); + newConfig.getRegistry().setUrl("localhost:" + newConfig.getRegistry().getInternalPort()); + } else { + // Registry not active, no need to set the following values + return; + } + + if (hasText(newConfig.getRegistry().getProxyUrl())) { + newConfig.getRegistry().setTwoRegistries(true); + if (!hasCredentials( + newConfig.getRegistry().getProxyCredentials(), + newConfig.getRegistry().getProxyUsername(), + newConfig.getRegistry().getProxyPassword() + )) { + throw new IllegalArgumentException("Proxy URL needs to be used with proxy-username and proxy-password"); + } + } + } + + private void addAdditionalApplicationConfig(Config newConfig) { + if (environment.apply("KUBERNETES_SERVICE_HOST") != null) { + log.debug("installation is running in kubernetes."); + newConfig.getApplication().setRunningInsideK8s(true); + } + } + + private void addScmConfig(Config newConfig) { + log.debug("Adding additional config for SCM"); + + if (newConfig.getScm().getScmManager() != null && hasText(newConfig.getScm().getScmManager().getUrl())) { + log.debug("Setting external scmm config"); + newConfig.getScm().getScmManager().setInternal(false); + newConfig.getScm().getScmManager().setUrlForJenkins(newConfig.getScm().getScmManager().getUrl()); + } else { + log.debug("Setting configs for internal SCM-Manager"); + newConfig.getScm().getScmManager().setInternal(true); + // We use the K8s service as default name here, because it is the only option: + // "scmm.localhost" will not work inside the Pods and k3d-container IP + Port (e.g. + // 172.x.y.z:9091) + // will not work on Windows and MacOS. + String urlForJenkins = new StringBuilder("http://scmm.") + .append(newConfig.getApplication().getNamePrefix()) + .append(newConfig.getScm().getScmManager().getNamespace()) + .append(".svc.cluster.local/scm") + .toString(); + newConfig.getScm().getScmManager().setUrlForJenkins(urlForJenkins); + } + + // We probably could get rid of some of the complexity by refactoring url, host and ingress into + // a single var + if (hasText(newConfig.getApplication().getBaseUrl())) { + try { + String scmUrl = injectSubdomain( + "scmm", + newConfig.getApplication().getBaseUrl(), + newConfig.getApplication().getUrlSeparatorHyphen() + ); + + newConfig.getScm() + .getScmManager() + .setIngress(URI.create(scmUrl).toURL().getHost()); + + } catch (IllegalArgumentException | MalformedURLException e) { + throw new UncheckedIOException("Failed to evaluate SCM ingress URL", new IOException(e)); + } + } + + // When specific user/pw are not set, set them to global values + if (Config.DEFAULT_ADMIN_PW.equals(newConfig.getScm().getScmManager().getPassword())) { + newConfig.getScm().getScmManager().setPassword(newConfig.getApplication().getPassword()); + } + if (Config.DEFAULT_ADMIN_USER.equals(newConfig.getScm().getScmManager().getUsername())) { + newConfig.getScm().getScmManager().setUsername(newConfig.getApplication().getUsername()); + } + } + + private void addJenkinsConfig(Config newConfig) { + log.debug("Adding additional config for Jenkins"); + if (hasText(newConfig.getJenkins().getUrl())) { + log.debug("Setting external jenkins config"); + newConfig.getJenkins().setActive(true); + newConfig.getJenkins().setInternal(false); + newConfig.getJenkins().setUrlForScm(newConfig.getJenkins().getUrl()); + } else if (newConfig.getJenkins().getActive()) { + log.debug("Setting configs for internal jenkins"); + // We use the K8s service as default name here, because it is the only option: + // "jenkins.localhost" will not work inside the Pods and k3d-container IP + Port (e.g. + // 172.x.y.z:9090) + // will not work on Windows and MacOS. + String defaultNamespace = newConfig.getJenkins().getNamespace(); + newConfig.getJenkins() + .setUrlForScm("http://jenkins." + newConfig.getApplication() + .getNamePrefix() + defaultNamespace + ".svc.cluster.local"); + } else { + // Jenkins not active, no need to set the following values + return; + } + + if (hasText(newConfig.getApplication().getBaseUrl())) { + try { + String jenkinsUrl = injectSubdomain( + "jenkins", + newConfig.getApplication().getBaseUrl(), + newConfig.getApplication().getUrlSeparatorHyphen() + ); + + newConfig.getJenkins().setIngress(URI.create(jenkinsUrl).toURL().getHost()); + + } catch (IllegalArgumentException | MalformedURLException e) { + throw new UncheckedIOException("Failed to evaluate Jenkins ingress URL ", new IOException(e)); + } + } + + // When specific user/pw are not set, set them to global values + if (Config.DEFAULT_ADMIN_USER.equals(newConfig.getJenkins().getUsername())) { + newConfig.getJenkins().setUsername(newConfig.getApplication().getUsername()); + } + if (Config.DEFAULT_ADMIN_PW.equals(newConfig.getJenkins().getPassword())) { + newConfig.getJenkins().setPassword(newConfig.getApplication().getPassword()); + } + } + + private void evaluateBaseUrl(Config newConfig) { + String baseUrl = newConfig.getApplication().getBaseUrl(); + if (!hasText(baseUrl)) { + return; + } + log.debug("Base URL set, adapting to individual tools"); + Config.ArgoCDSchema argocd = newConfig.getFeatures().getArgocd(); + Config.MonitoringSchema monitoring = newConfig.getFeatures().getMonitoring(); + Config.SecretsSchema.VaultSchema vault = newConfig.getFeatures().getSecrets().getVault(); + boolean urlSeparatorHyphen = newConfig.getApplication().getUrlSeparatorHyphen(); + + if (argocd.getActive() && !hasText(argocd.getUrl())) { + argocd.setUrl(injectSubdomain("argocd", baseUrl, urlSeparatorHyphen)); + log.debug("Setting ArgoCD URL {}", argocd.getUrl()); + } + if (monitoring.getActive() && !hasText(monitoring.getGrafanaUrl())) { + monitoring.setGrafanaUrl(injectSubdomain("grafana", baseUrl, urlSeparatorHyphen)); + log.debug("Setting Monitoring URL {}", monitoring.getGrafanaUrl()); + } + if (newConfig.getFeatures().getSecrets().getActive() && !hasText(vault.getUrl())) { + vault.setUrl(injectSubdomain("vault", baseUrl, urlSeparatorHyphen)); + log.debug("Setting Vault URL {}", vault.getUrl()); + } + } + + public void setMultiTenantModeConfig(Config newConfig) { + if (newConfig.getMultiTenant().getUseDedicatedInstance()) { + if (!hasText(newConfig.getApplication().getNamePrefix())) { + throw new IllegalArgumentException( + "To enable Central Multi-Tenant mode, you must define a name prefix to distinguish between instances."); + } + + if (!newConfig.getFeatures().getArgocd().getOperator()) { + newConfig.getFeatures().getArgocd().setOperator(true); + } + + // Removes trailing slash from the input URL to avoid duplicated slashes in further URL + // handling + if (newConfig.getMultiTenant().getScmManager().getUrl() != null) { + String urlString = newConfig.getMultiTenant().getScmManager().getUrl(); + if (urlString.endsWith("/")) { + urlString = urlString.substring(0, urlString.length() - 1); + } + newConfig.getMultiTenant().getScmManager().setUrl(urlString); + } + + // Disabling Ingress in DedicatedInstances Mode for now. + newConfig.getFeatures().getIngress().setActive(false); + } + } + + private static String injectSubdomain(String subdomain, String baseUrl, boolean urlSeparatorHyphen) { + try { + URI uri = URI.create(baseUrl); + + String separator = urlSeparatorHyphen ? "-" : "."; + + StringBuilder newUrl = new StringBuilder(uri.getScheme()) + .append("://") + .append(subdomain) + .append(separator) + .append(uri.getHost()); + + if (uri.getPort() != -1) { + newUrl.append(":").append(uri.getPort()); + } + + // getRawPath() preserves URL encoding (like %20), matching the old URL.getPath() behavior + if (uri.getRawPath() != null) { + newUrl.append(uri.getRawPath()); + } + + return newUrl.toString(); + + } catch (IllegalArgumentException e) { + throw new UncheckedIOException( + "Failed to inject subdomain '" + subdomain + "' into base URL: " + baseUrl, + new IOException(e) + ); + } + } + + private void setResourceInclusionsCluster(Config configToSet) { + // Return early if NOT deploying via operator + if (!configToSet.getFeatures().getArgocd().getOperator()) { + log.debug("ArgoCD operator is not enabled. Skipping features.argocd.resourceInclusionsCluster setup."); + return; + } + log.info("Starting setup of features.argocd.resourceInclusionsCluster for ArgoCD Operator"); + + if (!isUrlSetAndValid(configToSet)) { + buildAndValidateURLFromEnvironment(configToSet); + } + } + + public boolean isUrlSetAndValid(Config config) { + String url = config.getFeatures().getArgocd().getResourceInclusionsCluster(); + + if (hasText(url)) { + try { + log.debug("Validating user-provided features.argocd.resourceInclusionsCluster URL: {}", url); + + // Java 20+ compliant URL validation + URI.create(url).toURL(); + + log.info("Found valid URL in features.argocd.resourceInclusionsCluster: {}", url); + return true; + } catch (IllegalArgumentException | MalformedURLException e) { + throw new IllegalArgumentException( + "Invalid URL for 'features.argocd.resourceInclusionsCluster': " + url + ".", + e + ); + } + } + return false; + } + + public void buildAndValidateURLFromEnvironment(Config config) { + log.debug("Attempting to set features.argocd.resourceInclusionsCluster via Kubernetes ENV variables."); + + String host = environment.apply("KUBERNETES_SERVICE_HOST"); + String port = environment.apply("KUBERNETES_SERVICE_PORT"); + + String errorMessage = "Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true. " + "Ensure Kubernetes environment variables 'KUBERNETES_SERVICE_HOST' and 'KUBERNETES_SERVICE_PORT' are set properly. " + "Alternatively, try setting 'features.argocd.resourceInclusionsCluster' in the config to manually override."; + + if (!hasText(host) || !hasText(port)) { + throw new IllegalStateException(errorMessage); + } + + String internalClusterUrl = "https://" + host + ":" + port; + log.debug("Constructed internal Kubernetes API Server URL: {}", internalClusterUrl); + + try { + URI.create(internalClusterUrl).toURL(); + config.getFeatures().getArgocd().setResourceInclusionsCluster(internalClusterUrl); + log.info( + "Successfully set features.argocd.resourceInclusionsCluster via Kubernetes ENV to: {}", + internalClusterUrl + ); + } catch (IllegalArgumentException | MalformedURLException e) { + throw new IllegalArgumentException(errorMessage, e); + } + } + + public void checkAndSetNamespaces(Config config) { + if (hasText(config.getApplication().getNamespace())) { + String namespace = config.getApplication().getNamespace(); + config.getApplication().setGopNamespace(namespace); + config.getRegistry().setNamespace(namespace); + config.getJenkins().setNamespace(namespace); + config.getScm().getScmManager().setNamespace(namespace); + config.getFeatures().getArgocd().setNamespace(namespace); + config.getFeatures().getMonitoring().setNamespace(namespace); + config.getFeatures().getSecrets().setNamespace(namespace); + config.getFeatures().getIngress().setIngressNamespace(namespace); + config.getFeatures().getCertManager().setNamespace(namespace); + + config.getContent().getNamespaces().clear(); + String contentNamespace = config.getApplication().getNamePrefix() + namespace; + config.getContent().getNamespaces().add(contentNamespace); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/cli/GenerateJsonSchema.java b/src/main/java/com/cloudogu/gitops/cli/GenerateJsonSchema.java new file mode 100644 index 000000000..cb32aaa31 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/cli/GenerateJsonSchema.java @@ -0,0 +1,272 @@ +package com.cloudogu.gitops.cli; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.schema.JsonSchemaGenerator; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import io.micronaut.context.ApplicationContext; +import lombok.extern.slf4j.Slf4j; +import picocli.CommandLine.Option; +import tools.jackson.databind.ObjectMapper; +import tools.jackson.databind.node.ObjectNode; + +import java.io.File; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; +import java.lang.reflect.ParameterizedType; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +@Slf4j +public class GenerateJsonSchema { + + public static final String SCHEMA_FILE = "docs/configuration.schema.json"; + public static final String DOCS_FILE = "docs/Configuration.md"; + + private static final Pattern UPPERCASE_LETTER = Pattern.compile("\\p{Lu}", Pattern.UNICODE_CHARACTER_CLASS); + private static final Pattern WHITESPACE_RUN = Pattern.compile("\\s+", Pattern.UNICODE_CHARACTER_CLASS); + private static final Pattern WHITESPACE_AROUND_NEWLINE = Pattern.compile( + "\\s*\\n\\s*", + Pattern.UNICODE_CHARACTER_CLASS + ); + + static void main(String[] args) { + try { + ObjectNode jsonSchema = ApplicationContext.run().getBean(JsonSchemaGenerator.class).createSchema(); + String prettyJson = new ObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(jsonSchema); + + if (args.length > 0 && "-".equals(args[0])) { + log.info(prettyJson); + } else { + Files.writeString(new File(SCHEMA_FILE).toPath(), prettyJson); + log.info("Wrote schema to {}", SCHEMA_FILE); + + Files.writeString(new File(DOCS_FILE).toPath(), generateDocs()); + log.info("Wrote documentation to {}", DOCS_FILE); + } + } catch (Exception e) { + throw new RuntimeException("Failed to generate schema/documentation files", e); + } + } + + public static String generateDocs() { + Config config = new Config(); + StringBuilder md = new StringBuilder(); + + md.append("# Overview of all CLI and config options\n\n"); + md.append("All options can be set via a [config file](./configuration.schema.json). "); + md.append("Most options are also available as CLI parameters.\n\n"); + + List topFields = schemaFields(Config.class).stream() + .filter(field -> !Set.of("features", "stages") + .contains(field.getName())) + .toList(); + + // Table of contents and top-level sections are built from the same fields in one pass. + StringBuilder toc = new StringBuilder(); + StringBuilder sections = new StringBuilder(); + for (Field field : topFields) { + toc.append("- [") + .append(sectionTitle(field.getName())) + .append("](#") + .append(anchor(field.getName())) + .append(")\n"); + + field.setAccessible(true); + sections.append("## ").append(sectionTitle(field.getName())).append("\n\n"); + try { + sections.append(buildTable(field.get(config), field.getType(), field.getName())); + } catch (IllegalAccessException e) { + throw new IllegalStateException("Failed to read config field via reflection", e); + } + } + + md.append("## Table of Contents\n\n"); + md.append(toc); + md.append("- [Tools](#tools)\n"); + for (Field f : schemaFields(Config.FeaturesSchema.class)) { + md.append(" - [") + .append(sectionTitle(f.getName())) + .append("](#tools-") + .append(anchor(f.getName())) + .append(")\n"); + } + md.append("\n"); + + md.append(sections); + + // Tools sub-sections + md.append("## Tools\n\n"); + md.append("Configuration of optional tools supported by gitops-playground.\n\n"); + for (Field field : schemaFields(Config.FeaturesSchema.class)) { + field.setAccessible(true); + md.append("### Tool: ").append(sectionTitle(field.getName())).append("\n\n"); + try { + md.append(buildTable(field.get(config.getFeatures()), field.getType(), "features." + field.getName())); + } catch (IllegalAccessException e) { + throw new IllegalStateException("Failed to read config field via reflection", e); + } + } + + return md.toString(); + } + + public static String buildTable(Object instance, Class clazz, String prefix) { + List> rows = collectRows(instance, clazz, prefix); + if (rows.isEmpty()) { + return ""; + } + + StringBuilder sb = new StringBuilder(); + sb.append("| CLI | Config key | Type | Default | Description |\n"); + sb.append("| :--- | :--- | :--- | :--- | :--- |\n"); + for (Map r : rows) { + sb.append("| ") + .append(r.get("cli")) + .append(" | `") + .append(r.get("key")) + .append("` | ") + .append(r.get("type")) + .append(" | `") + .append(r.get("default")) + .append("` | ") + .append(r.get("desc")) + .append(" |\n"); + } + sb.append("\n"); + return sb.toString(); + } + + public static List> collectRows(Object instance, Class clazz, String prefix) { + List> rows = new ArrayList<>(); + for (Field field : allFields(clazz)) { + if (isInternalField(field)) { + continue; + } + collectFieldRows(field, instance, prefix, rows); + } + return rows; + } + + private static void collectFieldRows(Field field, Object instance, String prefix, List> rows) { + String key = prefix + "." + field.getName(); + + if (isSchemaType(field.getType()) && !field.getType().isEnum()) { + rows.addAll(collectRows(safeGet(field, instance), field.getType(), key)); + return; + } + + JsonPropertyDescription jsonDesc = field.getAnnotation(JsonPropertyDescription.class); + Option cliOpt = field.getAnnotation(Option.class); + if (jsonDesc == null && cliOpt == null) { + return; + } + + Map r = new HashMap<>(); + if (cliOpt != null) { + r.put("cli", Arrays.stream(cliOpt.names()).map(opt -> "`" + opt + "`").collect(Collectors.joining(", "))); + } else { + r.put("cli", "-"); + } + r.put("key", key); + r.put("type", typeName(field)); + r.put("default", formatDefault(safeGet(field, instance))); + r.put( + "desc", WHITESPACE_AROUND_NEWLINE.matcher(jsonDesc != null ? jsonDesc.value() : "-") + .replaceAll(" ") + .trim() + ); + rows.add(r); + } + + public static List allFields(Class clazz) { + List fields = new ArrayList<>(); + for (Class c = clazz; c != null && c != Object.class; c = c.getSuperclass()) { + fields.addAll(Arrays.asList(c.getDeclaredFields())); + } + return fields; + } + + public static List schemaFields(Class clazz) { + return Arrays.stream(clazz.getDeclaredFields()) + .filter(field -> !isInternalField(field) && isSchemaType(field.getType())) + .toList(); + } + + public static boolean isInternalField(Field field) { + if (field.isSynthetic()) { + return true; + } + if (Modifier.isStatic(field.getModifiers())) { + return true; + } + return field.isAnnotationPresent(JsonIgnore.class); + } + + public static boolean isSchemaType(Class type) { + return type.getName().startsWith("com.cloudogu.gitops"); + } + + public static Object safeGet(Field field, Object instance) { + try { + field.setAccessible(true); + return field.get(instance); + } catch (Exception e) { + log.debug("Failed to read field {} for documentation generation", field.getName(), e); + return null; + } + } + + public static String formatDefault(Object value) { + return switch (value) { + case null -> "-"; + case Map map -> map.isEmpty() ? "{}" : value.toString(); + case Collection collection -> collection.isEmpty() ? "[]" : value.toString(); + default -> value.toString(); + }; + } + + public static String typeName(Field field) { + Class t = field.getType(); + if (t == Boolean.class || t == boolean.class) { + return "Boolean"; + } + if (t == Integer.class || t == int.class) { + return "Integer"; + } + if (t == String.class) { + return "String"; + } + if (Map.class.isAssignableFrom(t)) { + return "Map"; + } + if (t.isEnum()) { + return t.getSimpleName(); + } + if (field.getGenericType() instanceof ParameterizedType pt) { + String args = Arrays.stream(pt.getActualTypeArguments()) + .map(typeArgument -> typeArgument instanceof Class type ? type.getSimpleName() : typeArgument.toString()) + .collect(Collectors.joining(", ")); + return ((Class) pt.getRawType()).getSimpleName() + "<" + args + ">"; + } + return t.getSimpleName(); + } + + public static String sectionTitle(String name) { + String title = UPPERCASE_LETTER.matcher(name).replaceAll(" $0").trim(); + return Character.toUpperCase(title.charAt(0)) + title.substring(1); + } + + public static String anchor(String name) { + return WHITESPACE_RUN.matcher(sectionTitle(name).toLowerCase(Locale.ROOT)).replaceAll("-"); + } +} diff --git a/src/main/java/com/cloudogu/gitops/cli/GitopsPlaygroundCli.java b/src/main/java/com/cloudogu/gitops/cli/GitopsPlaygroundCli.java new file mode 100644 index 000000000..4273b0bef --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/cli/GitopsPlaygroundCli.java @@ -0,0 +1,361 @@ +package com.cloudogu.gitops.cli; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.LoggerContext; +import ch.qos.logback.classic.encoder.PatternLayoutEncoder; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.Appender; +import ch.qos.logback.core.ConsoleAppender; +import ch.qos.logback.core.encoder.Encoder; +import com.cloudogu.gitops.application.Application; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.schema.JsonSchemaValidator; +import com.cloudogu.gitops.destroy.Destroyer; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractTool; +import com.cloudogu.gitops.tools.common.CommonToolConfig; +import com.cloudogu.gitops.tools.common.ConfigLifecycleHook; +import com.cloudogu.gitops.utils.YamlUtils; +import io.micronaut.context.ApplicationContext; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.slf4j.LoggerFactory; +import picocli.CommandLine; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.function.BiConsumer; +import java.util.regex.Pattern; + +import static com.cloudogu.gitops.config.ConfigConstants.APP_NAME; +import static com.cloudogu.gitops.utils.MapUtils.deepMerge; +import static com.cloudogu.gitops.utils.MapUtils.deepMergeDefaults; + +@RequiredArgsConstructor +@Slf4j +public class GitopsPlaygroundCli { + + private static final String STDOUT_APPENDER_NAME = "STDOUT"; + // Not exploitable: only ever matched against the trusted, developer-controlled pattern string + // from logback.xml, never against user input. + private static final Pattern THREAD_PATTERN_TOKEN = Pattern.compile( + " \\S*%thread\\S* ", + Pattern.UNICODE_CHARACTER_CLASS + ); + private static final Pattern LOGGER_PATTERN_TOKEN = Pattern.compile( + " \\S*%logger\\S* ", + Pattern.UNICODE_CHARACTER_CLASS + ); + + private final K8sClient k8sClient; + private final ApplicationConfigurator applicationConfigurator; + + public GitopsPlaygroundCli() { + this(new K8sClient(), new ApplicationConfigurator()); + } + + public ReturnCode run(String[] args) { + setLogging(args); + + log.debug("Reading initial CLI params"); + Config cliParams = new Config(); + new CommandLine(cliParams).parseArgs(args); + + if (cliParams.getApplication().getUsageHelpRequested()) { + new CommandLine(cliParams).execute(args); + return ReturnCode.SUCCESS; + } + + String version = createVersionOutput(); + if (cliParams.getApplication().getVersionInfoRequested()) { + log.info(version); + return ReturnCode.SUCCESS; + } + + ApplicationContext context = createApplicationContext(); + Application app = context.getBean(Application.class); + + Config config = readConfigs(args); + runHook(app, "preConfigInit", ConfigLifecycleHook::preConfigInit, config); + + if (config.getApplication().getOutputConfigFile()) { + log.info(config.toYaml(false)); + return ReturnCode.SUCCESS; + } + + config = applicationConfigurator.initConfig(config); + log.debug("Actual config: {}", config.toYaml(true)); + runHook(app, "postConfigInit", ConfigLifecycleHook::postConfigInit, config); + + context.close(); + context = createApplicationContext(); + register(config, context); + + if (config.getApplication().getDestroy()) { + log.info(version); + if (!confirm( + "Destroying gitops playground in kubernetes cluster '" + k8sClient.getCurrentContext() + "'.", + config + )) { + return ReturnCode.NOT_CONFIRMED; + } + + Destroyer destroyer = context.getBean(Destroyer.class); + destroyer.destroy(); + } else { + log.info(version); + if (!confirm( + "Applying gitops playground to kubernetes cluster '" + k8sClient.getCurrentContext() + "'.", + config + )) { + return ReturnCode.NOT_CONFIRMED; + } + app = context.getBean(Application.class); + app.start(); + + printWelcomeScreen(config.getApplication().getPassword()); + } + + return ReturnCode.SUCCESS; + } + + protected String createVersionOutput() { + String versionName = Version.NAME.replace("\\n", "\n"); + + if (versionName.trim().startsWith("(")) { + versionName = versionName.trim().replace("(", "").replace(")", ""); + } + return APP_NAME + " " + versionName; + } + + protected void register(Config config, ApplicationContext context) { + context.registerSingleton(config); + } + + private static boolean confirm(String message, Config config) { + log.debug( + "Calling confirm for message: {} | yes = {} | System.in class: {}", + message, + config.getApplication() + .getYes(), + System.in.getClass() + .getName() + ); + if (config.getApplication().getYes()) { + return true; + } + + log.info("\n{}\nContinue? y/n [n]", message); + + try { + BufferedReader reader = new BufferedReader(new InputStreamReader(System.in, StandardCharsets.UTF_8)); + String input = reader.readLine(); + return "y".equals(input); + } catch (IOException e) { + throw new UncheckedIOException("Failed to read user input", e); + } + } + + protected ApplicationContext createApplicationContext() { + return ApplicationContext.run(); + } + + private void setLogging(String[] args) { + List argList = Arrays.asList(args); + if (argList.contains("--trace") || argList.contains("-x")) { + log.info("Setting loglevel to trace"); + setGitopsLogLevel(Level.TRACE); + System.setProperty("picocli.trace", "DEBUG"); + } else if (argList.contains("--debug") || argList.contains("-d")) { + System.setProperty("picocli.trace", "INFO"); + setGitopsLogLevel(Level.DEBUG); + log.info("Setting loglevel to debug"); + } else { + setSimpleLogPattern(); + } + } + + private static void setGitopsLogLevel(Level level) { + ((Logger) LoggerFactory.getLogger("com.cloudogu.gitops")).setLevel(level); + } + + public void setSimpleLogPattern() { + LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory(); + Appender stdoutAppender = rootLogger(loggerContext).getAppender(STDOUT_APPENDER_NAME); + if (!(stdoutAppender instanceof ConsoleAppender)) { + return; + } + Encoder encoderObj = ((ConsoleAppender) stdoutAppender).getEncoder(); + if (!(encoderObj instanceof PatternLayoutEncoder)) { + return; + } + + String defaultPattern = ((PatternLayoutEncoder) encoderObj).getPattern(); + + rootLogger(loggerContext).detachAppender(STDOUT_APPENDER_NAME); + PatternLayoutEncoder encoder = new PatternLayoutEncoder(); + encoder.setPattern(LOGGER_PATTERN_TOKEN.matcher(THREAD_PATTERN_TOKEN.matcher(defaultPattern).replaceAll(" ")) + .replaceAll(" ")); + encoder.setContext(loggerContext); + encoder.start(); + ConsoleAppender appender = new ConsoleAppender<>(); + appender.setName(STDOUT_APPENDER_NAME); + appender.setContext(loggerContext); + appender.setEncoder(encoder); + appender.start(); + rootLogger(loggerContext).addAppender(appender); + } + + private static Logger rootLogger(LoggerContext loggerContext) { + return loggerContext.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME); + } + + private Config readConfigs(String[] args) { + Config cliParams = new Config(); + new CommandLine(cliParams).parseArgs(args); + + List> configFile = new ArrayList<>(); + + if (cliParams.getApplication().getConfigFiles() != null) { + for (String configFileItem : cliParams.getApplication().getConfigFiles()) { + log.debug("Reading config file {}", configFileItem); + try { + configFile.add(validateConfig(Files.readString(Path.of(configFileItem)))); + } catch (IOException e) { + throw new UncheckedIOException("Failed to read config file: " + configFileItem, e); + } + } + } + + List> configMap = new ArrayList<>(); + if (cliParams.getApplication().getConfigMaps() != null) { + for (String configMapItem : cliParams.getApplication().getConfigMaps()) { + log.debug("Reading config map {}", configMapItem); + String configValues = k8sClient.getConfigMap(configMapItem, "config.yaml"); + configMap.add(validateConfig(configValues)); + } + } + + Config profileConfig = extractProfile(cliParams); + Map mergedConfigs = new HashMap<>(); + deepMerge(profileConfig.toMap(), mergedConfigs); + for (Map map : configMap) { + deepMerge(map, mergedConfigs); + } + for (Map map : configFile) { + deepMerge(map, mergedConfigs); + } + + mergedConfigs = deepMergeDefaults(mergedConfigs, new Config().toMap()); + + log.debug("Writing CLI params into config"); + log.debug( + "mergedConfigs keys: {} | application map: {}", + mergedConfigs.keySet(), + mergedConfigs.get("application") + ); + Config mergedConfig = Config.fromMap(mergedConfigs); + log.debug( + "mergedConfig yes before parseArgs: {}", + mergedConfig.getApplication() != null ? mergedConfig.getApplication() + .getYes() : "null" + ); + new CommandLine(mergedConfig).parseArgs(args); + log.debug( + "mergedConfig yes after parseArgs: {}", + mergedConfig.getApplication() != null ? mergedConfig.getApplication() + .getYes() : "null" + ); + + return mergedConfig; + } + + public static Map validateConfig(String configValues) { + Map configMap = YamlUtils.parseYamlMap(configValues); + JsonSchemaValidator.validate(configMap); + return configMap; + } + + public void printWelcomeScreen(String password) { + log.info(""" + + |----------------------------------------------------------------------------------------------| + | Welcome to the GitOps playground by Cloudogu! + |----------------------------------------------------------------------------------------------| + | + | Please find the URLs of the individual applications in our README: + | https://github.com/cloudogu/gitops-playground/blob/main/README.md#table-of-contents + | + | A good starting point might also be the services or ingresses inside your cluster: \s + | kubectl get svc -A + | Or (depending on your config) + | kubectl get ing -A + | + | Please be aware, Jenkins and Argo CD may take some time to build and deploy all apps. + |\s + | Your initial password for all apps (if not set manually): %s + |\s + |----------------------------------------------------------------------------------------------| + """.formatted(password)); + } + + public static void runHook( + Application app, + String hookName, + BiConsumer hook, + Config config) { + List configLifecycleHooks = new ArrayList<>(); + configLifecycleHooks.add(new CommonToolConfig()); + for (AbstractTool tool : app.getTools()) { + if (tool instanceof ConfigLifecycleHook configLifecycleHook) { + configLifecycleHooks.add(configLifecycleHook); + } + } + + for (ConfigLifecycleHook configLifecycleHook : configLifecycleHooks) { + try { + log.debug("Executing {} hook on feature {}", hookName, configLifecycleHook.getClass().getName()); + hook.accept(configLifecycleHook, config); + } catch (Exception e) { + throw new RuntimeException( + "Failed to execute hook " + hookName + " on " + configLifecycleHook.getClass() + .getName(), e + ); + } + } + } + + private static Config extractProfile(Config newConfig) { + String profile = newConfig.getApplication().getProfile(); + + Config profileConfig = new Config(); + if (profile != null && !profile.isEmpty()) { + String resourceName = "application-" + profile + ".yaml"; + log.debug("Loading profile '{}' from classpath", resourceName); + + try (InputStream inputStream = GitopsPlaygroundCli.class.getResourceAsStream("/" + resourceName)) { + if (inputStream == null) { + throw new IllegalArgumentException("Profile '" + profile + "' does not exist (resource '" + resourceName + "' not found)."); + } + String content = new String(inputStream.readAllBytes(), StandardCharsets.UTF_8); + Map profileFile = validateConfig(content); + profileConfig = Config.fromMap(profileFile); + } catch (IOException e) { + throw new UncheckedIOException("Failed to read profile " + profile, e); + } + } + return profileConfig; + } +} diff --git a/src/main/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliMain.java b/src/main/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliMain.java new file mode 100644 index 000000000..157fde687 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliMain.java @@ -0,0 +1,28 @@ +package com.cloudogu.gitops.cli; + +import lombok.extern.slf4j.Slf4j; + +@Slf4j +public class GitopsPlaygroundCliMain { + + public static void main(String[] args) { + System.exit(new GitopsPlaygroundCliMain().exec(args, GitopsPlaygroundCli.class).ordinal()); + } + + public ReturnCode exec(String[] args, Class commandClass) { + try { + GitopsPlaygroundCli app = commandClass.getDeclaredConstructor().newInstance(); + return app.run(args); + } catch (RuntimeException e) { + if (log.isDebugEnabled()) { + log.error("", e); + } else { + log.error(e.getMessage()); + } + return ReturnCode.GENERIC_ERROR; + } catch (Exception e) { + log.error("Fatal error starting CLI", e); + return ReturnCode.GENERIC_ERROR; + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/cli/ReturnCode.java b/src/main/java/com/cloudogu/gitops/cli/ReturnCode.java new file mode 100644 index 000000000..dba16bce1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/cli/ReturnCode.java @@ -0,0 +1,7 @@ +package com.cloudogu.gitops.cli; + +public enum ReturnCode { + SUCCESS, + NOT_CONFIRMED, + GENERIC_ERROR +} diff --git a/src/main/java/com/cloudogu/gitops/cli/Version.java b/src/main/java/com/cloudogu/gitops/cli/Version.java new file mode 100644 index 000000000..2f9324dbd --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/cli/Version.java @@ -0,0 +1,27 @@ +package com.cloudogu.gitops.cli; + +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; + +public final class Version { + + private static final String VERSION_RESOURCE = "/com/cloudogu/gitops/cli/version-name.txt"; + + public static final String NAME = loadName(); + + private Version() { + } + + private static String loadName() { + try (InputStream input = Version.class.getResourceAsStream(VERSION_RESOURCE)) { + if (input == null) { + throw new IllegalStateException("Version resource not found: " + VERSION_RESOURCE); + } + return new String(input.readAllBytes(), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new UncheckedIOException("Failed to read version resource: " + VERSION_RESOURCE, e); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/Config.java b/src/main/java/com/cloudogu/gitops/config/Config.java new file mode 100644 index 000000000..c38454e6f --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/Config.java @@ -0,0 +1,1106 @@ +package com.cloudogu.gitops.config; + +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import com.fasterxml.jackson.annotation.JsonValue; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.BeanDescription; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationConfig; +import com.fasterxml.jackson.databind.module.SimpleModule; +import com.fasterxml.jackson.databind.ser.BeanPropertyWriter; +import com.fasterxml.jackson.databind.ser.BeanSerializerModifier; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import picocli.CommandLine.Command; +import picocli.CommandLine.ITypeConverter; +import picocli.CommandLine.Mixin; +import picocli.CommandLine.Option; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.security.SecureRandom; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +import static com.cloudogu.gitops.config.ConfigConstants.APPLICATION_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.APPLICATION_GOP_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.APPLICATION_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.APPLICATION_PROFIL; +import static com.cloudogu.gitops.config.ConfigConstants.APP_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_EMAIL_FROM_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_EMAIL_TO_USER_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_ENV_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_OPERATOR_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_RESOURCE_INCLUSIONS_CLUSTER; +import static com.cloudogu.gitops.config.ConfigConstants.ARGOCD_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.BASE_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.BINARY_NAME; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CLUSTER_ADMIN_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONFIG_FILE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONFIG_MAP_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_CHART_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_NAMESPACE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_NAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_RELEASE_NAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_REPO_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_VALUES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_VALUES_FILE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_HELM_RELEASE_VERSION_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_NAMESPACES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_CREATE_JENKINS_JOB_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_CREDENTIALS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_PATH_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_REF_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_TARGET_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_TARGET_OVERWRITE_MODE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_TARGET_REF_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_TEMPLATING_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_TYPE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_REPO_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_STATICSWHITELIST_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.CONTENT_VARIABLES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.DEBUG_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.DESTROY_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.ESO_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.EXTERNAL_SECRETS_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.FEATURES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GIT_EMAIL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GIT_NAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GRAFANA_EMAIL_FROM_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GRAFANA_EMAIL_TO_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GRAFANA_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GRAFANA_SIDECAR_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.GRAFANA_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_CHART_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_REPO_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_VALUES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_VERSION_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.INGRESS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.INGRESS_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.INGRESS_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.INSECURE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_ADDITIONAL_ENVS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_METRICS_PASSWORD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_METRICS_USERNAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_PASSWORD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_SKIP_PLUGINS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_SKIP_RESTART_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.JENKINS_USERNAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.MAIL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.MAVEN_CENTRAL_MIRROR_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.MIRROR_REPOS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.MONITORING_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.MONITORING_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.MONITORING_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.MULTITENANT_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.NAMESPACE_ISOLATION_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.NAME_PREFIX_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.NETPOLS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.OIDC_DESCPRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.OPENSHIFT_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.OUTPUT_CONFIG_FILE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.PASSWORD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.PIPE_YES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.POD_RESOURCES_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.PROMETHEUS_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_INTERNAL_PORT_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PASSWORD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PASSWORD_RO_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PATH_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PROXY_PASSWORD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PROXY_PATH_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PROXY_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_PROXY_USERNAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_URL_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_USERNAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.REGISTRY_USERNAME_RO_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SCM_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SECRETS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SECRETS_NAMESPACE; +import static com.cloudogu.gitops.config.ConfigConstants.SKIP_CRDS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SMTP_ADDRESS_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SMTP_PASSWORD_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SMTP_PORT_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.SMTP_USER_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.TRACE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.URL_SEPARATOR_HYPHEN_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.USERNAME_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.VAULT_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.VAULT_ENABLE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.VAULT_IMAGE_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.VAULT_URL_DESCRIPTION; +import static picocli.CommandLine.ScopeType; + +@Singleton +@Command(name = BINARY_NAME, description = APP_DESCRIPTION) +@Getter +@Setter +public class Config { + + // When updating please also adapt in Dockerfile, vars.tf and init-cluster.sh + public static final String K8S_VERSION = "1.36.4"; + public static final String DEFAULT_ADMIN_USER = "admin"; + + // Generated once when Config is initialized and intentionally shared by all Config instances in the JVM. + public static final String DEFAULT_ADMIN_PW = generatePassword(); + + public static final int DEFAULT_REGISTRY_PORT = 30000; + private static final int GENERATED_PASSWORD_LENGTH = 12; + + private static final ObjectMapper objectMapper = new ObjectMapper(); + + @JsonPropertyDescription(REGISTRY_DESCRIPTION) + @Mixin + private RegistrySchema registry = new RegistrySchema(); + + @JsonPropertyDescription(JENKINS_DESCRIPTION) + @Mixin + private JenkinsSchema jenkins = new JenkinsSchema(); + + @JsonPropertyDescription(MULTITENANT_DESCRIPTION) + @Mixin + private MultiTenantSchema multiTenant = new MultiTenantSchema(); + + @JsonPropertyDescription(SCM_DESCRIPTION) + @Mixin + private ScmTenantSchema scm = new ScmTenantSchema(); + + @JsonPropertyDescription(APPLICATION_DESCRIPTION) + @Mixin + private ApplicationSchema application = new ApplicationSchema(); + + @JsonPropertyDescription(FEATURES_DESCRIPTION) + @Mixin + private FeaturesSchema features = new FeaturesSchema(); + + @JsonPropertyDescription(CONTENT_DESCRIPTION) + @Mixin + private ContentSchema content = new ContentSchema(); + + private static String generatePassword() { + final SecureRandom sr = new SecureRandom(); + String chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@$%&"; + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < GENERATED_PASSWORD_LENGTH; i++) { + sb.append(chars.charAt(sr.nextInt(chars.length()))); + } + return sb.toString(); + } + + @Getter + @Setter + public static class ContentSchema { + @JsonPropertyDescription(CONTENT_NAMESPACES_DESCRIPTION) + private List namespaces = new ArrayList<>(); + + @JsonPropertyDescription(CONTENT_REPO_DESCRIPTION) + private List repos = new ArrayList<>(); + + @JsonPropertyDescription(CONTENT_VARIABLES_DESCRIPTION) + private Map variables = new HashMap<>(); + + @JsonPropertyDescription(CONTENT_HELM_RELEASES_DESCRIPTION) + private List helmReleases = new ArrayList<>(); + + @Option(names = {"--content-whitelist"}, description = CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION) + @JsonPropertyDescription(CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION) + private Boolean useWhitelist = false; + + @JsonPropertyDescription(CONTENT_STATICSWHITELIST_DESCRIPTION) + private Set allowedStaticsWhitelist = new HashSet<>(Arrays.asList( + "java.lang.String", + "java.lang.Integer", + "java.lang.Long", + "java.lang.Double", + "java.lang.Float", + "java.lang.Boolean", + "java.lang.Math", + "com.cloudogu.gitops.utils.DockerImageParser" + )); + + @Getter + @Setter + @NoArgsConstructor + public static class ContentRepositorySchema { + public static final String DEFAULT_PATH = "."; + public static final ContentRepoType DEFAULT_TYPE = ContentRepoType.MIRROR; + + @JsonPropertyDescription(CONTENT_REPO_URL_DESCRIPTION) + private String url = ""; + + @JsonPropertyDescription(CONTENT_REPO_PATH_DESCRIPTION) + private String path = DEFAULT_PATH; + + @JsonPropertyDescription(CONTENT_REPO_REF_DESCRIPTION) + private String ref = ""; + + @JsonPropertyDescription(CONTENT_REPO_TARGET_REF_DESCRIPTION) + private String targetRef = ""; + + @JsonPropertyDescription(CONTENT_REPO_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @JsonPropertyDescription(CONTENT_REPO_TEMPLATING_DESCRIPTION) + private Boolean templating = false; + + @JsonPropertyDescription(CONTENT_REPO_TYPE_DESCRIPTION) + private ContentRepoType type = DEFAULT_TYPE; + + @JsonPropertyDescription(CONTENT_REPO_TARGET_DESCRIPTION) + private String target = ""; + + @JsonPropertyDescription(CONTENT_REPO_TARGET_OVERWRITE_MODE_DESCRIPTION) + private OverwriteMode overwriteMode = OverwriteMode.INIT; + + @JsonPropertyDescription(CONTENT_REPO_CREATE_JENKINS_JOB_DESCRIPTION) + private Boolean createJenkinsJob = false; + } + + @Getter + @Setter + public static class HelmReleaseSchema { + @JsonPropertyDescription(CONTENT_HELM_RELEASE_NAME_DESCRIPTION) + private String name = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_REPO_URL_DESCRIPTION) + private String repoURL = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_CHART_DESCRIPTION) + private String chart = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_VERSION_DESCRIPTION) + private String version = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_NAMESPACE_DESCRIPTION) + private String namespace = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_RELEASE_NAME_DESCRIPTION) + private String releaseName = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_VALUES_FILE_DESCRIPTION) + private String valuesPath = ""; + + @JsonPropertyDescription(CONTENT_HELM_RELEASE_VALUES_DESCRIPTION) + private Map values = new HashMap<>(); + } + } + + @Getter + @Setter + @NoArgsConstructor + public static class HelmConfig { + @JsonPropertyDescription(HELM_CONFIG_CHART_DESCRIPTION) + private String chart; + + @JsonPropertyDescription(HELM_CONFIG_REPO_URL_DESCRIPTION) + private String repoURL; + + @JsonPropertyDescription(HELM_CONFIG_VERSION_DESCRIPTION) + private String version; + } + + @Getter + @Setter + public static class HelmConfigWithValues extends HelmConfig { + @JsonPropertyDescription(HELM_CONFIG_VALUES_DESCRIPTION) + private Map values = new HashMap<>(); + } + + @Getter + @Setter + public static class RegistrySchema { + private Boolean internal = true; + private Boolean twoRegistries = false; + + @Option(names = {"--registry"}, description = REGISTRY_ENABLE_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_ENABLE_DESCRIPTION) + private Boolean active = false; + + @Option(names = {"--internal-registry-port"}, description = REGISTRY_INTERNAL_PORT_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_INTERNAL_PORT_DESCRIPTION) + private Integer internalPort = DEFAULT_REGISTRY_PORT; + + @Option(names = {"--registry-url"}, description = REGISTRY_URL_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_URL_DESCRIPTION) + private String url = ""; + + @Option(names = {"--registry-path"}, description = REGISTRY_PATH_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_PATH_DESCRIPTION) + private String path = ""; + + @Option(names = {"--registry-username"}, description = REGISTRY_USERNAME_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_USERNAME_DESCRIPTION) + private String username = ""; + + @Option(names = {"--registry-password"}, description = REGISTRY_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_PASSWORD_DESCRIPTION) + private String password = ""; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @Option(names = {"--registry-proxy-url"}, description = REGISTRY_PROXY_URL_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_PROXY_URL_DESCRIPTION) + private String proxyUrl = ""; + + @Option(names = {"--registry-proxy-path"}, description = REGISTRY_PROXY_PATH_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_PROXY_PATH_DESCRIPTION) + private String proxyPath = ""; + + @Option(names = {"--registry-proxy-username"}, description = REGISTRY_PROXY_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_PROXY_USERNAME_DESCRIPTION) + private String proxyUsername = ""; + + @Option(names = {"--registry-proxy-password"}, description = "Optional when --registry-proxy-url is set") + @JsonPropertyDescription(REGISTRY_PROXY_PASSWORD_DESCRIPTION) + private String proxyPassword = ""; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials proxyCredentials; + + @Option(names = {"--registry-username-read-only"}, description = REGISTRY_USERNAME_RO_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_USERNAME_RO_DESCRIPTION) + private String readOnlyUsername = ""; + + @Option(names = {"--registry-password-read-only"}, description = REGISTRY_PASSWORD_RO_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_PASSWORD_RO_DESCRIPTION) + private String readOnlyPassword = ""; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials readOnlyCredentials; + + @Option(names = {"--create-image-pull-secrets"}, description = REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION) + @JsonPropertyDescription(REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION) + private Boolean createImagePullSecrets = false; + + @Option(names = {"--registry-namespace"}, description = REGISTRY_NAMESPACE) + @JsonPropertyDescription(REGISTRY_NAMESPACE) + private String namespace = "registry"; + + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private HelmConfigWithValues helm; + + public RegistrySchema() { + helm = new HelmConfigWithValues(); + helm.setChart("docker-registry"); + helm.setRepoURL("https://twuni.github.io/docker-registry.helm"); + // renovate: depName=docker-registry registryUrl=https://twuni.github.io/docker-registry.helm + helm.setVersion("3.0.0"); + } + } + + @Getter + @Setter + public static class JenkinsSchema { + private Boolean internal = true; + private String urlForScm = ""; + private String ingress = ""; + private String internalBashImage = "bash:5"; + private String internalDockerClientVersion = "27.1.2"; + + @Option(names = {"--jenkins"}, description = JENKINS_ENABLE_DESCRIPTION) + @JsonPropertyDescription(JENKINS_ENABLE_DESCRIPTION) + private Boolean active = false; + + @Option(names = {"--jenkins-skip-restart"}, description = JENKINS_SKIP_RESTART_DESCRIPTION) + @JsonPropertyDescription(JENKINS_SKIP_RESTART_DESCRIPTION) + private Boolean skipRestart = false; + + @Option(names = {"--jenkins-skip-plugins"}, description = JENKINS_SKIP_PLUGINS_DESCRIPTION) + @JsonPropertyDescription(JENKINS_SKIP_PLUGINS_DESCRIPTION) + private Boolean skipPlugins = false; + + @Option(names = {"--jenkins-url"}, description = JENKINS_URL_DESCRIPTION) + @JsonPropertyDescription(JENKINS_URL_DESCRIPTION) + private String url = ""; + + @Option(names = {"--jenkins-username"}, description = JENKINS_USERNAME_DESCRIPTION) + @JsonPropertyDescription(JENKINS_USERNAME_DESCRIPTION) + private String username = DEFAULT_ADMIN_USER; + + @Option(names = {"--jenkins-password"}, description = JENKINS_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(JENKINS_PASSWORD_DESCRIPTION) + private String password = DEFAULT_ADMIN_PW; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @Option(names = {"--jenkins-metrics-username"}, description = JENKINS_METRICS_USERNAME_DESCRIPTION) + @JsonPropertyDescription(JENKINS_METRICS_USERNAME_DESCRIPTION) + private String metricsUsername = "metrics"; + + @Option(names = {"--jenkins-metrics-password"}, description = JENKINS_METRICS_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(JENKINS_METRICS_PASSWORD_DESCRIPTION) + private String metricsPassword = "metrics"; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials metricsCredentials; + + @Option(names = {"--jenkins-image"}, description = JENKINS_IMAGE_DESCRIPTION) + @JsonPropertyDescription(JENKINS_IMAGE_DESCRIPTION) + private String jenkinsImage = ""; + + @Option(names = {"--maven-central-mirror"}, description = MAVEN_CENTRAL_MIRROR_DESCRIPTION) + @JsonPropertyDescription(MAVEN_CENTRAL_MIRROR_DESCRIPTION) + private String mavenCentralMirror = ""; + + @JsonPropertyDescription(OIDC_DESCPRIPTION) + private OidcSchema oidc = new OidcSchema("jenkins"); + + @Option(names = {"--jenkins-additional-envs"}, description = JENKINS_ADDITIONAL_ENVS_DESCRIPTION, split = ",", required = false) + @JsonPropertyDescription(JENKINS_ADDITIONAL_ENVS_DESCRIPTION) + private Map additionalEnvs = new HashMap<>(); + + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private HelmConfigWithValues helm; + + @Option(names = {"--jenkins-namespace"}, description = JENKINS_NAMESPACE) + @JsonPropertyDescription(JENKINS_NAMESPACE) + private String namespace = "jenkins"; + + public JenkinsSchema() { + helm = new HelmConfigWithValues(); + helm.setChart("jenkins"); + helm.setRepoURL("https://charts.jenkins.io"); + // renovate: depName=jenkins registryUrl=https://charts.jenkins.io + helm.setVersion("5.9.56"); + } + } + + @Getter + @Setter + @NoArgsConstructor + public static class ApplicationSchema { + private static final Pattern TRAILING_DASH = Pattern.compile("-$"); + + private Boolean runningInsideK8s = false; + private String namePrefixForEnvVars = ""; + private String internalKubernetesApiUrl = ""; + private String localHelmChartFolder = System.getenv("LOCAL_HELM_CHART_FOLDER"); + + private NamespaceSchema namespaces = new NamespaceSchema(); + + @Option(names = {"--config-file"}, description = CONFIG_FILE_DESCRIPTION, split = ",") + private List configFiles = new ArrayList<>(); + + @Option(names = {"--config-map"}, description = CONFIG_MAP_DESCRIPTION, split = ",") + private List configMaps = new ArrayList<>(); + + @Option(names = {"-d", "--debug"}, description = DEBUG_DESCRIPTION, scope = ScopeType.INHERIT) + private Boolean debug = false; + + @Option(names = {"-x", "--trace"}, description = TRACE_DESCRIPTION, scope = ScopeType.INHERIT) + private Boolean trace = false; + + @Option(names = {"--output-config-file"}, description = OUTPUT_CONFIG_FILE_DESCRIPTION, help = true) + private Boolean outputConfigFile = false; + + @Option(names = {"-v", "--version"}, help = true, description = "Display version and license info") + private Boolean versionInfoRequested = false; + + @Option(names = {"-h", "--help"}, usageHelp = true, description = "Display this help message") + private Boolean usageHelpRequested = false; + + @Option(names = {"--insecure"}, description = INSECURE_DESCRIPTION) + @JsonPropertyDescription(INSECURE_DESCRIPTION) + private Boolean insecure = false; + + @Option(names = {"--openshift"}, description = OPENSHIFT_DESCRIPTION) + @JsonPropertyDescription(OPENSHIFT_DESCRIPTION) + private Boolean openshift = false; + + @Option(names = {"--username"}, description = USERNAME_DESCRIPTION) + @JsonPropertyDescription(USERNAME_DESCRIPTION) + private String username = DEFAULT_ADMIN_USER; + + @Option(names = {"--password"}, description = PASSWORD_DESCRIPTION) + @JsonPropertyDescription(PASSWORD_DESCRIPTION) + private String password = DEFAULT_ADMIN_PW; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @Option(names = {"-y", "--yes"}, description = PIPE_YES_DESCRIPTION) + @JsonPropertyDescription(PIPE_YES_DESCRIPTION) + private Boolean yes = false; + + @Option(names = {"--name-prefix"}, description = NAME_PREFIX_DESCRIPTION) + @JsonPropertyDescription(NAME_PREFIX_DESCRIPTION) + private String namePrefix = ""; + + @Option(names = {"--destroy"}, description = DESTROY_DESCRIPTION) + @JsonPropertyDescription(DESTROY_DESCRIPTION) + private Boolean destroy = false; + + @Option(names = {"--pod-resources"}, description = POD_RESOURCES_DESCRIPTION) + @JsonPropertyDescription(POD_RESOURCES_DESCRIPTION) + private Boolean podResources = false; + + @Option(names = {"--git-name"}, description = GIT_NAME_DESCRIPTION) + @JsonPropertyDescription(GIT_NAME_DESCRIPTION) + private String gitName = "Cloudogu"; + + @Option(names = {"--git-email"}, description = GIT_EMAIL_DESCRIPTION) + @JsonPropertyDescription(GIT_EMAIL_DESCRIPTION) + private String gitEmail = "hello@cloudogu.com"; + + @Option(names = {"--base-url"}, description = BASE_URL_DESCRIPTION) + @JsonPropertyDescription(BASE_URL_DESCRIPTION) + private String baseUrl = ""; + + @Option(names = {"--url-separator-hyphen"}, description = URL_SEPARATOR_HYPHEN_DESCRIPTION) + @JsonPropertyDescription(URL_SEPARATOR_HYPHEN_DESCRIPTION) + private Boolean urlSeparatorHyphen = false; + + @Option(names = {"--mirror-repos"}, description = MIRROR_REPOS_DESCRIPTION) + @JsonPropertyDescription(MIRROR_REPOS_DESCRIPTION) + private Boolean mirrorRepos = false; + + @Option(names = {"--skip-crds"}, description = SKIP_CRDS_DESCRIPTION) + @JsonPropertyDescription(SKIP_CRDS_DESCRIPTION) + private Boolean skipCrds = false; + + @Option(names = {"--namespace-isolation"}, description = NAMESPACE_ISOLATION_DESCRIPTION) + @JsonPropertyDescription(NAMESPACE_ISOLATION_DESCRIPTION) + private Boolean namespaceIsolation = false; + + @Option(names = {"--netpols"}, description = NETPOLS_DESCRIPTION) + @JsonPropertyDescription(NETPOLS_DESCRIPTION) + private Boolean netpols = false; + + @Option(names = {"--cluster-admin"}, description = CLUSTER_ADMIN_DESCRIPTION) + @JsonPropertyDescription(CLUSTER_ADMIN_DESCRIPTION) + private Boolean clusterAdmin = false; + + @Option(names = {"-p", "--profile"}, description = APPLICATION_PROFIL) + @JsonPropertyDescription(APPLICATION_PROFIL) + private String profile; + + @Option(names = {"--gop-namespace"}, description = APPLICATION_GOP_NAMESPACE) + @JsonPropertyDescription(APPLICATION_GOP_NAMESPACE) + private String gopNamespace = ""; + + @Option(names = {"-n", "--namespace"}, description = APPLICATION_NAMESPACE) + @JsonPropertyDescription(APPLICATION_NAMESPACE) + private String namespace = ""; + + @Getter + @Setter + public static class NamespaceSchema { + private LinkedHashSet dedicatedNamespaces = new LinkedHashSet<>(); + private LinkedHashSet tenantNamespaces = new LinkedHashSet<>(); + + public Set getActiveNamespaces() { + LinkedHashSet active = new LinkedHashSet<>(dedicatedNamespaces); + active.addAll(tenantNamespaces); + return active; + } + } + + @JsonIgnore + public String getTenantName() { + return namePrefix != null ? TRAILING_DASH.matcher(namePrefix).replaceAll("") : ""; + } + } + + @Getter + @Setter + public static class FeaturesSchema { + @Mixin + @JsonPropertyDescription(ARGOCD_DESCRIPTION) + private ArgoCDSchema argocd = new ArgoCDSchema(); + + @Mixin + @JsonPropertyDescription(MAIL_DESCRIPTION) + private MailSchema mail = new MailSchema(); + + @Mixin + @JsonPropertyDescription(MONITORING_DESCRIPTION) + private MonitoringSchema monitoring = new MonitoringSchema(); + + @Mixin + @JsonPropertyDescription(SECRETS_DESCRIPTION) + private SecretsSchema secrets = new SecretsSchema(); + + @Mixin + @JsonPropertyDescription(INGRESS_DESCRIPTION) + private IngressSchema ingress = new IngressSchema(); + + @Mixin + @JsonPropertyDescription(CERTMANAGER_DESCRIPTION) + private CertManagerSchema certManager = new CertManagerSchema(); + } + + @Getter + @Setter + @NoArgsConstructor + public static class ArgoCDSchema { + private Boolean configOnly = false; + + @Option(names = {"--argocd"}, description = ARGOCD_ENABLE_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_ENABLE_DESCRIPTION) + private Boolean active = false; + + @Option(names = {"--argocd-operator"}, description = ARGOCD_OPERATOR_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_OPERATOR_DESCRIPTION) + private Boolean operator = false; + + @Option(names = {"--argocd-url"}, description = ARGOCD_URL_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_URL_DESCRIPTION) + private String url = ""; + + @JsonPropertyDescription(ARGOCD_ENV_DESCRIPTION) + private List> env; + + @Option(names = {"--argocd-email-from"}, description = ARGOCD_EMAIL_FROM_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_EMAIL_FROM_DESCRIPTION) + private String emailFrom = "argocd@example.org"; + + @Option(names = {"--argocd-email-to-user"}, description = ARGOCD_EMAIL_TO_USER_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_EMAIL_TO_USER_DESCRIPTION) + private String emailToUser = "app-team@example.org"; + + @Option(names = {"--argocd-email-to-admin"}, description = ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION) + private String emailToAdmin = "infra@example.org"; + + @Option(names = {"--argocd-resource-inclusions-cluster"}, description = ARGOCD_RESOURCE_INCLUSIONS_CLUSTER) + @JsonPropertyDescription(ARGOCD_RESOURCE_INCLUSIONS_CLUSTER) + private String resourceInclusionsCluster = ""; + + @Option(names = {"--argocd-namespace"}, description = ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION) + @JsonPropertyDescription(ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION) + private String namespace = "argocd"; + + @JsonPropertyDescription(HELM_CONFIG_VALUES_DESCRIPTION) + private Map values = new HashMap<>(); + + @JsonPropertyDescription(OIDC_DESCPRIPTION) + private OidcSchema oidc = new OidcSchema("argocd"); + } + + @Getter + @Setter + @NoArgsConstructor + public static class MailSchema { + private Boolean active = false; + + @Option(names = {"--smtp-address"}, description = SMTP_ADDRESS_DESCRIPTION) + @JsonPropertyDescription(SMTP_ADDRESS_DESCRIPTION) + private String smtpAddress = ""; + + @Option(names = {"--smtp-port"}, description = SMTP_PORT_DESCRIPTION) + @JsonPropertyDescription(SMTP_PORT_DESCRIPTION) + private Integer smtpPort; + + @Option(names = {"--smtp-user"}, description = SMTP_USER_DESCRIPTION) + @JsonPropertyDescription(SMTP_USER_DESCRIPTION) + private String smtpUser = ""; + + @Option(names = {"--smtp-password"}, description = SMTP_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(SMTP_PASSWORD_DESCRIPTION) + private String smtpPassword = ""; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + } + + @Getter + @Setter + public static class MonitoringSchema { + @Option(names = {"--metrics", "--monitoring"}, description = MONITORING_ENABLE_DESCRIPTION) + @JsonPropertyDescription(MONITORING_ENABLE_DESCRIPTION) + private Boolean active = false; + + @Option(names = {"--grafana-url"}, description = GRAFANA_URL_DESCRIPTION) + @JsonPropertyDescription(GRAFANA_URL_DESCRIPTION) + private String grafanaUrl = ""; + + @Option(names = {"--grafana-email-from"}, description = GRAFANA_EMAIL_FROM_DESCRIPTION) + @JsonPropertyDescription(GRAFANA_EMAIL_FROM_DESCRIPTION) + private String grafanaEmailFrom = "grafana@example.org"; + + @Option(names = {"--grafana-email-to"}, description = GRAFANA_EMAIL_TO_DESCRIPTION) + @JsonPropertyDescription(GRAFANA_EMAIL_TO_DESCRIPTION) + private String grafanaEmailTo = "infra@example.org"; + + @JsonPropertyDescription(OIDC_DESCPRIPTION) + private OidcSchema oidc = new OidcSchema("grafana"); + + @Mixin + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private MonitoringHelmSchema helm; + + @Option(names = {"--monitoring-namespace"}, description = MONITORING_NAMESPACE) + @JsonPropertyDescription(MONITORING_NAMESPACE) + private String namespace = "monitoring"; + + public MonitoringSchema() { + helm = new MonitoringHelmSchema(); + helm.setChart("kube-prometheus-stack"); + helm.setRepoURL("https://prometheus-community.github.io/helm-charts"); + // renovate: depName=kube-prometheus-stack registryUrl=https://prometheus-community.github.io/helm-charts + helm.setVersion("80.2.2"); + helm.setValues(new HashMap<>()); + } + + @Getter + @Setter + public static class MonitoringHelmSchema extends HelmConfigWithValues { + @Option(names = {"--grafana-image"}, description = GRAFANA_IMAGE_DESCRIPTION) + @JsonPropertyDescription(GRAFANA_IMAGE_DESCRIPTION) + private String grafanaImage = ""; + + @Option(names = {"--grafana-sidecar-image"}, description = GRAFANA_SIDECAR_IMAGE_DESCRIPTION) + @JsonPropertyDescription(GRAFANA_SIDECAR_IMAGE_DESCRIPTION) + private String grafanaSidecarImage = ""; + + @Option(names = {"--prometheus-image"}, description = PROMETHEUS_IMAGE_DESCRIPTION) + @JsonPropertyDescription(PROMETHEUS_IMAGE_DESCRIPTION) + private String prometheusImage = ""; + + @Option(names = {"--prometheus-operator-image"}, description = PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION) + @JsonPropertyDescription(PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION) + private String prometheusOperatorImage = ""; + + @Option(names = {"--prometheus-config-reloader-image"}, description = PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION) + @JsonPropertyDescription(PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION) + private String prometheusConfigReloaderImage = ""; + } + } + + @Getter + @Setter + public static class SecretsSchema { + private Boolean active = false; + + @Mixin + @JsonPropertyDescription(ESO_DESCRIPTION) + private ESOSchema externalSecrets = new ESOSchema(); + + @Mixin + @JsonPropertyDescription(VAULT_DESCRIPTION) + private VaultSchema vault = new VaultSchema(); + + @Option(names = {"--secrets-namespace"}, description = SECRETS_NAMESPACE) + @JsonPropertyDescription(SECRETS_NAMESPACE) + private String namespace = "secrets"; + + @Getter + @Setter + public static class ESOSchema { + @Mixin + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private ESOHelmSchema helm; + + public ESOSchema() { + helm = new ESOHelmSchema(); + helm.setChart("external-secrets"); + helm.setRepoURL("https://charts.external-secrets.io"); + // renovate: depName=external-secrets registryUrl=https://charts.external-secrets.io + helm.setVersion("0.9.16"); + } + + @Getter + @Setter + public static class ESOHelmSchema extends HelmConfigWithValues { + @Option(names = {"--external-secrets-image"}, description = EXTERNAL_SECRETS_IMAGE_DESCRIPTION) + @JsonPropertyDescription(EXTERNAL_SECRETS_IMAGE_DESCRIPTION) + private String image = ""; + + @Option(names = {"--external-secrets-certcontroller-image"}, description = EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION) + @JsonPropertyDescription(EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION) + private String certControllerImage = ""; + + @Option(names = {"--external-secrets-webhook-image"}, description = EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION) + @JsonPropertyDescription(EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION) + private String webhookImage = ""; + } + } + + @Getter + @Setter + public static class VaultSchema { + @Option(names = {"--vault"}, description = VAULT_ENABLE_DESCRIPTION, converter = VaultModeConverter.class) + @JsonPropertyDescription(VAULT_ENABLE_DESCRIPTION) + private VaultMode mode; + + @Option(names = {"--vault-url"}, description = VAULT_URL_DESCRIPTION) + @JsonPropertyDescription(VAULT_URL_DESCRIPTION) + private String url = ""; + + @JsonPropertyDescription(OIDC_DESCPRIPTION) + private OidcSchema oidc = new OidcSchema("vault"); + + @Mixin + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private VaultHelmSchema helm; + + public VaultSchema() { + helm = new VaultHelmSchema(); + helm.setChart("vault"); + helm.setRepoURL("https://helm.releases.hashicorp.com"); + // renovate: depName=vault registryUrl=https://helm.releases.hashicorp.com + helm.setVersion("0.34.1"); + } + + @Getter + @Setter + public static class VaultHelmSchema extends HelmConfigWithValues { + @Option(names = {"--vault-image"}, description = VAULT_IMAGE_DESCRIPTION) + @JsonPropertyDescription(VAULT_IMAGE_DESCRIPTION) + private String image = ""; + } + + } + } + + @Getter + @Setter + public static class OidcSchema { + @JsonPropertyDescription("Name of the OIDC provider displayed in tool login screens") + private String providerName = "Keycloak"; + + @JsonPropertyDescription("OIDC issuer URL, for example http://keycloak.local.gd/realms/gop") + private String issuerUrl = ""; + + @JsonPropertyDescription("OIDC client ID") + private String clientId = ""; + + @JsonPropertyDescription("OIDC client secret") + private String clientSecret = ""; + + @JsonPropertyDescription("OIDC scopes requested by the tool") + private List scopes = new ArrayList<>(Arrays.asList("openid", "profile", "email")); + + @JsonPropertyDescription("OIDC group that receives full admin permissions in all OIDC-enabled tools") + private String adminGroupName = ""; + + public OidcSchema() { + } + + private OidcSchema(String clientId) { + this.clientId = clientId; + } + + @JsonIgnore + public boolean isEnabled() { + return isNotBlank(clientSecret) && isNotBlank(issuerUrl) && isNotBlank(clientId); + } + + private static boolean isNotBlank(String value) { + return value != null && !value.trim().isEmpty(); + } + } + + @Getter + @Setter + public static class IngressSchema { + @Option(names = {"--ingress"}, description = INGRESS_ENABLE_DESCRIPTION) + @JsonPropertyDescription(INGRESS_ENABLE_DESCRIPTION) + private Boolean active = false; + + @Mixin + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private IngressHelmSchema helm; + + @Option(names = {"--ingress-namespace"}, description = INGRESS_NAMESPACE) + @JsonPropertyDescription(INGRESS_NAMESPACE) + private String ingressNamespace = "ingress"; + + public IngressSchema() { + helm = new IngressHelmSchema(); + helm.setChart("traefik"); + helm.setRepoURL("https://traefik.github.io/charts"); + // renovate: depName=traefik registryUrl=https://traefik.github.io/charts + helm.setVersion("39.0.9"); + } + + @Getter + @Setter + public static class IngressHelmSchema extends HelmConfigWithValues { + @Option(names = {"--ingress-image"}, description = HELM_CONFIG_IMAGE_DESCRIPTION) + @JsonPropertyDescription(HELM_CONFIG_IMAGE_DESCRIPTION) + private String image = ""; + } + } + + @Getter + @Setter + public static class CertManagerSchema { + @Option(names = {"--cert-manager"}, description = CERTMANAGER_ENABLE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_ENABLE_DESCRIPTION) + private Boolean active = false; + + @Option(names = {"--cert-manager-issuer"}, description = CERTMANAGER_ENABLE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_ENABLE_DESCRIPTION) + private String issuer = "cluster-selfsigned"; + + @Option(names = {"--cert-manager-namespace"}, description = CERTMANAGER_NAMESPACE) + @JsonPropertyDescription(CERTMANAGER_NAMESPACE) + private String namespace = "cert-manager"; + + @Mixin + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + private CertManagerHelmSchema helm; + + public CertManagerSchema() { + helm = new CertManagerHelmSchema(); + helm.setChart("cert-manager"); + helm.setRepoURL("https://charts.jetstack.io"); + // renovate: depName=cert-manager registryUrl=https://charts.jetstack.io + helm.setVersion("1.19.4"); + } + + @Getter + @Setter + public static class CertManagerHelmSchema extends HelmConfigWithValues { + @Option(names = {"--cert-manager-image"}, description = CERTMANAGER_IMAGE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_IMAGE_DESCRIPTION) + private String image = ""; + + @Option(names = {"--cert-manager-webhook-image"}, description = CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION) + private String webhookImage = ""; + + @Option(names = {"--cert-manager-cainjector-image"}, description = CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION) + private String cainjectorImage = ""; + + @Option(names = {"--cert-manager-acme-solver-image"}, description = CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION) + private String acmeSolverImage = ""; + + @Option(names = {"--cert-manager-startup-api-check-image"}, description = CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION) + @JsonPropertyDescription(CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION) + private String startupAPICheckImage = ""; + } + } + + public enum ContentRepoType { + FOLDER_BASED, + COPY, + MIRROR + } + + public enum VaultMode { + DEV("dev"), + PROD("prod"); + + private final String externalValue; + + VaultMode(String externalValue) { + this.externalValue = externalValue; + } + + @JsonCreator + public static VaultMode fromExternalValue(String value) { + return Arrays.stream(values()) + .filter(mode -> mode.externalValue.equalsIgnoreCase(value)) + .findFirst() + .orElseThrow(() -> new IllegalArgumentException("Unknown Vault mode: " + value)); + } + + @JsonValue + public String externalValue() { + return externalValue; + } + } + + public static class VaultModeConverter implements ITypeConverter { + @Override + public VaultMode convert(String value) { + return VaultMode.fromExternalValue(value); + } + } + + public enum OverwriteMode { + INIT, + RESET, + UPGRADE + } + + public static Config fromMap(Map map) { + return objectMapper.convertValue(map, Config.class); + } + + public Map toMap() { + return objectMapper.convertValue( + this, new TypeReference<>() { + } + ); + } + + public String toYaml(boolean includeInternals) { + try { + return createYamlMapper(includeInternals).writeValueAsString(this); + } catch (IOException e) { + throw new UncheckedIOException("Failed to write Config as YAML string", e); + } + } + + private static YAMLMapper createYamlMapper(boolean includeInternals) { + if (!includeInternals) { + YAMLMapper mapper = new YAMLMapper(); + mapper.registerModule(new SimpleModule().setSerializerModifier(new BeanSerializerModifier() { + @Override + public List changeProperties( + SerializationConfig serializationConfig, + BeanDescription beanDesc, + List beanProperties) { + return beanProperties.stream() + .filter(writer -> writer.getAnnotation(JsonPropertyDescription.class) != null) + .toList(); + } + })); + return mapper; + } else { + return new YAMLMapper(); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/ConfigConstants.java b/src/main/java/com/cloudogu/gitops/config/ConfigConstants.java new file mode 100644 index 000000000..43cb0e2e0 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/ConfigConstants.java @@ -0,0 +1,186 @@ +package com.cloudogu.gitops.config; + +public final class ConfigConstants { + + public static final String BINARY_NAME = "apply-ng"; + public static final String APP_NAME = "gitops-playground (GOP)"; + public static final String APP_DESCRIPTION = "CLI-tool to deploy gitops-playground."; + public static final String KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION = "Credentials for authentication. They can reference a Kubernetes Secret via secretName and secretNamespace; usernameKey and passwordKey select the Secret data keys."; + + // group registry + public static final String REGISTRY_ENABLE_DESCRIPTION = "Installs a simple cluster-local registry for demonstration purposes. Warning: Registry does not provide authentication!"; + public static final String REGISTRY_DESCRIPTION = "Config parameters for Registry"; + public static final String REGISTRY_INTERNAL_PORT_DESCRIPTION = "Port of registry registry. Ignored when a registry*url params are set"; + public static final String REGISTRY_URL_DESCRIPTION = "The url of your external registry, used for pushing images"; + public static final String REGISTRY_OPTIONAL_WHEN_URL_SET_DESCRIPTION = "Optional when registry-url is set"; + public static final String REGISTRY_PATH_DESCRIPTION = REGISTRY_OPTIONAL_WHEN_URL_SET_DESCRIPTION; + public static final String REGISTRY_USERNAME_DESCRIPTION = REGISTRY_OPTIONAL_WHEN_URL_SET_DESCRIPTION; + public static final String REGISTRY_PASSWORD_DESCRIPTION = REGISTRY_OPTIONAL_WHEN_URL_SET_DESCRIPTION; + + public static final String REGISTRY_PROXY_URL_DESCRIPTION = "The url of your proxy-registry. Used in pipelines to authorize pull base images. Use in conjunction with petclinic base image. Used in helm charts when create-image-pull-secrets is set. Use in conjunction with helm.*image fields."; + public static final String REGISTRY_PROXY_PATH_DESCRIPTION = "Optional when registry-proxy-url is set and the registry is running on a non root web path."; + public static final String REGISTRY_PROXY_USERNAME_DESCRIPTION = "Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set."; + public static final String REGISTRY_PROXY_PASSWORD_DESCRIPTION = "Use with registry-proxy-url, added to Jenkins as credentials and created as pull secrets, when create-image-pull-secrets is set."; + + public static final String REGISTRY_USERNAME_RO_DESCRIPTION = "Optional alternative username for registry-url with read-only permissions that is used when create-image-pull-secrets is set."; + public static final String REGISTRY_PASSWORD_RO_DESCRIPTION = "Optional alternative password for registry-url with read-only permissions that is used when create-image-pull-secrets is set."; + public static final String REGISTRY_CREATE_IMAGE_PULL_SECRETS_DESCRIPTION = "Create image pull secrets for registry and proxy-registry for all GOP namespaces and helm charts. Uses proxy-username, read-only-username or registry-username (in this order). Use this if your cluster is not auto-provisioned with credentials for your private registries or if you configure individual helm images to be pulled from the proxy-registry that requires authentication."; + public static final String REGISTRY_NAMESPACE = "Optional defines the kubernetes namespace for registry."; + + public static final String FEATURES_DESCRIPTION = "Config parameters for features or tools"; + + public static final String CONTENT_DESCRIPTION = "Config parameters for content, i.e. end-user or tenant applications as opposed to cluster-resources"; + + // ContentLoader + public static final String CONTENT_NAMESPACES_DESCRIPTION = "Additional kubernetes namespaces. These are authorized to Argo CD, supplied with image pull secrets, monitored by prometheus, etc. Namespaces can be templates, e.g. ${config.application.namePrefix}staging"; + public static final String CONTENT_REPO_DESCRIPTION = "ContentLoader repos to push into target environment"; + public static final String CONTENT_REPO_URL_DESCRIPTION = "URL of the content repo. Mandatory for each type."; + public static final String CONTENT_REPO_PATH_DESCRIPTION = "Path within the content repo to process"; + public static final String CONTENT_REPO_REF_DESCRIPTION = "Reference for a specific branch, tag, or commit. Emtpy defaults to default branch of the repo. With type MIRROR: ref must not be a commit hash; Choosing a ref only mirrors the ref but does not delete other branches/tags!"; + public static final String CONTENT_REPO_TARGET_REF_DESCRIPTION = "Reference for a specific branch or tag in the target repo of a MIRROR or COPY repo. If ref is a tag, targetRef is treated as tag as well. Except: targetRef is full ref like refs/heads/my-branch or refs/tags/my-tag. Empty defaults to the source ref."; + public static final String CONTENT_REPO_CREDENTIALS_DESCRIPTION = "Credentials Object to authenticate against content repo. Allows using a K8s Secret"; + public static final String CONTENT_REPO_TEMPLATING_DESCRIPTION = "When true, template all files ending in .ftl within the repo"; + public static final String CONTENT_REPO_TYPE_DESCRIPTION = "ContentLoader Repos can either be:\ncopied (only the files, starting on ref, starting at path within the repo. Requires target)\n, mirrored (FORCE pushes ref or the whole git repo if no ref set). Requires target, does not allow path and template.)\nfolderBased (folder structure is interpreted as repos. That is, root folder becomes namespace in SCM, sub folders become repository names in SCM, files are copied. Requires target.)"; + public static final String CONTENT_REPO_TARGET_DESCRIPTION = "Target repo for the repository in the for of namespace/name. Must contain one slash to separate namespace from name."; + public static final String CONTENT_REPO_TARGET_OVERWRITE_MODE_DESCRIPTION = "This defines, how customer repos will be updated.\nINIT - push only if repo does not exist.\nRESET - delete all files after cloning source - files not in content are deleted\nUPGRADE - clone and copy - existing files will be overwritten, files not in content are kept. For type: MIRROR reset and upgrade have same result: in both cases source repo will be force pushed to target repo."; + public static final String CONTENT_REPO_CREATE_JENKINS_JOB_DESCRIPTION = "If true, creates a Jenkins job, if jenkinsfile exists in one of the content repo's branches."; + public static final String CONTENT_VARIABLES_DESCRIPTION = "Additional variables to use in custom templates."; + public static final String CONTENT_STATICSWHITELIST_ENABLED_DESCRIPTION = "Enables the whitelist for statics in content templating"; + public static final String CONTENT_STATICSWHITELIST_DESCRIPTION = "Whitelist for Statics freemarker is allowing in user templates"; + public static final String CONTENT_HELM_RELEASES_DESCRIPTION = "Additional Helm releases to deploy through Argo CD without requiring a content Git repository."; + public static final String CONTENT_HELM_RELEASE_NAME_DESCRIPTION = "Logical name of the Helm release. Used as the feature folder name under 'apps/' and as default for 'releaseName' if not set."; + + public static final String CONTENT_HELM_RELEASE_REPO_URL_DESCRIPTION = "Helm repository URL to fetch the chart from. Use an HTTP(S) Helm repo (must provide an index.yaml) or an OCI registry URL (oci://...)."; + public static final String CONTENT_HELM_RELEASE_CHART_DESCRIPTION = "Helm chart name to install. For HTTP(S) repos this is the chart name from the repo index; for OCI this is the chart artifact name."; + public static final String CONTENT_HELM_RELEASE_VERSION_DESCRIPTION = "Chart version to deploy. Required for Helm charts in Argo CD. For HTTP(S) Helm repos you may use a SemVer range like '*' to always pick the newest version. For OCI registries, specify an explicit version/tag."; + public static final String CONTENT_HELM_RELEASE_NAMESPACE_DESCRIPTION = "Kubernetes namespace to deploy the release into."; + public static final String CONTENT_HELM_RELEASE_RELEASE_NAME_DESCRIPTION = "Helm release name. If empty, the value of 'name' is used."; + public static final String CONTENT_HELM_RELEASE_VALUES_FILE_DESCRIPTION = "Optional path to a YAML values file to load Helm values from.The file must be accessible locally on the machine running GOP. Inline 'values' will be merged on top (inline overrides file)."; + public static final String CONTENT_HELM_RELEASE_VALUES_DESCRIPTION = "Optional inline Helm values. These values are merged on top of 'valuesFile' (if set) and override keys from the file. Use this for small overrides without maintaining a separate file."; + + // group jenkins + public static final String JENKINS_ENABLE_DESCRIPTION = "Installs Jenkins as CI server"; + public static final String JENKINS_SKIP_RESTART_DESCRIPTION = "Skips restarting Jenkins after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades."; + public static final String JENKINS_SKIP_PLUGINS_DESCRIPTION = "Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades."; + public static final String JENKINS_DESCRIPTION = "Config parameters for Jenkins CI/CD Pipeline Server"; + public static final String JENKINS_URL_DESCRIPTION = "The url of your external jenkins"; + public static final String JENKINS_USERNAME_DESCRIPTION = "Mandatory when jenkins-url is set"; + public static final String JENKINS_PASSWORD_DESCRIPTION = "Mandatory when jenkins-url is set"; + public static final String JENKINS_METRICS_USERNAME_DESCRIPTION = "Mandatory when jenkins-url is set and monitoring enabled"; + public static final String JENKINS_METRICS_PASSWORD_DESCRIPTION = "Mandatory when jenkins-url is set and monitoring enabled"; + public static final String JENKINS_IMAGE_DESCRIPTION = "Sets image for Jenkins"; + public static final String MAVEN_CENTRAL_MIRROR_DESCRIPTION = "URL for maven mirror, used by applications built in Jenkins"; + public static final String JENKINS_ADDITIONAL_ENVS_DESCRIPTION = "Set additional environments to Jenkins"; + public static final String JENKINS_NAMESPACE = "Optional defines the kubernetes namespace for Jenkins."; + + // group scmm + public static final String SCM_DESCRIPTION = "Config parameters for Scm"; + public static final String GIT_NAME_DESCRIPTION = "Sets git author and committer name used for initial commits"; + public static final String GIT_EMAIL_DESCRIPTION = "Sets git author and committer email used for initial commits"; + + // MutliTentant + public static final String MULTITENANT_DESCRIPTION = "Multi Tenant Configs"; + + // group remote + public static final String INSECURE_DESCRIPTION = "Sets insecure-mode in cURL which skips cert validation"; + + // group tool configuration + public static final String APPLICATION_DESCRIPTION = "Application configuration parameter for GOP"; + public static final String GRAFANA_IMAGE_DESCRIPTION = "Sets image for grafana"; + public static final String GRAFANA_SIDECAR_IMAGE_DESCRIPTION = "Sets image for grafana's sidecar"; + public static final String PROMETHEUS_IMAGE_DESCRIPTION = "Sets image for prometheus"; + public static final String PROMETHEUS_OPERATOR_IMAGE_DESCRIPTION = "Sets image for prometheus-operator"; + public static final String PROMETHEUS_CONFIG_RELOADER_IMAGE_DESCRIPTION = "Sets image for prometheus-operator's config-reloader"; + public static final String EXTERNAL_SECRETS_IMAGE_DESCRIPTION = "Sets image for external secrets operator"; + public static final String EXTERNAL_SECRETS_CERT_CONTROLLER_IMAGE_DESCRIPTION = "Sets image for external secrets operator's controller"; + public static final String EXTERNAL_SECRETS_WEBHOOK_IMAGE_DESCRIPTION = "Sets image for external secrets operator's webhook"; + public static final String VAULT_IMAGE_DESCRIPTION = "Sets image for vault"; + public static final String BASE_URL_DESCRIPTION = "the external base url (TLD) for all tools, e.g. https://example.com or http://localhost:8080. The individual -url params for argocd, grafana and vault take precedence."; + public static final String URL_SEPARATOR_HYPHEN_DESCRIPTION = "Use hyphens instead of dots to separate application name from base-url"; + public static final String SKIP_CRDS_DESCRIPTION = "Skip installation of CRDs. This requires prior installation of CRDs"; + public static final String NAMESPACE_ISOLATION_DESCRIPTION = "Configure tools to explicitly work with the given namespaces only, and not cluster-wide. This way GOP can be installed without having cluster-admin permissions."; + public static final String MIRROR_REPOS_DESCRIPTION = "Changes the sources of deployed tools so they are not pulled from the internet, but are pulled from git and work in air-gapped environments."; + public static final String NETPOLS_DESCRIPTION = "Sets Network Policies"; + public static final String CLUSTER_ADMIN_DESCRIPTION = "Binds ArgoCD controllers to cluster-admin ClusterRole"; + public static final String OPENSHIFT_DESCRIPTION = "When set, openshift specific resources and configurations are applied"; + public static final String APPLICATION_PROFIL = "Use predefined profile (full, only-argocd, operator-mandants aso.)"; + public static final String APPLICATION_GOP_NAMESPACE = "If set, GOP stores specific information in this namespace."; + public static final String APPLICATION_NAMESPACE = "If set, GOP uses the same Kubernetes namespace for all tools and examples. Attention! Only use for test purposes."; + // group metrics + public static final String MONITORING_DESCRIPTION = "Config parameters for the Monitoring system (prometheus)"; + public static final String MONITORING_ENABLE_DESCRIPTION = "Installs the Kube-Prometheus-Stack. This includes Prometheus, the Prometheus operator, Grafana and some extra resources"; + public static final String MONITORING_NAMESPACE = "Optional defines the kubernetes namespace for monitoring."; + public static final String GRAFANA_URL_DESCRIPTION = "Sets url for grafana"; + public static final String GRAFANA_EMAIL_FROM_DESCRIPTION = "Notifications, define grafana alerts sender email address"; + public static final String GRAFANA_EMAIL_TO_DESCRIPTION = "Notifications, define grafana alerts recipient email address"; + + // group vault / secrets + public static final String SECRETS_DESCRIPTION = "Config parameters for the secrets management"; + public static final String ESO_DESCRIPTION = "Config parameters for the external secrets operator"; + public static final String VAULT_DESCRIPTION = "Config parameters for the secrets-vault"; + public static final String VAULT_ENABLE_DESCRIPTION = "Installs Hashicorp vault and the external secrets operator. Possible values: dev, prod."; + public static final String VAULT_URL_DESCRIPTION = "Sets url for vault ui"; + public static final String SECRETS_NAMESPACE = "Optional defines the kubernetes namespace for secrets."; + + // group external Mailserver + public static final String MAIL_DESCRIPTION = "Config parameters for mail servers"; + public static final String SMTP_ADDRESS_DESCRIPTION = "Sets smtp port of external Mailserver"; + public static final String SMTP_PORT_DESCRIPTION = "Sets smtp port of external Mailserver"; + public static final String SMTP_USER_DESCRIPTION = "Sets smtp username for external Mailserver"; + public static final String SMTP_PASSWORD_DESCRIPTION = "Sets smtp password of external Mailserver"; + + // group debug + public static final String DEBUG_DESCRIPTION = "Debug output"; + public static final String TRACE_DESCRIPTION = "Debug + Show each command executed (set -x)"; + + // group configuration + public static final String USERNAME_DESCRIPTION = "Set initial admin username"; + public static final String PASSWORD_DESCRIPTION = "Set initial admin passwords"; + public static final String PIPE_YES_DESCRIPTION = "Skip confirmation"; + public static final String NAME_PREFIX_DESCRIPTION = "Set name-prefix for repos, jobs, namespaces"; + public static final String DESTROY_DESCRIPTION = "Unroll playground"; + public static final String CONFIG_FILE_DESCRIPTION = "Config file for the application"; + public static final String CONFIG_MAP_DESCRIPTION = "Kubernetes configuration map. Should contain a key `config.yaml`."; + public static final String OUTPUT_CONFIG_FILE_DESCRIPTION = "Output current config as config file as much as possible"; + public static final String POD_RESOURCES_DESCRIPTION = "Write kubernetes resource requests and limits on each pod"; + + // group ArgoCD Operator + public static final String ARGOCD_DESCRIPTION = "Config Parameter for the ArgoCD Operator"; + public static final String ARGOCD_ENABLE_DESCRIPTION = "Install ArgoCD"; + public static final String ARGOCD_URL_DESCRIPTION = "The URL where argocd is accessible. It has to be the full URL with http:// or https://"; + public static final String ARGOCD_EMAIL_FROM_DESCRIPTION = "Notifications, define Argo CD sender email address"; + public static final String ARGOCD_EMAIL_TO_USER_DESCRIPTION = "Notifications, define Argo CD user / app-team recipient email address"; + public static final String ARGOCD_EMAIL_TO_ADMIN_DESCRIPTION = "Notifications, define Argo CD admin recipient email address"; + public static final String ARGOCD_OPERATOR_DESCRIPTION = "Install ArgoCD via an already running ArgoCD Operator"; + public static final String ARGOCD_ENV_DESCRIPTION = "Pass a list of env vars to Argo CD components. Currently only works with operator"; + public static final String ARGOCD_RESOURCE_INCLUSIONS_CLUSTER = "Internal Kubernetes API Server URL https://IP:PORT (kubernetes.default.svc). Needed in argocd-operator resourceInclusions. Use this parameter if argocd.operator=true and NOT running inside a Pod (remote mode). Full URL needed, for example: https://100.125.0.1:443"; + public static final String ARGOCD_CUSTOM_NAMESPACE_DESCRIPTION = "Defines the kubernetes namespace for ArgoCD"; + + // group ingress-class + public static final String INGRESS_DESCRIPTION = "Config parameters for the Ingress Controller"; + public static final String INGRESS_ENABLE_DESCRIPTION = "Sets and enables Ingress Controller"; + public static final String INGRESS_NAMESPACE = "Optional defines the kubernetes namespace for Ingress Controller"; + + // group CERTMANAGER + public static final String CERTMANAGER_DESCRIPTION = "Config parameters for the Cert Manager"; + public static final String CERTMANAGER_ENABLE_DESCRIPTION = "Sets and enables Cert Manager"; + public static final String CERTMANAGER_IMAGE_DESCRIPTION = "Sets image for Cert Manager"; + public static final String CERTMANAGER_WEBHOOK_IMAGE_DESCRIPTION = "Sets webhook Image for Cert Manager"; + public static final String CERTMANAGER_CAINJECTOR_IMAGE_DESCRIPTION = "Sets cainjector Image for Cert Manager"; + public static final String CERTMANAGER_ACME_SOLVER_IMAGE_DESCRIPTION = "Sets acmeSolver Image for Cert Manager"; + public static final String CERTMANAGER_STARTUP_API_CHECK_IMAGE_DESCRIPTION = "Sets startupAPICheck Image for Cert Manager"; + public static final String CERTMANAGER_NAMESPACE = "Optional defines the kubernetes namespace for Cert Manager"; + + // group helm + public static final String HELM_CONFIG_DESCRIPTION = "Common Config parameters for the Helm package manager: Name of Chart (chart), URl of Helm-Repository (repoURL) and Chart Version (version). Note: These config is intended to obtain the chart from a different source (e.g. in air-gapped envs), not to use a different version of a helm chart. Using a different helm chart or version to the one used in the GOP version will likely cause errors."; + public static final String HELM_CONFIG_CHART_DESCRIPTION = "Name of the Helm chart"; + public static final String HELM_CONFIG_REPO_URL_DESCRIPTION = "Repository url from which the Helm chart should be obtained"; + public static final String HELM_CONFIG_VERSION_DESCRIPTION = "The version of the Helm chart to be installed"; + public static final String HELM_CONFIG_IMAGE_DESCRIPTION = "The image of the Helm chart to be installed"; + public static final String HELM_CONFIG_VALUES_DESCRIPTION = "Helm values of the chart, allows overriding defaults and setting values that are not exposed as explicit configuration"; + + public static final String OIDC_DESCPRIPTION = "OIDC Config for this tool. See docs for more infos"; + + private ConfigConstants() { + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/Credentials.java b/src/main/java/com/cloudogu/gitops/config/Credentials.java new file mode 100644 index 000000000..ec63eb276 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/Credentials.java @@ -0,0 +1,84 @@ +package com.cloudogu.gitops.config; + +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import lombok.ToString; + +import static com.cloudogu.gitops.config.ConfigConstants.KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION; + +@Getter +@Setter +@ToString(exclude = "password") +@NoArgsConstructor +public class Credentials { + + private static final String DEFAULT_USERNAME_KEY = "username"; + private static final String DEFAULT_PASSWORD_KEY = "password"; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private String username; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + @JsonIgnore + private String password; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private String secretNamespace; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private String secretName; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private String usernameKey = DEFAULT_USERNAME_KEY; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private String passwordKey = DEFAULT_PASSWORD_KEY; + + public Credentials(String username, String password) { + this(username, password, "", "", DEFAULT_USERNAME_KEY, DEFAULT_PASSWORD_KEY); + } + + public Credentials(String username, String password, String secretName) { + this(username, password, secretName, "", DEFAULT_USERNAME_KEY, DEFAULT_PASSWORD_KEY); + } + + public Credentials(String username, String password, String secretName, String secretNamespace) { + this(username, password, secretName, secretNamespace, DEFAULT_USERNAME_KEY, DEFAULT_PASSWORD_KEY); + } + + public Credentials( + String username, + String password, + String secretName, + String secretNamespace, + String usernameKey) { + this(username, password, secretName, secretNamespace, usernameKey, DEFAULT_PASSWORD_KEY); + } + + public Credentials( + String username, + String password, + String secretName, + String secretNamespace, + String usernameKey, + String passwordKey) { + this.username = username; + this.password = password; + this.secretNamespace = secretNamespace; + this.secretName = secretName; + this.usernameKey = usernameKey; + this.passwordKey = passwordKey; + } + + public Credentials(Credentials unsafeCredentials) { + if (unsafeCredentials != null) { + this.secretNamespace = unsafeCredentials.secretNamespace; + this.secretName = unsafeCredentials.secretName; + this.usernameKey = unsafeCredentials.usernameKey; + this.passwordKey = unsafeCredentials.passwordKey; + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/MultiTenantSchema.java b/src/main/java/com/cloudogu/gitops/config/MultiTenantSchema.java new file mode 100644 index 000000000..d31f754d6 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/MultiTenantSchema.java @@ -0,0 +1,43 @@ +package com.cloudogu.gitops.config; + +import com.cloudogu.gitops.config.scm.ScmCentralSchema.GitlabCentralConfig; +import com.cloudogu.gitops.config.scm.ScmCentralSchema.ScmManagerCentralConfig; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import picocli.CommandLine.Mixin; +import picocli.CommandLine.Option; + +@Getter +@Setter +@NoArgsConstructor +public class MultiTenantSchema { + + public static final String SCM_PROVIDER_TYPE_DESCRIPTION = "The SCM provider type. Possible values: SCM_MANAGER, GITLAB"; + public static final String GITLAB_CONFIG_DESCRIPTION = "Config for GITLAB"; + public static final String SCMM_CONFIG_DESCRIPTION = "Config for SCM-Manager"; + public static final String CENTRAL_ARGOCD_NAMESPACE_DESCRIPTION = "Namespace for the centralized Argocd"; + public static final String CENTRAL_USEDEDICATED_DESCRIPTION = "Toggles the Dedicated Instances Mode. See docs for more info"; + + @Option(names = {"--central-scm-provider"}, description = SCM_PROVIDER_TYPE_DESCRIPTION, defaultValue = "SCM_MANAGER") + @JsonPropertyDescription(SCM_PROVIDER_TYPE_DESCRIPTION) + private ScmProviderType scmProviderType = ScmProviderType.SCM_MANAGER; + + @JsonPropertyDescription(GITLAB_CONFIG_DESCRIPTION) + @Mixin + private GitlabCentralConfig gitlab; + + @JsonPropertyDescription(SCMM_CONFIG_DESCRIPTION) + @Mixin + private ScmManagerCentralConfig scmManager; + + @Option(names = {"--central-argocd-namespace"}, description = CENTRAL_ARGOCD_NAMESPACE_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_ARGOCD_NAMESPACE_DESCRIPTION) + private String centralArgocdNamespace = "argocd"; + + @Option(names = {"--dedicated-instance"}, description = CENTRAL_USEDEDICATED_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_USEDEDICATED_DESCRIPTION) + private Boolean useDedicatedInstance = false; +} diff --git a/src/main/java/com/cloudogu/gitops/config/schema/JsonSchemaGenerator.java b/src/main/java/com/cloudogu/gitops/config/schema/JsonSchemaGenerator.java new file mode 100644 index 000000000..f17a70635 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/schema/JsonSchemaGenerator.java @@ -0,0 +1,41 @@ +package com.cloudogu.gitops.config.schema; + +import com.cloudogu.gitops.config.Config; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import com.github.victools.jsonschema.generator.FieldScope; +import com.github.victools.jsonschema.generator.Option; +import com.github.victools.jsonschema.generator.OptionPreset; +import com.github.victools.jsonschema.generator.SchemaGenerator; +import com.github.victools.jsonschema.generator.SchemaGeneratorConfigBuilder; +import com.github.victools.jsonschema.generator.SchemaVersion; +import com.github.victools.jsonschema.module.jackson.JacksonOption; +import com.github.victools.jsonschema.module.jackson.JacksonSchemaModule; +import jakarta.inject.Singleton; +import tools.jackson.databind.node.ObjectNode; + +@Singleton +public class JsonSchemaGenerator { + + public ObjectNode createSchema() { + SchemaGeneratorConfigBuilder configBuilder = new SchemaGeneratorConfigBuilder( + SchemaVersion.DRAFT_2020_12, + OptionPreset.PLAIN_JSON + ) + // Make the schema strict: Only allow our fields, warn when additional fields are passed + .with(Option.FORBIDDEN_ADDITIONAL_PROPERTIES_BY_DEFAULT) + // Exception to the above: For Maps allow additional fields. + // We use this to allow inline helm values without having to validate them + .with(Option.MAP_VALUES_AS_ADDITIONAL_PROPERTIES) + // All fields can be set to null to use the default + .with(Option.NULLABLE_FIELDS_BY_DEFAULT).with(new JacksonSchemaModule(JacksonOption.FLATTENED_ENUMS_FROM_JSONVALUE)); + + // Apply the rule to include only fields with @JsonProperty annotation (or here, + // @JsonPropertyDescription) + configBuilder.forFields() + .withIgnoreCheck((FieldScope field) -> field.getAnnotation(JsonPropertyDescription.class) == null); + + SchemaGenerator generator = new SchemaGenerator(configBuilder.build()); + + return generator.generateSchema(Config.class); + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/schema/JsonSchemaValidator.java b/src/main/java/com/cloudogu/gitops/config/schema/JsonSchemaValidator.java new file mode 100644 index 000000000..0457b414a --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/schema/JsonSchemaValidator.java @@ -0,0 +1,36 @@ +package com.cloudogu.gitops.config.schema; + +import com.networknt.schema.Schema; +import com.networknt.schema.SchemaRegistry; +import lombok.extern.slf4j.Slf4j; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.ObjectMapper; + +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +@Slf4j +public final class JsonSchemaValidator { + + private static final ObjectMapper objectMapper = new ObjectMapper(); + private static final SchemaRegistry schemaRegistry = SchemaRegistry.builder().build(); + + private JsonSchemaValidator() { + } + + public static void validate(Map yaml) { + JsonNode json = objectMapper.convertValue(yaml, JsonNode.class); + tools.jackson.databind.node.ObjectNode schemaNode = new JsonSchemaGenerator().createSchema(); + Schema schema = schemaRegistry.getSchema(schemaNode); + + log.debug("yaml configuration converted to json for validate {}", json); + + List validationMessages = schema.validate(json); + + if (!validationMessages.isEmpty()) { + String errorMsg = validationMessages.stream().map(Object::toString).collect(Collectors.joining("\n")); + throw new IllegalArgumentException("Config file invalid: " + errorMsg); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/scm/ScmCentralSchema.java b/src/main/java/com/cloudogu/gitops/config/scm/ScmCentralSchema.java new file mode 100644 index 000000000..0b1d24f6b --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/scm/ScmCentralSchema.java @@ -0,0 +1,106 @@ +package com.cloudogu.gitops.config.scm; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.GitlabConfig; +import com.cloudogu.gitops.config.scm.util.ScmManagerConfig; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import lombok.Getter; +import lombok.Setter; +import picocli.CommandLine.Option; + +import static com.cloudogu.gitops.config.ConfigConstants.KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION; + +public final class ScmCentralSchema { + + private ScmCentralSchema() { + } + + @Getter + @Setter + public static class GitlabCentralConfig implements GitlabConfig { + + public static final String CENTRAL_GITLAB_URL_DESCRIPTION = "URL for external Gitlab"; + public static final String CENTRAL_GITLAB_USERNAME_DESCRIPTION = "GitLab username for API access. Must be 'oauth2' when using Personal Access Token (PAT) authentication"; + public static final String CENTRAL_GITLAB_PASSWORD_DESCRIPTION = "Password for SCM Manager authentication"; + public static final String CENTRAL_GITLAB_PARENTGROUP_ID_DESCRIPTION = "Main Group for Gitlab where the GOP creates it's groups/repos"; + + @Option(names = {"--central-gitlab-url"}, description = CENTRAL_GITLAB_URL_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_GITLAB_URL_DESCRIPTION) + private String url = "https://gitlab.com/"; + + @Option(names = {"--central-gitlab-username"}, description = CENTRAL_GITLAB_USERNAME_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_GITLAB_USERNAME_DESCRIPTION) + private String username = "oauth2.0"; + + @Option(names = {"--central-gitlab-token"}, description = CENTRAL_GITLAB_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_GITLAB_PASSWORD_DESCRIPTION) + private String password = ""; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @Option(names = {"--central-gitlab-group-id"}, description = CENTRAL_GITLAB_PARENTGROUP_ID_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_GITLAB_PARENTGROUP_ID_DESCRIPTION) + private String parentGroupId = ""; + + private String gitOpsUsername = ""; + private String defaultVisibility = ""; + + @Override + public Credentials getCredentials() { + return credentials != null ? credentials : new Credentials(username, password); + } + } + + @Getter + @Setter + public static class ScmManagerCentralConfig implements ScmManagerConfig { + + public static final String CENTRAL_SCMM_INTERNAL_DESCRIPTION = "SCM for Central Management is running on the same cluster, so k8s internal URLs can be used for access"; + public static final String CENTRAL_SCMM_URL_DESCRIPTION = "URL for the centralized Management Repo"; + public static final String CENTRAL_SCMM_USERNAME_DESCRIPTION = "CENTRAL SCMM username"; + public static final String CENTRAL_SCMM_PASSWORD_DESCRIPTION = "CENTRAL SCMM password"; + public static final String CENTRAL_SCMM_NAMESPACE_DESCRIPTION = "Namespace where to find the Central SCMM"; + + @Option(names = {"--central-scmm-internal"}, description = CENTRAL_SCMM_INTERNAL_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_SCMM_INTERNAL_DESCRIPTION) + private Boolean internal = false; + + @Option(names = {"--central-scmm-url"}, description = CENTRAL_SCMM_URL_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_SCMM_URL_DESCRIPTION) + private String url = ""; + + @Option(names = {"--central-scmm-username"}, description = CENTRAL_SCMM_USERNAME_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_SCMM_USERNAME_DESCRIPTION) + private String username = ""; + + @Option(names = {"--central-scmm-password"}, description = CENTRAL_SCMM_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_SCMM_PASSWORD_DESCRIPTION) + private String password = ""; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @Option(names = {"--central-scmm-namespace"}, description = CENTRAL_SCMM_NAMESPACE_DESCRIPTION) + @JsonPropertyDescription(CENTRAL_SCMM_NAMESPACE_DESCRIPTION) + private String namespace = "scm-manager"; + + private String gitOpsUsername = ""; + + @Override + public String getIngress() { + return null; // Needed for setup + } + + @Override + public Config.HelmConfigWithValues getHelm() { + return null; // Needed for setup + } + + @Override + public Credentials getCredentials() { + return credentials != null ? credentials : new Credentials(username, password); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/scm/ScmTenantSchema.java b/src/main/java/com/cloudogu/gitops/config/scm/ScmTenantSchema.java new file mode 100644 index 000000000..d89c91ed9 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/scm/ScmTenantSchema.java @@ -0,0 +1,162 @@ +package com.cloudogu.gitops.config.scm; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.GitlabConfig; +import com.cloudogu.gitops.config.scm.util.ScmManagerConfig; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonMerge; +import com.fasterxml.jackson.annotation.JsonPropertyDescription; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import picocli.CommandLine.Mixin; +import picocli.CommandLine.Option; + +import java.util.HashMap; + +import static com.cloudogu.gitops.config.ConfigConstants.HELM_CONFIG_DESCRIPTION; +import static com.cloudogu.gitops.config.ConfigConstants.KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION; + +@Getter +@Setter +@NoArgsConstructor +public class ScmTenantSchema { + + public static final String GITLAB_CONFIG_DESCRIPTION = "Config for GITLAB"; + public static final String SCMM_CONFIG_DESCRIPTION = "Config for SCM-Manager"; + public static final String SCM_PROVIDER_TYPE_DESCRIPTION = "The SCM provider type. Possible values: SCM_MANAGER, GITLAB"; + public static final String GITOPSUSERNAME_DESCRIPTION = "Username for the Gitops User"; + + @Option(names = {"--scm-provider"}, description = SCM_PROVIDER_TYPE_DESCRIPTION, defaultValue = "SCM_MANAGER") + @JsonPropertyDescription(SCM_PROVIDER_TYPE_DESCRIPTION) + private ScmProviderType scmProviderType = ScmProviderType.SCM_MANAGER; + + @JsonPropertyDescription(GITLAB_CONFIG_DESCRIPTION) + @Mixin + private GitlabTenantConfig gitlab; + + @JsonPropertyDescription(SCMM_CONFIG_DESCRIPTION) + @Mixin + private ScmManagerTenantConfig scmManager; + + @JsonIgnore + public Boolean getInternal() { + return (gitlab != null && gitlab.getInternal()) || (scmManager != null && scmManager.getInternal()); + } + + @Getter + @Setter + @NoArgsConstructor + public static class GitlabTenantConfig implements GitlabConfig { + + public static final String GITLAB_INTERNAL_DESCRIPTION = "True if Gitlab is running in the same K8s cluster. For now we only support access by external URL"; + public static final String GITLAB_URL_DESCRIPTION = "Base URL for the Gitlab instance"; + public static final String GITLAB_USERNAME_DESCRIPTION = "Defaults to: oauth2.0 when PAT token is given."; + public static final String GITLAB_TOKEN_DESCRIPTION = "PAT Token for the account. Needs read/write repo permissions. See docs for mor information"; + public static final String GITLAB_PARENT_GROUP_ID = "Number for the Gitlab Group where the repos and subgroups should be created"; + + @JsonPropertyDescription(GITLAB_INTERNAL_DESCRIPTION) + private Boolean internal = false; + + @Option(names = {"--gitlab-url"}, description = GITLAB_URL_DESCRIPTION) + @JsonPropertyDescription(GITLAB_URL_DESCRIPTION) + private String url; + + @Option(names = {"--gitlab-username"}, description = GITLAB_USERNAME_DESCRIPTION) + @JsonPropertyDescription(GITLAB_USERNAME_DESCRIPTION) + private String username = "oauth2.0"; + + @Option(names = {"--gitlab-token"}, description = GITLAB_TOKEN_DESCRIPTION) + @JsonPropertyDescription(GITLAB_TOKEN_DESCRIPTION) + private String password; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @Option(names = {"--gitlab-group-id"}, description = GITLAB_PARENT_GROUP_ID) + @JsonPropertyDescription(GITLAB_PARENT_GROUP_ID) + private String parentGroupId = ""; + + @JsonPropertyDescription(GITOPSUSERNAME_DESCRIPTION) + private String gitOpsUsername = ""; + + private String defaultVisibility = ""; + + @Override + public Credentials getCredentials() { + return credentials != null ? credentials : new Credentials(username, password); + } + } + + @Getter + @Setter + public static class ScmManagerTenantConfig implements ScmManagerConfig { + + public static final String SCMM_SKIP_RESTART_DESCRIPTION = "Skips restarting SCM-Manager after plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades.'"; + public static final String SCMM_SKIP_PLUGINS_DESCRIPTION = "Skips plugin installation. Use with caution! If the plugins are not installed up front, the installation will likely fail. The intended use case for this is after the first installation, for config changes only. Do not use on first installation or upgrades."; + public static final String SCMM_URL_DESCRIPTION = "The host of your external scm-manager"; + public static final String SCMM_USERNAME_DESCRIPTION = "Mandatory when scmm-url is set"; + public static final String SCMM_PASSWORD_DESCRIPTION = "Mandatory when scmm-url is set"; + public static final String SCMM_NAMESPACE_DESCRIPTION = "Namespace where SCM-Manager should run"; + public static final String SCMM_IMAGE = "Sets image for SCM-Manager"; + + private Boolean internal = true; + + @Option(names = {"--scmm-url"}, description = SCMM_URL_DESCRIPTION) + @JsonPropertyDescription(SCMM_URL_DESCRIPTION) + private String url = ""; + + @Option(names = {"--scmm-namespace"}, description = SCMM_NAMESPACE_DESCRIPTION) + @JsonPropertyDescription(SCMM_NAMESPACE_DESCRIPTION) + private String namespace = "scm-manager"; + + @Option(names = {"--scmm-username"}, description = SCMM_USERNAME_DESCRIPTION) + @JsonPropertyDescription(SCMM_USERNAME_DESCRIPTION) + private String username = Config.DEFAULT_ADMIN_USER; + + @Option(names = {"--scmm-password"}, description = SCMM_PASSWORD_DESCRIPTION) + @JsonPropertyDescription(SCMM_PASSWORD_DESCRIPTION) + private String password = Config.DEFAULT_ADMIN_PW; + + @JsonPropertyDescription(KUBERNETES_SECRET_CREDENTIALS_DESCRIPTION) + private Credentials credentials; + + @JsonPropertyDescription(HELM_CONFIG_DESCRIPTION) + @JsonMerge + private Config.HelmConfigWithValues helm; + + @Option(names = {"--scmm-image"}, description = SCMM_IMAGE) + @JsonPropertyDescription(SCMM_IMAGE) + private String scmmImage = ""; + + private String urlForJenkins = ""; + private String ingress = ""; + + @Option(names = {"--scmm-skip-restart"}, description = SCMM_SKIP_RESTART_DESCRIPTION) + @JsonPropertyDescription(SCMM_SKIP_RESTART_DESCRIPTION) + private Boolean skipRestart = false; + + @Option(names = {"--scmm-skip-plugins"}, description = SCMM_SKIP_PLUGINS_DESCRIPTION) + @JsonPropertyDescription(SCMM_SKIP_PLUGINS_DESCRIPTION) + private Boolean skipPlugins = false; + + @JsonPropertyDescription(GITOPSUSERNAME_DESCRIPTION) + private String gitOpsUsername = ""; + + public ScmManagerTenantConfig() { + helm = new Config.HelmConfigWithValues(); + helm.setChart("scm-manager"); + helm.setRepoURL("https://packages.scm-manager.org/repository/helm-v2-releases/"); + // renovate: depName=scm-manager registryUrl=https://packages.scm-manager.org/repository/helm-v2-releases/ + helm.setVersion("3.11.10"); + helm.setValues(new HashMap<>()); + } + + @Override + public Credentials getCredentials() { + return credentials != null ? credentials : new Credentials(username, password); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/config/scm/util/GitlabConfig.java b/src/main/java/com/cloudogu/gitops/config/scm/util/GitlabConfig.java new file mode 100644 index 000000000..cfd9d1b95 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/scm/util/GitlabConfig.java @@ -0,0 +1,15 @@ +package com.cloudogu.gitops.config.scm.util; + +import com.cloudogu.gitops.config.Credentials; + +public interface GitlabConfig { + String getUrl(); + + String getParentGroupId(); + + String getDefaultVisibility(); + + String getGitOpsUsername(); + + Credentials getCredentials(); +} diff --git a/src/main/java/com/cloudogu/gitops/config/scm/util/ScmManagerConfig.java b/src/main/java/com/cloudogu/gitops/config/scm/util/ScmManagerConfig.java new file mode 100644 index 000000000..8c54d50b8 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/scm/util/ScmManagerConfig.java @@ -0,0 +1,24 @@ +package com.cloudogu.gitops.config.scm.util; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; + +public interface ScmManagerConfig { + Boolean getInternal(); + + String getUrl(); + + String getUsername(); + + String getPassword(); + + String getNamespace(); + + String getIngress(); + + Config.HelmConfigWithValues getHelm(); + + String getGitOpsUsername(); + + Credentials getCredentials(); +} diff --git a/src/main/java/com/cloudogu/gitops/config/scm/util/ScmProviderType.java b/src/main/java/com/cloudogu/gitops/config/scm/util/ScmProviderType.java new file mode 100644 index 000000000..c35c9b5ae --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/config/scm/util/ScmProviderType.java @@ -0,0 +1,6 @@ +package com.cloudogu.gitops.config.scm.util; + +public enum ScmProviderType { + GITLAB, + SCM_MANAGER +} diff --git a/src/main/java/com/cloudogu/gitops/dependencyinjection/HttpClientFactory.java b/src/main/java/com/cloudogu/gitops/dependencyinjection/HttpClientFactory.java new file mode 100644 index 000000000..ff82859fc --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/dependencyinjection/HttpClientFactory.java @@ -0,0 +1,103 @@ +package com.cloudogu.gitops.dependencyinjection; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.dependencyinjection.okhttp.RetryInterceptor; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.AuthorizationInterceptor; +import io.micronaut.context.annotation.Factory; +import jakarta.inject.Named; +import jakarta.inject.Singleton; +import lombok.Value; +import okhttp3.JavaNetCookieJar; +import okhttp3.OkHttpClient; +import okhttp3.logging.HttpLoggingInterceptor; +import org.slf4j.LoggerFactory; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocketFactory; +import javax.net.ssl.TrustManager; +import javax.net.ssl.X509TrustManager; +import java.net.CookieManager; +import java.security.GeneralSecurityException; +import java.security.SecureRandom; +import java.security.cert.CertificateException; +import java.security.cert.X509Certificate; + +@Factory +public class HttpClientFactory { + + public static OkHttpClient buildOkHttpClient(Credentials credentials, Boolean isInsecure) { + OkHttpClient.Builder builder = new OkHttpClient.Builder().addInterceptor(new AuthorizationInterceptor( + credentials.getUsername(), + credentials.getPassword() + )) + .addInterceptor(createLoggingInterceptor()) + .addInterceptor(new RetryInterceptor()); + + if (Boolean.TRUE.equals(isInsecure)) { + InsecureSslContext context = insecureSslContext(); + builder.sslSocketFactory(context.getSocketFactory(), context.getTrustManager()); + builder.hostnameVerifier((hostname, session) -> true); + } + + return builder.build(); + } + + @Singleton + @Named("jenkins") + public OkHttpClient okHttpClientJenkins(Config config) { + OkHttpClient.Builder builder = new OkHttpClient.Builder().cookieJar(new JavaNetCookieJar(new CookieManager())) + .addInterceptor(createLoggingInterceptor()) + .addInterceptor(new RetryInterceptor()); + + if (config.getApplication().getInsecure()) { + InsecureSslContext sslContext = insecureSslContext(); + builder.sslSocketFactory(sslContext.getSocketFactory(), sslContext.getTrustManager()); + builder.hostnameVerifier((hostname, session) -> true); + } + + return builder.build(); + } + + public static HttpLoggingInterceptor createLoggingInterceptor() { + HttpLoggingInterceptor ret = new HttpLoggingInterceptor(LoggerFactory.getLogger("com.cloudogu.gitops.HttpClient")::trace); + + ret.setLevel(HttpLoggingInterceptor.Level.HEADERS); + ret.redactHeader("Authorization"); + + return ret; + } + + public static InsecureSslContext insecureSslContext() { + try { + X509TrustManager noCheckTrustManager = new X509TrustManager() { + @Override + public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException { + // Intentionally empty: this trust manager accepts all client certificates + } + + @Override + public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException { + // Intentionally empty: this trust manager accepts all server certificates + } + + @Override + public X509Certificate[] getAcceptedIssuers() { + return new X509Certificate[0]; + } + }; + SSLContext sslCtxt = SSLContext.getInstance("TLS"); + sslCtxt.init(null, new TrustManager[]{noCheckTrustManager}, new SecureRandom()); + + return new InsecureSslContext(sslCtxt.getSocketFactory(), noCheckTrustManager); + } catch (GeneralSecurityException e) { + throw new IllegalStateException("Failed to construct insecure SSL context", e); + } + } + + @Value + public static class InsecureSslContext { + SSLSocketFactory socketFactory; + X509TrustManager trustManager; + } +} diff --git a/src/main/java/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptor.java b/src/main/java/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptor.java new file mode 100644 index 000000000..82e94682a --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptor.java @@ -0,0 +1,84 @@ +package com.cloudogu.gitops.dependencyinjection.okhttp; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import okhttp3.Interceptor; +import okhttp3.Response; +import org.jetbrains.annotations.NotNull; + +import java.io.IOException; +import java.net.SocketTimeoutException; +import java.util.Set; + +@RequiredArgsConstructor +@Slf4j +public class RetryInterceptor implements Interceptor { + + private static final Set STATUS_CODES_TO_RETRY = Set.of( + 408, // Request Timeout + 429, // Too Many Requests + 500, // Internal Server Error + 502, // Bad Gateway + 503, // Service Unavailable + 504 // Gateway Timeout + ); + + private static final int DEFAULT_RETRIES = 180; + private static final int DEFAULT_WAIT_PERIOD_MS = 2000; + + private final int retries; + private final int waitPeriodInMs; + + public RetryInterceptor() { + this(DEFAULT_RETRIES, DEFAULT_WAIT_PERIOD_MS); + } + + @NotNull + @Override + public Response intercept(@NotNull Chain chain) throws IOException { + int i = 0; + int lastStatusCode = -1; + IOException lastException = null; + + do { + try { + Response response = chain.proceed(chain.request()); + + if (!STATUS_CODES_TO_RETRY.contains(response.code())) { + // Success or non-retriable error - return the response + return response; + } + + log.trace("Retry HTTP Request to {} due to status code {}", chain.request().url(), response.code()); + lastStatusCode = response.code(); + response.close(); + + } catch (SocketTimeoutException e) { + lastException = e; + log.trace( + "Retry HTTP Request to {} due to SocketTimeoutException: {}", chain.request() + .url(), e.getMessage() + ); + } + + // Wait before next retry (but not after the last attempt) + if (i < retries) { + try { + Thread.sleep(waitPeriodInMs); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("Retry interceptor interrupted", e); + } + } + ++i; + + } while (i <= retries); + + // If we got here, all retries failed + if (lastException != null) { + throw lastException; + } + throw new IOException("Request to " + chain.request() + .url() + " failed after " + retries + " retries, last status code " + lastStatusCode); + } +} diff --git a/src/main/java/com/cloudogu/gitops/destroy/ArgoCDDestructionHandler.java b/src/main/java/com/cloudogu/gitops/destroy/ArgoCDDestructionHandler.java new file mode 100644 index 000000000..3c2be6c46 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/destroy/ArgoCDDestructionHandler.java @@ -0,0 +1,102 @@ +package com.cloudogu.gitops.destroy; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient.CustomResource; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.MapUtils; +import com.cloudogu.gitops.utils.Tuple; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.nio.file.Path; +import java.util.List; +import java.util.Map; + +@Singleton +@Order(100) +@RequiredArgsConstructor +public class ArgoCDDestructionHandler implements DestructionHandler { + + private static final String ARGOCD = "argocd"; + + private final Config config; + private final K8sClient k8sClient; + private final HelmClient helmClient; + private final GitRepoFactory repoProvider; + private final FileSystemUtils fileSystemUtils; + private final GitHandler gitHandler; + + @Override + public void destroy() { + String namePrefix = config.getApplication().getNamePrefix(); + + GitRepo repo = repoProvider.create("argocd/cluster-resources", gitHandler.getResourcesScm()); + try { + repo.cloneRepo(); + } catch (Exception e) { + throw new RuntimeException("Failed to clone argocd cluster-resources repo", e); + } + + for (CustomResource app : k8sClient.getCustomResource("app")) { + if ("bootstrap".equals(app.name()) || ARGOCD.equals(app.name()) || "projects".equals(app.name())) { + continue; + } + + k8sClient.patch( + "app", + app.name(), + app.namespace(), + "merge", + Map.of("metadata", Map.of("finalizers", List.of("resources-finalizer.argocd.argoproj.io"))) + ); + } + + String argocdNamespace = namePrefix + config.getFeatures().getArgocd().getNamespace(); + List> appsToBeDeleted = List.of( + new Tuple<>(argocdNamespace, "bootstrap"), + new Tuple<>(argocdNamespace, "cluster-resources"), + new Tuple<>(argocdNamespace, "example-apps") + ); + + for (Tuple app : appsToBeDeleted) { + k8sClient.delete("app", app.getV1(), app.getV2()); + } + + installArgoCDViaHelm(repo, argocdNamespace); + helmClient.uninstall(ARGOCD, ARGOCD); + for (CustomResource project : k8sClient.getCustomResource("appprojects")) { + k8sClient.delete("appproject", project.namespace(), project.name()); + } + + k8sClient.delete("app", argocdNamespace, "projects"); + k8sClient.delete("app", argocdNamespace, ARGOCD); + + String jenkinsNamespace = config.getJenkins().getInternal() ? (namePrefix + config.getJenkins() + .getNamespace()) : null; + if (jenkinsNamespace != null) { + k8sClient.delete("secret", jenkinsNamespace, "jenkins-credentials"); + } + k8sClient.delete("secret", argocdNamespace, "argocd-repo-creds-scm"); + } + + public void installArgoCDViaHelm(GitRepo repo, String argocdNamespace) { + String umbrellaChartPath = Path.of(repo.getAbsoluteLocalRepoTmpDir(), "argocd/").toString(); + + List> helmDependencies = MapUtils.asListOfStringObjectMaps(fileSystemUtils.readYaml(Path.of( + umbrellaChartPath, + "Chart.yaml" + )) + .get( + "dependencies")); + helmClient.addRepo("argo", (String) helmDependencies.get(0).get("repository")); + helmClient.dependencyBuild(umbrellaChartPath); + helmClient.upgrade(ARGOCD, umbrellaChartPath, Map.of("namespace", argocdNamespace)); + } + +} diff --git a/src/main/java/com/cloudogu/gitops/destroy/Destroyer.java b/src/main/java/com/cloudogu/gitops/destroy/Destroyer.java new file mode 100644 index 000000000..6c9615835 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/destroy/Destroyer.java @@ -0,0 +1,26 @@ +package com.cloudogu.gitops.destroy; + +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.util.List; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class Destroyer { + + @Getter + private final List destructionHandlers; + + public void destroy() { + log.info("Start destroying"); + for (DestructionHandler handler : destructionHandlers) { + log.info("Running handler {}", handler.getClass().getSimpleName()); + handler.destroy(); + } + log.info("Finished destroying"); + } +} diff --git a/src/main/java/com/cloudogu/gitops/destroy/DestructionHandler.java b/src/main/java/com/cloudogu/gitops/destroy/DestructionHandler.java new file mode 100644 index 000000000..f7ea74fc1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/destroy/DestructionHandler.java @@ -0,0 +1,5 @@ +package com.cloudogu.gitops.destroy; + +public interface DestructionHandler { + void destroy(); +} diff --git a/src/main/java/com/cloudogu/gitops/destroy/JenkinsDestructionHandler.java b/src/main/java/com/cloudogu/gitops/destroy/JenkinsDestructionHandler.java new file mode 100644 index 000000000..f23baf3bb --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/destroy/JenkinsDestructionHandler.java @@ -0,0 +1,32 @@ +package com.cloudogu.gitops.destroy; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.jenkins.GlobalPropertyManager; +import com.cloudogu.gitops.infrastructure.jenkins.JobManager; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@Order(300) +@RequiredArgsConstructor +public class JenkinsDestructionHandler implements DestructionHandler { + + private final JobManager jobManager; + private final Config config; + private final GlobalPropertyManager globalPropertyManager; + + @Override + public void destroy() { + String namePrefixForEnvVars = config.getApplication().getNamePrefixForEnvVars(); + + jobManager.deleteJob(config.getApplication().getNamePrefix() + "example-apps"); + globalPropertyManager.deleteGlobalProperty("SCMM_URL"); + globalPropertyManager.deleteGlobalProperty(namePrefixForEnvVars + "REGISTRY_URL"); + globalPropertyManager.deleteGlobalProperty(namePrefixForEnvVars + "REGISTRY_PATH"); + globalPropertyManager.deleteGlobalProperty(namePrefixForEnvVars + "REGISTRY_PROXY_URL"); + globalPropertyManager.deleteGlobalProperty(namePrefixForEnvVars + "REGISTRY_PROXY_PATH"); + + globalPropertyManager.deleteGlobalProperty(namePrefixForEnvVars + "K8S_VERSION"); + } +} diff --git a/src/main/java/com/cloudogu/gitops/destroy/ScmmDestructionHandler.java b/src/main/java/com/cloudogu/gitops/destroy/ScmmDestructionHandler.java new file mode 100644 index 000000000..8c4206983 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/destroy/ScmmDestructionHandler.java @@ -0,0 +1,102 @@ +package com.cloudogu.gitops.destroy; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManagerUrlResolver; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.NetworkingUtils; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import okhttp3.ResponseBody; +import retrofit2.Response; + +import java.io.IOException; + +@Singleton +@Order(200) +@RequiredArgsConstructor +public class ScmmDestructionHandler implements DestructionHandler { + + private static final String ARGOCD = "argocd"; + private static final String THIRD_PARTY_DEPENDENCIES = "3rd-party-dependencies"; + private static final int HTTP_NO_CONTENT = 204; + private static final int HTTP_NOT_FOUND = 404; + + private final Config config; + private final K8sClient k8sClient; + private final NetworkingUtils networkingUtils; + + private ScmManagerApiClient scmmApiClient; + + @Override + public void destroy() { + deleteUser("gitops"); + deleteRepository(ARGOCD, ARGOCD); + deleteRepository(ARGOCD, "cluster-resources"); + deleteRepository(ARGOCD, "example-apps"); + deleteRepository(THIRD_PARTY_DEPENDENCIES, "ces-build-lib", false); + deleteRepository(THIRD_PARTY_DEPENDENCIES, "gitops-build-lib", false); + deleteRepository(THIRD_PARTY_DEPENDENCIES, "spring-boot-helm-chart", false); + deleteRepository(THIRD_PARTY_DEPENDENCIES, "spring-boot-helm-chart-with-dependency", false); + } + + private void deleteRepository(String namespace, String repository, boolean prefixNamespace) { + String namePrefix = prefixNamespace ? config.getApplication().getNamePrefix() : ""; + try { + Response response = getScmmApiClient().repositoryApi() + .delete(namePrefix + namespace, repository) + .execute(); + if (response.code() != HTTP_NO_CONTENT && response.code() != HTTP_NOT_FOUND) { + throw new IllegalStateException("Could not delete repository " + namespace + "/" + repository + " (" + response.code() + " " + response.message() + "): " + readErrorBody( + response)); + } + } catch (Exception e) { + throw new RuntimeException("Failed to delete repository " + namespace + "/" + repository, e); + } + } + + private void deleteRepository(String namespace, String repository) { + deleteRepository(namespace, repository, true); + } + + private void deleteUser(String name) { + try { + Response response = getScmmApiClient().usersApi() + .delete(config.getApplication().getNamePrefix() + name) + .execute(); + if (response.code() != HTTP_NO_CONTENT && response.code() != HTTP_NOT_FOUND) { + throw new IllegalStateException("Could not delete user " + name + " (" + response.code() + " " + response.message() + "): " + readErrorBody( + response)); + } + } catch (Exception e) { + throw new RuntimeException("Failed to delete user " + name, e); + } + } + + private static String readErrorBody(Response response) throws IOException { + try (ResponseBody errorBody = response.errorBody()) { + return errorBody != null ? errorBody.string() : ""; + } + } + + private ScmManagerApiClient getScmmApiClient() { + if (scmmApiClient == null) { + ScmManagerUrlResolver urls = new ScmManagerUrlResolver( + config.getScm().getScmManager(), + k8sClient, + networkingUtils, + config.getApplication().getNamePrefix(), + config.getApplication().getRunningInsideK8s() + ); + + scmmApiClient = new ScmManagerApiClient( + urls.clientApiBase().toString(), config.getScm() + .getScmManager() + .getCredentials(), config.getApplication() + .getInsecure() + ); + } + return scmmApiClient; + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategy.java b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategy.java new file mode 100644 index 000000000..76cad65f1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategy.java @@ -0,0 +1,238 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class ArgoCdApplicationStrategy implements DeploymentStrategy { + + // Git repository paths always use '/', regardless of the host OS + private static final String GIT_PATH_SEPARATOR = "/"; + + private final ArgoCdApplicationTargetResolver targetResolver; + + @Override + public void deployFeature( + String repoURL, + String repoName, + String chartOrPath, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + RepoType repoType, + DeploymentContext context, + RepositoryWorkspace repositoryWorkspace) { + + log.trace("Deploying helm chart via ArgoCD: {}. Reading values from {}", releaseName, helmValuesPath); + + GitRepo clusterResourcesRepo = repositoryWorkspace.getClusterResourcesRepository(); + + String toolName = repoName; + + String toolPath = "apps/" + toolName; + String repoRoot = clusterResourcesRepo.getAbsoluteLocalRepoTmpDir(); + + String inlineValues; + try { + Files.createDirectories(Path.of(repoRoot, toolPath)); + Files.createDirectories(Path.of(repoRoot, "apps/argocd/applications")); + inlineValues = Files.readString(helmValuesPath); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + + ValuesFilePaths valuesFilePaths = ValuesFilePaths.of(toolPath, toolName, repoRoot); + + boolean bootstrapDeploymentRequired = requiresBootstrapDeployment(toolName); + ArgoCdApplicationTarget target = targetResolver.resolve(context, repoName); + + if (bootstrapDeploymentRequired) { + log.info( + "Using bootstrap deployment for tool '{}': applicationName='{}', releaseName='{}', namespace='{}'. " + "Helm values will be embedded into the ArgoCD Application and no external values source will be referenced.", + toolName, + target.getApplicationName(), + releaseName, + namespace + ); + } else { + writeValuesFiles(clusterResourcesRepo, toolName, valuesFilePaths, inlineValues); + } + + Map helmSource = new LinkedHashMap<>(); + helmSource.put("repoURL", repoURL); + helmSource.put(chooseKeyChartOrPath(repoType), chartOrPath); + helmSource.put("targetRevision", version); + helmSource.put( + "helm", + buildHelmValuesConfig(releaseName, bootstrapDeploymentRequired, toolName, inlineValues, valuesFilePaths) + ); + + List> sources = new ArrayList<>(); + sources.add(helmSource); + + if (!bootstrapDeploymentRequired) { + sources.add(buildGitValuesSource(clusterResourcesRepo, toolPath)); + } + + String yamlResult = renderApplicationYaml(target, namespace, sources); + + String appManifestPath = "apps/argocd/applications/" + releaseName + ".yaml"; + + try { + clusterResourcesRepo.writeFile(appManifestPath, yamlResult); + } catch (Exception e) { + throw new RuntimeException("Failed to write ArgoCD application manifest for " + releaseName, e); + } + + log.debug( + "Prepared ArgoCD application for helm release {} basing on chart {} from {}, version {}, into namespace {}. Application was written to shared repository workspace:\n{}", + releaseName, + chartOrPath, + repoURL, + version, + namespace, + yamlResult + ); + } + + private static void writeValuesFiles( + GitRepo clusterResourcesRepo, + String toolName, + ValuesFilePaths valuesFilePaths, + String inlineValues) { + try { + clusterResourcesRepo.writeFile(valuesFilePaths.gopValuesPath(), inlineValues); + + if (!valuesFilePaths.userValuesAbsPath().toFile().exists()) { + clusterResourcesRepo.writeFile(valuesFilePaths.userValuesPath(), ""); + } + } catch (Exception e) { + throw new RuntimeException("Failed to write values files for " + toolName, e); + } + } + + private static Map buildHelmValuesConfig( + String releaseName, + boolean bootstrapDeploymentRequired, + String toolName, + String inlineValues, + ValuesFilePaths valuesFilePaths) { + Map helmConfig = new LinkedHashMap<>(); + helmConfig.put("releaseName", releaseName); + + if (bootstrapDeploymentRequired) { + log.trace( + "Embedding Helm values for bootstrap tool '{}' directly into the ArgoCD Application to avoid a self-referencing values source.", + toolName + ); + helmConfig.put("values", inlineValues); + } else { + helmConfig.put( + "valueFiles", + List.of("$values/" + valuesFilePaths.gopValuesPath(), "$values/" + valuesFilePaths.userValuesPath()) + ); + helmConfig.put("ignoreMissingValueFiles", true); + } + return helmConfig; + } + + /** + * Locations of the gop and user Helm values files of a tool within the cluster-resources repo. + */ + private record ValuesFilePaths( + String gopValuesPath, + + String userValuesPath, + + Path userValuesAbsPath + ) { + + static ValuesFilePaths of(String toolPath, String toolName, String repoRoot) { + String gopValuesPath = toolPath + GIT_PATH_SEPARATOR + toolName + "-gop-helm.yaml"; + String userValuesPath = toolPath + GIT_PATH_SEPARATOR + toolName + "-user-values.yaml"; + return new ValuesFilePaths(gopValuesPath, userValuesPath, Path.of(repoRoot, userValuesPath)); + } + } + + private static Map buildGitValuesSource(GitRepo clusterResourcesRepo, String toolPath) { + String toolRepoUrl = clusterResourcesRepo.getGitProvider().repoPrefix() + "argocd/cluster-resources.git"; + + Map gitSource = new LinkedHashMap<>(); + gitSource.put("repoURL", toolRepoUrl); + gitSource.put("targetRevision", "main"); + gitSource.put("ref", "values"); + gitSource.put("path", toolPath); + gitSource.put("directory", Map.of("recurse", true)); + return gitSource; + } + + private static String renderApplicationYaml( + ArgoCdApplicationTarget target, + String namespace, + List> sources) { + String namespaceCreationSyncOption = "CreateNamespace=" + target.isCreateDestinationNamespace(); + + Map syncPolicy = new LinkedHashMap<>(); + Map automated = new LinkedHashMap<>(); + automated.put("prune", true); + automated.put("selfHeal", true); + syncPolicy.put("automated", automated); + syncPolicy.put("syncOptions", List.of("ServerSideApply=true", namespaceCreationSyncOption)); + + Map application = new LinkedHashMap<>(); + application.put("apiVersion", "argoproj.io/v1alpha1"); + application.put("kind", "Application"); + + Map metadata = new LinkedHashMap<>(); + metadata.put("name", target.getApplicationName()); + metadata.put("namespace", target.getNamespace()); + application.put("metadata", metadata); + + Map spec = new LinkedHashMap<>(); + Map destination = new LinkedHashMap<>(); + destination.put("server", "https://kubernetes.default.svc"); + destination.put("namespace", namespace); + spec.put("destination", destination); + spec.put("project", target.getProject()); + spec.put("sources", sources); + spec.put("syncPolicy", syncPolicy); + application.put("spec", spec); + + YAMLMapper yamlMapper = YAMLMapper.builder().enable(YAMLGenerator.Feature.LITERAL_BLOCK_STYLE).build(); + try { + return yamlMapper.writeValueAsString(application); + } catch (JsonProcessingException e) { + throw new UncheckedIOException("Failed to generate YAML for ArgoCD application", e); + } + } + + public String chooseKeyChartOrPath(RepoType repoType) { + return switch (repoType) { + case HELM -> "chart"; + case GIT -> "path"; + }; + } + + private static boolean requiresBootstrapDeployment(String toolName) { + return "scm-manager".equals(toolName); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTarget.java b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTarget.java new file mode 100644 index 000000000..fe016f9c8 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTarget.java @@ -0,0 +1,14 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import lombok.Getter; +import lombok.RequiredArgsConstructor; + +@Getter +@RequiredArgsConstructor +public class ArgoCdApplicationTarget { + + private final String applicationName; + private final String namespace; + private final String project; + private final boolean createDestinationNamespace; +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTargetResolver.java b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTargetResolver.java new file mode 100644 index 000000000..801dabe99 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTargetResolver.java @@ -0,0 +1,38 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.regex.Pattern; + +@Singleton +@RequiredArgsConstructor +public class ArgoCdApplicationTargetResolver { + + private static final Pattern TRAILING_DASH = Pattern.compile("-$"); + + private final Config config; + + public ArgoCdApplicationTarget resolve(DeploymentContext context, String repoName) { + + String namePrefix = config.getApplication().getNamePrefix() != null ? config.getApplication() + .getNamePrefix() : ""; + String prefix = namePrefix.strip(); + + String applicationName = !prefix.isEmpty() ? (prefix + repoName) : repoName; + String namespace = namePrefix + config.getFeatures().getArgocd().getNamespace(); + String project = "cluster-resources"; + + boolean isOperatorMode = config.getFeatures().getArgocd().getOperator(); + boolean createDestinationNamespace = !isOperatorMode; + + if (context.isMultiTenant()) { + namespace = config.getMultiTenant().getCentralArgocdNamespace(); + project = TRAILING_DASH.matcher(prefix).replaceFirst(""); + } + + return new ArgoCdApplicationTarget(applicationName, namespace, project, createDestinationNamespace); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/deployment/Deployer.java b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/Deployer.java new file mode 100644 index 000000000..d4a2c8cd1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/Deployer.java @@ -0,0 +1,89 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import jakarta.inject.Provider; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.RequiredArgsConstructor; + +import java.nio.file.Path; + +@Singleton +@RequiredArgsConstructor +public class Deployer { + + private final Provider argoCdStrategyProvider; + + @Getter + private final HelmStrategy helmStrategy; + + public void deployFeature( + String repoURL, + String repoName, + String chartOrPath, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + DeploymentStrategy.RepoType repoType, + boolean bootstrapWithHelm, + DeploymentContext context, + RepositoryWorkspace repositoryWorkspace) { + + if (bootstrapWithHelm) { + helmStrategy.deployFeature( + repoURL, + repoName, + chartOrPath, + version, + namespace, + releaseName, + helmValuesPath, + repoType, + context, + repositoryWorkspace + ); + } + + argoCdStrategyProvider.get() + .deployFeature( + repoURL, + repoName, + chartOrPath, + version, + namespace, + releaseName, + helmValuesPath, + repoType, + context, + repositoryWorkspace + ); + } + + public void deployFeature( + String repoURL, + String repoName, + String chartOrPath, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + DeploymentStrategy.RepoType repoType, + DeploymentContext context, + RepositoryWorkspace repositoryWorkspace) { + deployFeature( + repoURL, + repoName, + chartOrPath, + version, + namespace, + releaseName, + helmValuesPath, + repoType, + false, + context, + repositoryWorkspace + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/deployment/DeploymentStrategy.java b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/DeploymentStrategy.java new file mode 100644 index 000000000..9060b6375 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/DeploymentStrategy.java @@ -0,0 +1,50 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; + +import java.nio.file.Path; + +public interface DeploymentStrategy { + + void deployFeature( + String repoURL, + String repoName, + String chartOrPath, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + RepoType repoType, + DeploymentContext context, + RepositoryWorkspace repositoryWorkspace); + + default void deployFeature( + String repoURL, + String repoName, + String chart, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + DeploymentContext context, + RepositoryWorkspace repositoryWorkspace) { + deployFeature( + repoURL, + repoName, + chart, + version, + namespace, + releaseName, + helmValuesPath, + RepoType.HELM, + context, + repositoryWorkspace + ); + } + + enum RepoType { + HELM, + GIT + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/deployment/HelmStrategy.java b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/HelmStrategy.java new file mode 100644 index 000000000..9c398dfc1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/deployment/HelmStrategy.java @@ -0,0 +1,75 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class HelmStrategy implements DeploymentStrategy { + + private final HelmClient helmClient; + + @Override + public void deployFeature( + String repoURL, + String repoName, + String chartOrPath, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + RepoType repoType, + DeploymentContext context, + RepositoryWorkspace repositoryWorkspace) { + deployFeature(repoURL, repoName, chartOrPath, version, namespace, releaseName, helmValuesPath, repoType); + } + + public void deployFeature( + String repoURL, + String repoName, + String chartOrPath, + String version, + String namespace, + String releaseName, + Path helmValuesPath, + RepoType repoType) { + + if (repoType == RepoType.GIT) { + throw new IllegalArgumentException( + "Unable to deploy helm chart via Helm CLI from Git URL, because helm does not support this out of the box.\n" + "Repo URL: " + repoURL); + } + + try { + String valuesText = Files.readString(helmValuesPath); + log.debug( + "Imperatively deploying helm release {} basing on chart {} from {}, version {}, into namespace {}. Using values:\n{}", + releaseName, + chartOrPath, + repoURL, + version, + namespace, + valuesText + ); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + + helmClient.addRepo(repoName, repoURL); + helmClient.upgrade( + releaseName, + repoName + "/" + chartOrPath, + Map.of("namespace", namespace, "version", version, "values", helmValuesPath.toString()) + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/GitRepo.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/GitRepo.java new file mode 100644 index 000000000..1fd53c191 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/GitRepo.java @@ -0,0 +1,512 @@ +package com.cloudogu.gitops.infrastructure.git; + +import com.cloudogu.gitops.cli.Version; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.cloudogu.gitops.utils.jgit.helpers.InsecureCredentialProvider; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.ListBranchCommand; +import org.eclipse.jgit.api.PushCommand; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.lib.Constants; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.revwalk.RevCommit; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.transport.ChainingCredentialsProvider; +import org.eclipse.jgit.transport.CredentialsProvider; +import org.eclipse.jgit.transport.PushResult; +import org.eclipse.jgit.transport.RefSpec; +import org.eclipse.jgit.transport.RemoteRefUpdate; +import org.eclipse.jgit.transport.RemoteRefUpdate.Status; +import org.eclipse.jgit.transport.URIish; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.eclipse.jgit.treewalk.TreeWalk; +import org.eclipse.jgit.treewalk.filter.PathFilter; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.URISyntaxException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; + +@Slf4j +public class GitRepo implements AutoCloseable { + + public static final String NAMESPACE_3RD_PARTY_DEPENDENCIES = "3rd-party-dependencies"; + private static final String GIT_REMOTE_ORIGIN = "origin"; + private static final String DEFAULT_PUSH_REF_SPEC = "HEAD:refs/heads/main"; + private static final String MAIN_BRANCH = "main"; + private static final String REF_HEADS_MAIN = "refs/heads/main"; + private static final String REF_REMOTES_ORIGIN_MAIN = "refs/remotes/origin/main"; + private static final Pattern REFS_HEADS_PREFIX = Pattern.compile("^refs/heads/"); + private static final Pattern REFS_TAGS_PREFIX = Pattern.compile("^refs/tags/"); + + @Getter + @Setter + private GitProvider gitProvider; + private final FileSystemUtils fileSystemUtils; + + @Getter + private final String repoTarget; + private final boolean insecure; + private final String gitName; + private final String gitEmail; + + private Git gitMemoization; + + @Getter + private final String absoluteLocalRepoTmpDir; + + public GitRepo(Config config, GitProvider gitProvider, String repoTarget, FileSystemUtils fileSystemUtils) { + try { + File tmpDir = Files.createTempDirectory("gitops-playground-").toFile(); + tmpDir.deleteOnExit(); + this.absoluteLocalRepoTmpDir = tmpDir.getAbsolutePath(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to create temporary directory", e); + } + this.gitProvider = gitProvider; + this.fileSystemUtils = fileSystemUtils; + + this.repoTarget = repoTarget; + + this.insecure = config.getApplication().getInsecure(); + this.gitName = config.getApplication().getGitName(); + this.gitEmail = config.getApplication().getGitEmail(); + } + + public boolean createRepositoryAndSetPermission(String description, boolean initialize) { + boolean isNewRepo = this.gitProvider.createRepository(repoTarget, description, initialize); + String gitOpsUsername = gitProvider.getGitOpsUsername(); + if (gitOpsUsername != null && !gitOpsUsername.isEmpty()) { + gitProvider.setRepositoryPermission(repoTarget, gitOpsUsername, AccessRole.WRITE, Scope.USER); + } + return isNewRepo; + } + + public void cloneRepo() throws GitAPIException { + String cloneUrl = getGitRepositoryUrl(); + log.debug("Cloning {}, Origin: {}", repoTarget, cloneUrl); + try (Git git = Git.cloneRepository() + .setURI(cloneUrl) + .setDirectory(new File(absoluteLocalRepoTmpDir)) + .setCredentialsProvider(getCredentialProvider()) + .call()) { + // Cloned successfully, try-with-resources closes the git reference + } + } + + public void initLocalRepoIfNeeded() throws GitAPIException { + File localRepoDir = new File(getAbsoluteLocalRepoTmpDir()); + File gitDir = new File(localRepoDir, ".git"); + + if (gitDir.exists()) { + log.debug("Local git repository already initialized at {}", localRepoDir); + return; + } + + log.debug("Initializing local git repository at {}", localRepoDir); + + if (!localRepoDir.exists() && !localRepoDir.mkdirs()) { + log.warn("Failed to create directory {}", localRepoDir); + } + + try (Git git = Git.init().setDirectory(localRepoDir).call()) { + + // Configure the 'origin' remote so init'd repos behave like cloned ones. + // pullRebaseMain() pulls from the remote name 'origin'; without this the + // repo has no remote.origin.url and JGit fails with + // "No value for key remote.origin.url found in configuration". + git.remoteAdd().setName(GIT_REMOTE_ORIGIN).setUri(new URIish(getGitRepositoryUrl())).call(); + } catch (URISyntaxException e) { + throw new IllegalArgumentException("Invalid git repository URL: " + getGitRepositoryUrl(), e); + } + } + + /** + * Commits and pushes to the default {@code main} branch. + */ + public void commitAndPush(String message, String tag) throws GitAPIException { + commitAndPush(message, tag, DEFAULT_PUSH_REF_SPEC); + } + + public void commitAndPush(String commitMessage, String tag, String refSpec) throws GitAPIException { + log.debug("Adding files to {}", repoTarget); + + Git git = getGit(); + ensureLocalMainBranchForInitialCommit(git); + git.add().addFilepattern(".").call(); + + if (git.status().call().hasUncommittedChanges()) { + log.debug("Commiting {}", repoTarget); + + String cleanVersion = Version.NAME.split(",")[0].replace("(", ""); + String committerName = gitName + " - GOP v" + cleanVersion; + + git.commit() + .setSign(false) + .setMessage(commitMessage) + .setAuthor(gitName, gitEmail) + .setCommitter(committerName, gitEmail) + .call(); + + PushCommand pushCommand = createPushCommand(refSpec); + + if (tag != null && !tag.isEmpty()) { + log.debug("Setting tag '{}' on repo: {}", tag, repoTarget); + + // Delete existing tags first to get idempotence + git.tagDelete().setTags(tag).call(); + git.tag().setName(tag).call(); + + pushCommand.setPushTags(); + } + + log.debug("Pushing repo: {}, refSpec: {}", repoTarget, refSpec); + + Iterable pushResults = pushCommand.call(); + validatePushResults(pushResults, repoTarget); + } else { + log.debug("No changes after add, nothing to commit or push on repo: {}", repoTarget); + } + } + + private void ensureLocalMainBranchForInitialCommit(Git git) throws GitAPIException { + try { + Ref localMain = git.getRepository().findRef(REF_HEADS_MAIN); + Ref head = git.getRepository().exactRef(Constants.HEAD); + + if (localMain != null) { + git.checkout() + .setName(MAIN_BRANCH) + .call(); + return; + } + + if (head == null || head.isSymbolic()) { + createUnbornMainBranch(git); + } + } catch (IOException e) { + throw new IllegalStateException("Failed to prepare local main branch for repo '" + repoTarget + "'", e); + } + } + + private static void validatePushResults(Iterable pushResults, String repoTarget) { + for (PushResult result : pushResults) { + for (RemoteRefUpdate update : result.getRemoteUpdates()) { + log.debug( + "Push result for repo '{}': remoteName='{}', status='{}', message='{}'", + repoTarget, + update.getRemoteName(), + update.getStatus(), + update.getMessage() + ); + + if (update.getStatus() != Status.OK && update.getStatus() != Status.UP_TO_DATE) { + throw new IllegalStateException("Push failed for repo '" + repoTarget + "', remoteName='" + update.getRemoteName() + "', status='" + update.getStatus() + "', message='" + update.getMessage() + "'"); + } + } + } + } + + public void commitAndPush(String commitMessage) throws GitAPIException { + commitAndPush(commitMessage, null, DEFAULT_PUSH_REF_SPEC); + } + + /** + * Push all refs, i.e. all tags and branches + */ + public void pushAll(boolean force) throws GitAPIException { + createPushCommand("refs/*:refs/*").setForce(force).call(); + } + + public void pushRef(String ref, boolean force) throws GitAPIException { + pushRef(ref, ref, force); + } + + public void pushRef(String ref, String targetRef, boolean force) throws GitAPIException { + createPushCommand(ref + ":" + targetRef).setForce(force).call(); + } + + /** + * Delete all files in this repository + */ + public void clearRepo() { + fileSystemUtils.deleteFilesExcept(new File(absoluteLocalRepoTmpDir), ".git"); + } + + public void copyDirectoryContents(String srcDir) { + copyDirectoryContents(srcDir, null); + } + + public void copyDirectoryContents(String srcDir, FileFilter fileFilter) { + if (srcDir == null || srcDir.isEmpty()) { + log.warn("Source directory is not defined. Nothing to copy?"); + return; + } + + log.debug("Initializing repo {} from {}", repoTarget, srcDir); + String absoluteSrcDirLocation = new File(srcDir).isAbsolute() ? srcDir : Path.of( + fileSystemUtils.getRootDir(), + srcDir + ) + .toString(); + fileSystemUtils.copyDirectory(absoluteSrcDirLocation, absoluteLocalRepoTmpDir, fileFilter); + } + + public void writeFile(String path, String content) throws IOException { + File file = new File(absoluteLocalRepoTmpDir, path); + fileSystemUtils.createDirectory(file.getParent()); + if (file.isDirectory()) { + throw new java.io.FileNotFoundException(file.getAbsolutePath() + " (Is a directory)"); + } + // Files.writeString creates the file if it doesn't exist yet. + Files.writeString(file.toPath(), content); + } + + public void replaceTemplates(Map parameters) { + try { + new TemplatingEngine().replaceTemplates(new File(absoluteLocalRepoTmpDir), parameters); + } catch (IOException | freemarker.template.TemplateException e) { + throw new RuntimeException("Failed to replace templates in: " + absoluteLocalRepoTmpDir, e); + } + } + + public String getGitRepositoryUrl() { + return this.gitProvider.repoUrl(repoTarget, RepoUrlScope.CLIENT); + } + + public void checkoutRemoteMainIfLocalMainMissing() throws GitAPIException, IOException { + initLocalRepoIfNeeded(); + + Git git = getGit(); + + git.fetch() + .setRemote(GIT_REMOTE_ORIGIN) + .setCredentialsProvider(getCredentialProvider()) + .call(); + + Ref localMain = git.getRepository().findRef(REF_HEADS_MAIN); + + if (localMain != null) { + git.checkout() + .setName(MAIN_BRANCH) + .call(); + return; + } + + Ref remoteMain = git.getRepository().findRef(REF_REMOTES_ORIGIN_MAIN); + + if (remoteMain != null) { + log.debug("Creating local main branch from origin/main for repo '{}'", repoTarget); + + git.checkout() + .setCreateBranch(true) + .setName(MAIN_BRANCH) + .setStartPoint("origin/main") + .call(); + return; + } + + log.debug( + "Remote branch origin/main does not exist for repo '{}'. Creating local main branch for initial GOP bootstrap.", + repoTarget + ); + + createUnbornMainBranch(git); + } + + private void createUnbornMainBranch(Git git) throws IOException { + git.getRepository() + .updateRef(Constants.HEAD, true) + .link(REF_HEADS_MAIN); + } + + public static boolean isCommit(File repoPath, String ref) { + if (ref == null || ref.isEmpty()) { + return false; + } + + return withGitOrFalse( + repoPath, + "checking if ref '" + ref + "' is a commit in repo '" + repoPath + "'", + (Git git) -> resolveIsCommit(git, ref) + ); + } + + private static boolean resolveIsCommit(Git git, String ref) throws IOException, GitAPIException { + // Get all branch and tag names + List allRefs = new ArrayList<>(); + + // Add all branch names (without refs/heads/ prefix) + List branches = git.branchList().call(); + for (Ref branch : branches) { + allRefs.add(REFS_HEADS_PREFIX.matcher(branch.getName()).replaceFirst("")); + } + + // Add all tag names (without refs/tags/ prefix) + List tags = git.tagList().call(); + for (Ref tag : tags) { + allRefs.add(REFS_TAGS_PREFIX.matcher(tag.getName()).replaceFirst("")); + } + + // If the ref matches any branch or tag name, it's not a commit hash + if (allRefs.contains(ref)) { + return false; + } + + // If it's not a branch or tag, try to resolve it as a commit + ObjectId objectId = git.getRepository().resolve(ref); + return objectId != null; + } + + /** + * Checks if a file exists in the repository in some branch. + * + * @param repo the repository path + * @param filename the filename to search for + * @return true if the file exists in some branch, false otherwise + */ + public static boolean existFileInSomeBranch(String repo, String filename) { + File repoPath = new File(repo); + + boolean found = withGitOrFalse( + repoPath, + "checking if file '" + filename + "' exists in repo '" + repoPath + "'", + (Git git) -> resolveExistsInSomeBranch(git, filename) + ); + + if (!found) { + log.debug("File {} not found in repository {}", filename, repoPath); + } + return found; + } + + private static boolean resolveExistsInSomeBranch(Git git, String filename) throws IOException, GitAPIException { + List branches = git.branchList().setListMode(ListBranchCommand.ListMode.ALL).call(); + + for (Ref branch : branches) { + String branchName = branch.getName(); + + ObjectId commitId = git.getRepository().resolve(branchName); + if (commitId != null && branchContainsFile(git, commitId, filename, branchName)) { + return true; + } + } + return false; + } + + private static boolean branchContainsFile( + Git git, + ObjectId commitId, + String filename, + String branchName) throws IOException { + try (RevWalk revWalk = new RevWalk(git.getRepository())) { + RevCommit commit = revWalk.parseCommit(commitId); + try (TreeWalk treeWalk = new TreeWalk(git.getRepository())) { + treeWalk.addTree(commit.getTree()); + treeWalk.setFilter(PathFilter.create(filename)); + + if (treeWalk.next()) { + log.debug("File {} found in branch {}", filename, branchName); + return true; + } + } + } + return false; + } + + public static boolean isTag(File repo, String ref) { + if (ref == null || ref.isEmpty()) { + return false; + } + return withGitOrFalse( + repo, "checking if ref '" + ref + "' is a tag in repo '" + repo + "'", (Git git) -> { + List tags = git.tagList().call(); + for (Ref tag : tags) { + if (tag.getName().endsWith("/" + ref) || tag.getName().equals(ref)) { + return true; + } + } + return false; + } + ); + } + + /** + * Opens the git repository at {@code repoPath} and runs {@code operation} against it, returning + * its result. If the repository can't be opened or the operation throws, logs a warning with + * {@code errorContext} and returns {@code false}. Centralizes the try-with-resources/catch + * pattern shared by the static ref-inspection helpers above. + */ + private static boolean withGitOrFalse(File repoPath, String errorContext, GitBooleanOperation operation) { + try (Git git = Git.open(repoPath)) { + return operation.execute(git); + } catch (IOException | GitAPIException e) { + log.warn("Error {}: {}", errorContext, e.getMessage()); + return false; + } + } + + @FunctionalInterface + private interface GitBooleanOperation { + boolean execute(Git git) throws IOException, GitAPIException; + } + + private PushCommand createPushCommand(String refSpec) { + return getGit().push() + .setRemote(getGitRepositoryUrl()) + .setRefSpecs(new RefSpec(refSpec)) + .setCredentialsProvider(getCredentialProvider()); + } + + private Git getGit() { + if (gitMemoization != null) { + return gitMemoization; + } + + try { + gitMemoization = Git.open(new File(absoluteLocalRepoTmpDir)); + return gitMemoization; + } catch (IOException e) { + throw new UncheckedIOException("Failed to open git repository at: " + absoluteLocalRepoTmpDir, e); + } + } + + private CredentialsProvider getCredentialProvider() { + Credentials auth = this.gitProvider.getCredentials(); + UsernamePasswordCredentialsProvider passwordAuthentication = new UsernamePasswordCredentialsProvider( + auth.getUsername(), + auth.getPassword() + ); + return insecure ? new ChainingCredentialsProvider( + new InsecureCredentialProvider(), + passwordAuthentication + ) : passwordAuthentication; + } + + @Override + public void close() { + if (gitMemoization != null) { + gitMemoization.close(); + gitMemoization = null; + } + FileSystemUtils.deleteDir(absoluteLocalRepoTmpDir); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/GitRepoFactory.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/GitRepoFactory.java new file mode 100644 index 000000000..09ca5a17d --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/GitRepoFactory.java @@ -0,0 +1,20 @@ +package com.cloudogu.gitops.infrastructure.git; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.utils.FileSystemUtils; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class GitRepoFactory { + protected final Config config; + protected final FileSystemUtils fileSystemUtils; + + public GitRepo create(String repoTarget, GitProvider gitProvider) { + // GitRepo receives the final repository target and does not apply naming rules itself. + String prefixedRepoTarget = config.getApplication().getNamePrefix() + repoTarget; + return new GitRepo(config, gitProvider, prefixedRepoTarget, fileSystemUtils); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/AccessRole.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/AccessRole.java new file mode 100644 index 000000000..2c690eb89 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/AccessRole.java @@ -0,0 +1,9 @@ +package com.cloudogu.gitops.infrastructure.git.providers; + +public enum AccessRole { + READ, + WRITE, + MAINTAIN, + ADMIN, + OWNER +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/GitProvider.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/GitProvider.java new file mode 100644 index 000000000..8ffba8405 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/GitProvider.java @@ -0,0 +1,48 @@ +package com.cloudogu.gitops.infrastructure.git.providers; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.utils.Tuple; + +import java.net.URI; + +public interface GitProvider { + + int REPO_TARGET_SEGMENT_COUNT = 2; + + /** + * Splits a "namespace/repoName" repo target into its namespace and name segments. Shared by + * providers that address repositories via a flat "namespace/name" string. + */ + static Tuple splitRepoTarget(String repoTarget) { + String[] parts = repoTarget.split("/", REPO_TARGET_SEGMENT_COUNT); + return new Tuple<>(parts[0], parts[1]); + } + + default boolean createRepository(String repoTarget, String description) { + return createRepository(repoTarget, description, true); + } + + boolean createRepository(String repoTarget, String description, boolean initialize); + + void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope); + + default String repoUrl(String repoTarget) { + return repoUrl(repoTarget, RepoUrlScope.IN_CLUSTER); + } + + String repoUrl(String repoTarget, RepoUrlScope scope); + + String repoPrefix(); + + Credentials getCredentials(); + + URI prometheusMetricsEndpoint(); + + String getUrl(); + + String getProtocol(); + + String getHost(); + + String getGitOpsUsername(); +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/RepoUrlScope.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/RepoUrlScope.java new file mode 100644 index 000000000..fe02e1981 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/RepoUrlScope.java @@ -0,0 +1,15 @@ +package com.cloudogu.gitops.infrastructure.git.providers; + +/** + * IN_CLUSTER: URLs intended for workloads running inside the Kubernetes cluster (e.g., ArgoCD, + * Jobs, in-cluster automation). + * + *

CLIENT : URLs intended for interactive or CI clients performing push/clone operations, + * regardless of their location. If the application itself runs inside Kubernetes, the Service DNS + * is used; otherwise, NodePort (for internal installations) or externalBase (for external ones) is + * selected automatically. + */ +public enum RepoUrlScope { + IN_CLUSTER, + CLIENT +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/Scope.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/Scope.java new file mode 100644 index 000000000..3d28946e8 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/Scope.java @@ -0,0 +1,6 @@ +package com.cloudogu.gitops.infrastructure.git.providers; + +public enum Scope { + USER, + GROUP +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/gitlab/GitlabProvider.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/gitlab/GitlabProvider.java new file mode 100644 index 000000000..775910de2 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/gitlab/GitlabProvider.java @@ -0,0 +1,322 @@ +package com.cloudogu.gitops.infrastructure.git.providers.gitlab; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.GitlabConfig; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import com.cloudogu.gitops.utils.Tuple; +import lombok.extern.slf4j.Slf4j; +import org.gitlab4j.api.GitLabApi; +import org.gitlab4j.api.GitLabApiException; +import org.gitlab4j.api.GroupApi; +import org.gitlab4j.api.models.AccessLevel; +import org.gitlab4j.api.models.Group; +import org.gitlab4j.api.models.Project; +import org.gitlab4j.api.models.Visibility; + +import java.net.URI; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.logging.Level; +import java.util.regex.Pattern; + +@Slf4j +public class GitlabProvider implements GitProvider { + + // GitLab API paths always use '/', regardless of the host OS + private static final String PATH_SEPARATOR = "/"; + private static final String NOT_FOUND_SUFFIX = "' not found"; + private static final Pattern NUMERIC = Pattern.compile("\\d+"); + private static final Pattern LEADING_SLASHES = Pattern.compile("^/+"); + private static final int HTTP_BAD_REQUEST = 400; + private static final int HTTP_CONFLICT = 409; + private static final int HTTP_NOT_FOUND = 404; + + private final String namePrefix; + private final GitLabApi api; + private final GitlabConfig gitlabConfig; + private final Credentials runtimeCredentials; + private Group parentGroupCache; + + public GitlabProvider(GitlabConfig gitlabConfig, Credentials runtimeCredentials, String namePrefix) { + this.gitlabConfig = gitlabConfig; + this.runtimeCredentials = Objects.requireNonNull(runtimeCredentials, "Missing gitlab credentials"); + this.namePrefix = namePrefix; + + String url = Objects.requireNonNull(gitlabConfig.getUrl(), "Missing gitlab url in config.scm.gitlab.url") + .trim(); + String pat = Objects.requireNonNull(runtimeCredentials.getPassword(), "Missing gitlab token").trim(); + + this.api = new GitLabApi(url, pat); + this.api.enableRequestResponseLogging(Level.ALL); + } + + @Override + public boolean createRepository(String repoTarget, String description, boolean initialize) { + Tuple target = GitProvider.splitRepoTarget(repoTarget); + String repoNamespace = target.getFirst(); + String repoName = target.getSecond(); + + Group parent = parentGroup(); + String repoNamespacePath = repoNamespace.toLowerCase(Locale.ROOT); + String projectPath = repoName.toLowerCase(Locale.ROOT); + + String fullProjectPath = parent.getFullPath() + PATH_SEPARATOR + repoNamespacePath + PATH_SEPARATOR + projectPath; + + if (findProject(fullProjectPath).isPresent()) { + log.info("GitLab project already exists: " + fullProjectPath); + return false; + } + + long subgroupId = ensureSubgroupUnderParentId(parent, repoNamespacePath); + Project project = new Project().withName(repoName) + .withPath(projectPath) + .withDescription(description != null ? description : "") + .withIssuesEnabled(false) + .withMergeRequestsEnabled(false) + .withWikiEnabled(false) + .withSnippetsEnabled(false) + .withNamespaceId(subgroupId) + .withInitializeWithReadme(initialize); + project.setVisibility(toVisibility(gitlabConfig.getDefaultVisibility())); + + try { + Project created = api.getProjectApi().createProject(project); + log.info("Created GitLab project " + created.getPathWithNamespace() + " (id=" + created.getId() + ")"); + return true; + } catch (GitLabApiException e) { + throw new RuntimeException("Failed to create GitLab project", e); + } + } + + @Override + public void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { + String fullPath = resolveFullPath(repoTarget); + Project project = findProjectOrThrow(fullPath); + AccessLevel level = toAccessLevel(role, scope); + try { + if (scope == Scope.GROUP) { + Group group = api.getGroupApi() + .getGroups(principal) + .stream() + .filter(candidateGroup -> principal.equals(candidateGroup.getFullPath()) || principal.equals( + candidateGroup.getPath()) || principal.equals(candidateGroup.getName())) + .findFirst() + .orElseThrow(() -> new IllegalArgumentException("Group '" + principal + NOT_FOUND_SUFFIX)); + api.getProjectApi().shareProject(project.getId(), group.getId(), level, null); + } else { + org.gitlab4j.api.models.User user = api.getUserApi() + .findUsers(principal) + .stream() + .filter(candidateUser -> principal.equals(candidateUser.getUsername()) || principal.equals( + candidateUser.getEmail())) + .findFirst() + .orElseThrow(() -> new IllegalArgumentException("User '" + principal + NOT_FOUND_SUFFIX)); + api.getProjectApi().addMember(project.getId(), user.getId(), level); + } + } catch (GitLabApiException e) { + throw new RuntimeException("Failed to set repository permission", e); + } + } + + @Override + public String repoUrl(String repoTarget, RepoUrlScope scope) { + String base = gitlabConfig.getUrl().strip(); + return base + "/" + parentFullPath() + "/" + repoTarget + ".git"; + } + + @Override + public String repoPrefix() { + String base = gitlabConfig.getUrl().strip(); + String prefix = (namePrefix != null ? namePrefix : "").strip(); + return base + "/" + parentFullPath() + "/" + prefix; + } + + @Override + public Credentials getCredentials() { + return this.runtimeCredentials; + } + + @Override + public String getProtocol() { + return gitlabConfig.getUrl(); + } + + @Override + public String getHost() { + return gitlabConfig.getUrl(); + } + + @Override + public String getGitOpsUsername() { + return gitlabConfig.getGitOpsUsername(); + } + + @Override + public String getUrl() { + return this.gitlabConfig.getUrl(); + } + + /** + * Prometheus integration is only required for SCM-Manager. GitLab provides its own built-in + * Prometheus metrics, so we don't expose an endpoint here. + */ + @Override + public URI prometheusMetricsEndpoint() { + return null; + } + + private Group parentGroup() { + if (parentGroupCache != null) { + return parentGroupCache; + } + + String raw = gitlabConfig.getParentGroupId(); + if (raw != null) { + raw = raw.trim(); + } + if (raw == null || raw.isEmpty()) { + throw new IllegalArgumentException("--gitlab-group-id is required"); + } + + boolean isNumeric = NUMERIC.matcher(raw).matches(); + + try { + GroupApi groupApi = api.getGroupApi(); + parentGroupCache = isNumeric ? groupApi.getGroup(Long.parseLong(raw)) : groupApi.getGroup(LEADING_SLASHES.matcher( + raw) + .replaceFirst( + "")); + return parentGroupCache; + } catch (GitLabApiException e) { + throw new RuntimeException("Failed to get parent group: " + raw, e); + } + } + + private String parentFullPath() { + return parentGroup().getFullPath(); + } + + /** + * Ensure a single-level subgroup exists under 'parent'; return its namespace (group) ID. + */ + private long ensureSubgroupUnderParentId(Group parent, String segPath) { + Group existing = findDirectSubgroupByPath(parent.getId(), segPath); + if (existing != null) { + return existing.getId(); + } + + Project collision = findDirectProjectByPath(parent.getId(), segPath); + if (collision != null) { + throw new IllegalStateException("Cannot create subgroup '" + segPath + "' under '" + parent.getFullPath() + "': " + "a project with that path already exists at '" + parent.getFullPath() + "/" + segPath + "'. " + "Rename/transfer the project first or choose a different subgroup name."); + } + + Group toCreate = new Group().withName(segPath).withPath(segPath).withParentId(parent.getId()); + + try { + Group created = api.getGroupApi().addGroup(toCreate); + log.info("Created group {}", created.getFullPath()); + return created.getId(); + } catch (GitLabApiException e) { + if (e.getHttpStatus() == HTTP_BAD_REQUEST || e.getHttpStatus() == HTTP_CONFLICT) { + Group retry = findDirectSubgroupByPath(parent.getId(), segPath); + if (retry != null) { + return retry.getId(); + } + } + Map> ve = e.hasValidationErrors() ? e.getValidationErrors() : null; + log.error( + "addGroup failed (parent={}, segPath={}, status={}, message={}, validationErrors={})", + parent.getFullPath(), + segPath, + e.getHttpStatus(), + e.getMessage(), + ve + ); + throw new RuntimeException("Failed to add GitLab group", e); + } + } + + /** + * Find a direct subgroup of 'parentId' with the exact path . + */ + private Group findDirectSubgroupByPath(Long parentId, String segPath) { + try { + List subGroups = api.getGroupApi().getSubGroups(parentId); + if (subGroups == null) { + return null; + } + return subGroups.stream().filter(subGroup -> segPath.equals(subGroup.getPath())).findFirst().orElse(null); + } catch (GitLabApiException e) { + throw new RuntimeException("Failed to list subgroups of group " + parentId, e); + } + } + + /** + * Find a direct project of 'parentId' with the exact path . + */ + private Project findDirectProjectByPath(Long parentId, String path) { + try { + List projects = api.getGroupApi().getProjects(parentId); + if (projects == null) { + return null; + } + return projects.stream().filter(project -> path.equals(project.getPath())).findFirst().orElse(null); + } catch (GitLabApiException e) { + throw new RuntimeException("Failed to list projects of group " + parentId, e); + } + } + + // ---- Helpers ---- + private Optional findProject(String fullPath) { + try { + return Optional.ofNullable(api.getProjectApi().getProject(fullPath)); + } catch (GitLabApiException e) { + if (e.getHttpStatus() == HTTP_NOT_FOUND) { + return Optional.empty(); + } + throw new RuntimeException("Failed to look up GitLab project: " + fullPath, e); + } + } + + private Project findProjectOrThrow(String fullPath) { + return findProject(fullPath).orElseThrow(() -> new IllegalStateException("GitLab project '" + fullPath + NOT_FOUND_SUFFIX)); + } + + private String resolveFullPath(String repoTarget) { + if (gitlabConfig.getParentGroupId() == null) { + throw new IllegalStateException("gitlab.parentGroup is not set"); + } + Tuple target = GitProvider.splitRepoTarget(repoTarget); + return parentGroup().getFullPath() + "/" + target.getFirst().toLowerCase(Locale.ROOT) + "/" + target.getSecond() + .toLowerCase( + Locale.ROOT); + } + + private static Visibility toVisibility(String s) { + if (s == null) { + s = "private"; + } + return switch (s.toLowerCase(Locale.ROOT)) { + case "public" -> Visibility.PUBLIC; + case "internal" -> Visibility.INTERNAL; + default -> Visibility.PRIVATE; + }; + } + + // provider-agnostic AccessRole → GitLab AccessLevel + private static AccessLevel toAccessLevel(AccessRole role, Scope scope) { + return switch (role) { + case READ -> AccessLevel.REPORTER; + case WRITE -> AccessLevel.DEVELOPER; + case MAINTAIN, ADMIN -> AccessLevel.MAINTAINER; + case OWNER -> (scope == Scope.GROUP) ? AccessLevel.OWNER : AccessLevel.MAINTAINER; + default -> throw new IllegalArgumentException("Unknown role: " + role); + }; + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/Permission.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/Permission.java new file mode 100644 index 000000000..19f482d05 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/Permission.java @@ -0,0 +1,33 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager; + +import java.util.ArrayList; +import java.util.List; + +public record Permission( + String name, + + Role role, + + boolean groupPermission, + + List verbs +) { + + public Permission(String name, Role role) { + this(name, role, false, new ArrayList<>()); + } + + public Permission(String name, Role role, boolean groupPermission) { + this(name, role, groupPermission, new ArrayList<>()); + } + + public Permission { + verbs = verbs != null ? List.copyOf(verbs) : List.of(); + } + + public enum Role { + READ, + WRITE, + OWNER + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerProvider.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerProvider.java new file mode 100644 index 000000000..45ab638c7 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerProvider.java @@ -0,0 +1,182 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.ScmManagerConfig; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.NetworkingUtils; +import com.cloudogu.gitops.utils.Tuple; +import lombok.extern.slf4j.Slf4j; +import retrofit2.Response; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.URI; + +@Slf4j +public class ScmManagerProvider implements GitProvider { + + private static final int HTTP_CREATED = 201; + private static final int HTTP_CONFLICT = 409; + + private ScmManagerUrlResolver urls; + private ScmManagerApiClient apiClient; + private final ScmManagerConfig scmmConfig; + private final Credentials runtimeCredentials; + + private final boolean insecure; + + public ScmManagerProvider( + ScmManagerConfig scmmConfig, + Credentials runtimeCredentials, + K8sClient k8sClient, + NetworkingUtils networkingUtils, + String repositoryNamePrefix, + boolean runningInsideK8s, + boolean insecure, + String servicePrefix) { + this.scmmConfig = scmmConfig; + this.runtimeCredentials = runtimeCredentials; + this.insecure = insecure; + this.urls = new ScmManagerUrlResolver( + scmmConfig, + k8sClient, + networkingUtils, + repositoryNamePrefix, + runningInsideK8s, + servicePrefix + ); + } + + public ScmManagerConfig getScmmConfig() { + return scmmConfig; + } + + public ScmManagerApiClient getApiClient() { + if (this.apiClient == null) { + this.apiClient = new ScmManagerApiClient( + this.urls.clientApiBase().toString(), + this.runtimeCredentials, + insecure + ); + } + + return this.apiClient; + } + + @Override + public boolean createRepository(String repoTarget, String description, boolean initialize) { + Tuple target = GitProvider.splitRepoTarget(repoTarget); + String repoNamespace = target.getFirst(); + String repoName = target.getSecond(); + Repository repo = new Repository(repoNamespace, repoName, description != null ? description : ""); + + try { + Response response = getApiClient().repositoryApi().create(repo, initialize).execute(); + return handle201or409(response, "Repository " + repoNamespace + "/" + repoName); + } catch (IOException e) { + throw new UncheckedIOException("Failed to create repository " + repoTarget, e); + } + } + + @Override + public void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { + Tuple target = GitProvider.splitRepoTarget(repoTarget); + String repoNamespace = target.getFirst(); + String repoName = target.getSecond(); + + boolean isGroup = (scope == Scope.GROUP); + Permission.Role scmManagerRole = mapToScmManager(role); + Permission permission = new Permission(principal, scmManagerRole, isGroup); + + try { + Response response = getApiClient().repositoryApi() + .createPermission(repoNamespace, repoName, permission) + .execute(); + + handle201or409(response, "Permission on " + repoNamespace + "/" + repoName); + } catch (IOException e) { + throw new UncheckedIOException("Failed to set permission on repository " + repoTarget, e); + } + } + + @Override + public Credentials getCredentials() { + return this.runtimeCredentials; + } + + @Override + public String getGitOpsUsername() { + return scmmConfig.getGitOpsUsername(); + } + + @Override + public String getUrl() { + return urls.inClusterBase().toString(); + } + + @Override + public String repoPrefix() { + return urls.inClusterRepoPrefix(); + } + + @Override + public String repoUrl(String repoTarget, RepoUrlScope scope) { + return switch (scope) { + case CLIENT -> urls.clientRepoUrl(repoTarget); + case IN_CLUSTER -> urls.inClusterRepoUrl(repoTarget); + }; + } + + @Override + public String getProtocol() { + return urls.inClusterBase().getScheme(); + } + + @Override + public String getHost() { + return urls.inClusterBase().getHost(); + } + + @Override + public URI prometheusMetricsEndpoint() { + return urls.prometheusEndpoint(); + } + + private static Permission.Role mapToScmManager(AccessRole role) { + switch (role) { + case READ: + return Permission.Role.READ; + case WRITE: + return Permission.Role.WRITE; + case MAINTAIN: + log.warn("SCM-Manager: Mapping MAINTAIN to WRITE"); + return Permission.Role.WRITE; + case ADMIN: + return Permission.Role.OWNER; + case OWNER: + return Permission.Role.OWNER; + default: + throw new IllegalArgumentException("Unsupported access role: " + role); + } + } + + private static boolean handle201or409(Response response, String resourceName) { + if (response.code() == HTTP_CREATED) { + log.debug("{} created successfully", resourceName); + return true; + } + + if (response.code() == HTTP_CONFLICT) { + log.debug("{} already exists", resourceName); + return false; + } + + throw new IllegalStateException("Failed to create " + resourceName + ". HTTP Status: " + response.code() + " - " + response.message()); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolver.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolver.java new file mode 100644 index 000000000..5fb577933 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolver.java @@ -0,0 +1,198 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager; + +import com.cloudogu.gitops.config.scm.util.ScmManagerConfig; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.NetworkingUtils; +import lombok.extern.slf4j.Slf4j; + +import java.net.URI; +import java.net.URISyntaxException; + +@Slf4j +public class ScmManagerUrlResolver { + + private static final String HTTP_PREFIX = "http://"; + private static final String RELEASE_NAME = "scmm"; + private static final String REPO_ROOT = "repo"; + + private final ScmManagerConfig scmmConfig; + private final K8sClient k8s; + private final NetworkingUtils net; + private final String repositoryNamePrefix; + private final boolean runningInsideK8s; + private final String servicePrefix; + + private URI cachedClusterBind; + + public ScmManagerUrlResolver( + ScmManagerConfig scmmConfig, + K8sClient k8s, + NetworkingUtils net, + String repositoryNamePrefix, + boolean runningInsideK8s) { + this(scmmConfig, k8s, net, repositoryNamePrefix, runningInsideK8s, ""); + } + + public ScmManagerUrlResolver( + ScmManagerConfig scmmConfig, + K8sClient k8s, + NetworkingUtils net, + String repositoryNamePrefix, + boolean runningInsideK8s, + String servicePrefix) { + this.scmmConfig = scmmConfig; + this.k8s = k8s; + this.net = net; + this.repositoryNamePrefix = repositoryNamePrefix != null ? repositoryNamePrefix : ""; + this.runningInsideK8s = runningInsideK8s; + this.servicePrefix = servicePrefix != null ? servicePrefix : ""; + } + + // ---------- Public API used by ScmManager ---------- + + /** + * Client base …/scm (no trailing slash) + */ + public URI clientBase() { + return noTrailSlash(ensureScm(clientBaseRaw())); + } + + /** + * Client API base …/scm/api/ + */ + public URI clientApiBase() { + return withSlash(clientBase()).resolve("api/"); + } + + /** + * Client repo base …/scm/repo (no trailing slash) + */ + public URI clientRepoBase() { + return noTrailSlash(withSlash(clientBase()).resolve(REPO_ROOT + "/")); + } + + /** + * In-cluster base …/scm (no trailing slash) + */ + public URI inClusterBase() { + return noTrailSlash(ensureScm(inClusterBaseRaw())); + } + + /** + * In-cluster repo prefix …/scm/repo/[] + */ + public String inClusterRepoPrefix() { + String prefix = repositoryNamePrefix.trim(); + URI base = withSlash(inClusterBase()); + URI url = withSlash(base.resolve(REPO_ROOT)); + + return URI.create(url.toString() + prefix).toString(); + } + + /** + * In-cluster repo URL …/scm/repo// + */ + public String inClusterRepoUrl(String repoTarget) { + String repo = repoTarget.trim(); + return noTrailSlash(withSlash(inClusterBase()).resolve(REPO_ROOT + "/" + repo + "/")).toString(); + } + + /** + * Client repo URL …/scm/repo// (no trailing slash) + */ + public String clientRepoUrl(String repoTarget) { + String repo = repoTarget.trim(); + return noTrailSlash(withSlash(clientRepoBase()).resolve(repo + "/")).toString(); + } + + /** + * …/scm/api/v2/metrics/prometheus + */ + public URI prometheusEndpoint() { + return withSlash(clientBase()).resolve("api/v2/metrics/prometheus"); + } + + // ---------- Base resolution ---------- + + private URI clientBaseRaw() { + if (scmmConfig.getInternal()) { + return runningInsideK8s ? serviceDnsBase() : nodePortBase(); + } + return externalBase(); + } + + private URI inClusterBaseRaw() { + return scmmConfig.getInternal() ? serviceDnsBase() : externalBase(); + } + + private URI serviceDnsBase() { + return URI.create(HTTP_PREFIX + serviceName() + "." + serviceNamespace() + ".svc.cluster.local"); + } + + private URI externalBase() { + String url = scmmConfig.getUrl() != null ? scmmConfig.getUrl().trim() : ""; + if (!url.isEmpty()) { + return URI.create(url); + } + + String ingress = scmmConfig.getIngress() != null ? scmmConfig.getIngress().trim() : ""; + if (!ingress.isEmpty()) { + return URI.create(HTTP_PREFIX + ingress); + } + throw new IllegalArgumentException("Either scmm.url or scmm.ingress must be set when internal=false"); + } + + private URI nodePortBase() { + if (cachedClusterBind != null) { + return cachedClusterBind; + } + + String port = k8s.waitForNodePort(serviceName(), serviceNamespace()); + String host = net.findClusterBindAddress(); + try { + cachedClusterBind = new URI(HTTP_PREFIX + host + ":" + port); + } catch (URISyntaxException e) { + throw new IllegalStateException("Failed to construct ScmManager node port base URI", e); + } + return cachedClusterBind; + } + + private String serviceName() { + String prefix = servicePrefix.trim(); + + if (!prefix.isEmpty()) { + return prefix + RELEASE_NAME; + } + + return RELEASE_NAME; + } + + private String serviceNamespace() { + String namespace = scmmConfig.getNamespace() != null ? scmmConfig.getNamespace().trim() : "scm-manager"; + String prefix = servicePrefix.trim(); + + if (!prefix.isEmpty() && !namespace.startsWith(prefix)) { + return prefix + namespace; + } + + return namespace; + } + + // ---------- Helpers ---------- + + private static URI ensureScm(URI u) { + URI us = withSlash(u); + String path = us.getPath() != null ? us.getPath() : ""; + return path.endsWith("/scm/") ? us : us.resolve("scm/"); + } + + private static URI withSlash(URI u) { + String s = u.toString(); + return s.endsWith("/") ? u : URI.create(s + "/"); + } + + private static URI noTrailSlash(URI u) { + String s = u.toString(); + return s.endsWith("/") ? URI.create(s.substring(0, s.length() - 1)) : u; + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/AuthorizationInterceptor.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/AuthorizationInterceptor.java new file mode 100644 index 000000000..adae575b5 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/AuthorizationInterceptor.java @@ -0,0 +1,29 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import lombok.RequiredArgsConstructor; +import okhttp3.Credentials; +import okhttp3.Interceptor; +import okhttp3.Request; +import okhttp3.Response; +import org.jetbrains.annotations.NotNull; + +import java.io.IOException; + +/** + * OkHttp interceptor that adds HTTP basic auth credentials to every SCM-Manager request. + */ +@RequiredArgsConstructor +public class AuthorizationInterceptor implements Interceptor { + private final String username; + private final String password; + + @Override + public Response intercept(@NotNull Chain chain) throws IOException { + Request newRequest = chain.request() + .newBuilder() + .header("Authorization", Credentials.basic(username, password)) + .build(); + + return chain.proceed(newRequest); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/PluginApi.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/PluginApi.java new file mode 100644 index 000000000..3434e6d15 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/PluginApi.java @@ -0,0 +1,37 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import retrofit2.Call; +import retrofit2.http.Body; +import retrofit2.http.Headers; +import retrofit2.http.POST; +import retrofit2.http.PUT; +import retrofit2.http.Path; +import retrofit2.http.Query; + +import java.util.Map; + +/** + * Retrofit client for the SCM-Manager plugin REST API. + */ +public interface PluginApi { + /** + * Installs the given plugin from the list of available plugins, optionally restarting SCM-Manager + * afterwards. + * + * @param name name of the plugin to install + * @param restart whether SCM-Manager should restart after the installation + * @return call that completes when the installation was triggered + */ + @POST("v2/plugins/available/{name}/install") + Call install(@Path("name") String name, @Query("restart") Boolean restart); + + /** + * Writes the configuration of the SCM-Manager Jenkins plugin. + * + * @param config Jenkins plugin configuration to store + * @return call that completes when the configuration was written + */ + @PUT("v2/config/jenkins/") + @Headers("Content-Type: application/json") + Call configureJenkinsPlugin(@Body Map config); +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/Repository.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/Repository.java new file mode 100644 index 000000000..081333f2b --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/Repository.java @@ -0,0 +1,71 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import lombok.Getter; +import lombok.ToString; + +/** + * Request payload describing an SCM-Manager repository to be created via {@link RepositoryApi}. + */ +@Getter +@ToString +public class Repository { + private final String name; + private final String namespace; + private final String type; + private final String contact; + private final String description; + + /** + * Creates a git repository payload without description and contact. + * + * @param namespace SCM-Manager namespace of the repository + * @param name name of the repository + */ + public Repository(String namespace, String name) { + this(namespace, name, null, null, "git"); + } + + /** + * Creates a git repository payload without contact. + * + * @param namespace SCM-Manager namespace of the repository + * @param name name of the repository + * @param description free-text description of the repository + */ + public Repository(String namespace, String name, String description) { + this(namespace, name, description, null, "git"); + } + + /** + * Creates a git repository payload. + * + * @param namespace SCM-Manager namespace of the repository + * @param name name of the repository + * @param description free-text description of the repository + * @param contact contact mail address shown for the repository + */ + public Repository(String namespace, String name, String description, String contact) { + this(namespace, name, description, contact, "git"); + } + + /** + * Creates a repository payload. + * + * @param namespace SCM-Manager namespace of the repository + * @param name name of the repository + * @param description free-text description of the repository + * @param contact contact mail address shown for the repository + * @param type repository type; defaults to {@code git} when {@code null} + */ + public Repository(String namespace, String name, String description, String contact, String type) { + this.namespace = namespace; + this.name = name; + this.type = type != null ? type : "git"; + this.contact = contact; + this.description = description; + } + + public String getFullRepoName() { + return namespace + "/" + name; + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/RepositoryApi.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/RepositoryApi.java new file mode 100644 index 000000000..af6977c69 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/RepositoryApi.java @@ -0,0 +1,51 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.Permission; +import retrofit2.Call; +import retrofit2.http.Body; +import retrofit2.http.DELETE; +import retrofit2.http.Headers; +import retrofit2.http.POST; +import retrofit2.http.Path; +import retrofit2.http.Query; + +/** + * Retrofit client for the SCM-Manager repository REST API. + */ +public interface RepositoryApi { + /** + * Deletes the repository identified by namespace and name. + * + * @param namespace SCM-Manager namespace of the repository + * @param name name of the repository + * @return call that completes when the repository was deleted + */ + @DELETE("v2/repositories/{namespace}/{name}") + Call delete(@Path("namespace") String namespace, @Path("name") String name); + + /** + * Creates a new repository, optionally initializing it with an initial branch. + * + * @param repository payload describing the repository to create + * @param initialize whether the repository should be initialized with an initial branch + * @return call that completes when the repository was created + */ + @POST("v2/repositories/") + @Headers("Content-Type: application/vnd.scmm-repository+json;v=2") + Call create(@Body Repository repository, @Query("initialize") boolean initialize); + + /** + * Adds a permission entry to the repository identified by namespace and name. + * + * @param namespace SCM-Manager namespace of the repository + * @param name name of the repository + * @param permission permission entry to add + * @return call that completes when the permission was created + */ + @POST("v2/repositories/{namespace}/{name}/permissions/") + @Headers("Content-Type: application/vnd.scmm-repositoryPermission+json") + Call createPermission( + @Path("namespace") String namespace, + @Path("name") String name, + @Body Permission permission); +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApi.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApi.java new file mode 100644 index 000000000..0dd6ef8c5 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApi.java @@ -0,0 +1,33 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import retrofit2.Call; +import retrofit2.http.Body; +import retrofit2.http.GET; +import retrofit2.http.Headers; +import retrofit2.http.PUT; + +import java.util.Map; + +/** + * Retrofit client for the general SCM-Manager REST API (availability check, global config). + */ +public interface ScmManagerApi { + + /** + * Probes the API root to check whether SCM-Manager is up and reachable. + * + * @return call that succeeds when SCM-Manager is available + */ + @GET("v2") + Call checkScmmAvailable(); + + /** + * Writes the global SCM-Manager configuration. + * + * @param config global configuration to store + * @return call that completes when the configuration was written + */ + @PUT("v2/config") + @Headers("Content-Type: application/vnd.scmm-config+json;v=2") + Call setConfig(@Body Map config); +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApiClient.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApiClient.java new file mode 100644 index 000000000..bf58f5efa --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerApiClient.java @@ -0,0 +1,114 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.dependencyinjection.HttpClientFactory; +import lombok.extern.slf4j.Slf4j; +import okhttp3.OkHttpClient; +import retrofit2.Call; +import retrofit2.Response; +import retrofit2.Retrofit; +import retrofit2.converter.jackson.JacksonConverterFactory; + +/** + * Parent class for all SCMM Apis that lazily creates the APIs + */ +@Slf4j +public class ScmManagerApiClient { + + private static final int HTTP_CREATED = 201; + private static final int HTTP_CONFLICT = 409; + + private final OkHttpClient okHttpClient; + private final String url; + + /** + * Creates a client for the SCM-Manager REST API. + * + * @param url base URL of the SCM-Manager REST API + * @param credentials basic auth credentials used for every request + * @param isInsecure whether TLS certificate and hostname verification should be disabled + */ + public ScmManagerApiClient(String url, Credentials credentials, Boolean isInsecure) { + this.url = url; + this.okHttpClient = HttpClientFactory.buildOkHttpClient(credentials, isInsecure); + } + + /** + * Creates a {@link UsersApi} bound to this client's base URL and credentials. + * + * @return a users API client + */ + public UsersApi usersApi() { + return retrofit().create(UsersApi.class); + } + + /** + * Creates a {@link RepositoryApi} bound to this client's base URL and credentials. + * + * @return a repository API client + */ + public RepositoryApi repositoryApi() { + return retrofit().create(RepositoryApi.class); + } + + /** + * Creates a {@link ScmManagerApi} bound to this client's base URL and credentials. + * + * @return a general API client + */ + public ScmManagerApi generalApi() { + return retrofit().create(ScmManagerApi.class); + } + + /** + * Creates a {@link PluginApi} bound to this client's base URL and credentials. + * + * @return a plugin API client + */ + public PluginApi pluginApi() { + return retrofit().create(PluginApi.class); + } + + /** + * Executes the API call without additional context, see {@link #handleApiResponse(Call, String)}. + * + * @param apiCall the call to execute + */ + public static void handleApiResponse(Call apiCall) { + handleApiResponse(apiCall, ""); + } + + /** + * Executes the API call and throws when the response is neither successful nor an acceptable + * status (201 Created, 409 Conflict for already existing resources). + * + * @param apiCall the call to execute + * @param additionalMessage extra context appended to the error message on failure + */ + public static void handleApiResponse(Call apiCall, String additionalMessage) { + try { + Response response = apiCall.execute(); + + if (!response.isSuccessful() && response.code() != HTTP_CONFLICT && response.code() != HTTP_CREATED) { + String errorMessage = "API call failed!'. HTTP Status: " + response.code() + " - " + response.message(); + if (additionalMessage != null && !additionalMessage.isEmpty()) { + errorMessage += " Additional Info: " + additionalMessage; + } + log.error(errorMessage); + throw new IllegalStateException(errorMessage); + } else { + log.debug("Successfully completed " + apiCall); + } + } catch (Exception e) { + String errorMessage = "Error executing API: " + e.getMessage(); + log.error(errorMessage, e); + throw new RuntimeException(errorMessage, e); + } + } + + protected Retrofit retrofit() { + return new Retrofit.Builder().baseUrl(this.url).client(okHttpClient) + // Converts HTTP body objects to JSON + .addConverterFactory(JacksonConverterFactory.create()).build(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerUser.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerUser.java new file mode 100644 index 000000000..be70b1847 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/ScmManagerUser.java @@ -0,0 +1,27 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import com.fasterxml.jackson.annotation.JsonProperty; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; + +import java.util.HashMap; +import java.util.Map; + +/** + * Request payload describing an SCM-Manager user account, created via {@link UsersApi}. + */ +@Getter +@Setter +@NoArgsConstructor +public class ScmManagerUser { + private String name; + private String displayName; + private String mail; + private boolean external; + private String password; + private boolean active = true; + + @JsonProperty("_links") + private Map links = new HashMap<>(); +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApi.java b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApi.java new file mode 100644 index 000000000..b0b99ac0f --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApi.java @@ -0,0 +1,47 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import retrofit2.Call; +import retrofit2.http.Body; +import retrofit2.http.DELETE; +import retrofit2.http.Headers; +import retrofit2.http.POST; +import retrofit2.http.PUT; +import retrofit2.http.Path; + +import java.util.List; +import java.util.Map; + +/** + * Retrofit client for the SCM-Manager user REST API. + */ +public interface UsersApi { + /** + * Deletes the user with the given username. + * + * @param id username of the user to delete + * @return call that completes when the user was deleted + */ + @DELETE("v2/users/{id}") + Call delete(@Path("id") String id); + + /** + * Creates a new user account. + * + * @param user payload describing the user to create + * @return call that completes when the user was created + */ + @Headers("Content-Type: application/vnd.scmm-user+json;v=2") + @POST("v2/users") + Call addUser(@Body ScmManagerUser user); + + /** + * Replaces the global permissions of the given user. + * + * @param username username of the user to update + * @param permissions permission collection to set + * @return call that completes when the permissions were set + */ + @Headers("Content-Type: application/vnd.scmm-permissionCollection+json;v=2") + @PUT("v2/users/{username}/permissions") + Call setPermissionForUser(@Path("username") String username, @Body Map> permissions); +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/helm/HelmClient.java b/src/main/java/com/cloudogu/gitops/infrastructure/helm/HelmClient.java new file mode 100644 index 000000000..22f3d2711 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/helm/HelmClient.java @@ -0,0 +1,69 @@ +package com.cloudogu.gitops.infrastructure.helm; + +import com.cloudogu.gitops.utils.CommandExecutor; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class HelmClient { + + private final CommandExecutor commandExecutor; + + public String addRepo(String repoName, String url) { + return helm(List.of("repo", "add", repoName, url)); + } + + public String dependencyBuild(String path) { + return helm(List.of("dependency", "build", path)); + } + + public String upgrade(String release, String chartOrPath) { + return upgrade(release, chartOrPath, Map.of()); + } + + public String upgrade(String release, String chartOrPath, Map args) { + return helm(List.of("upgrade", "-i", release, chartOrPath, "--create-namespace"), args); + } + + public String template(String release, String chartOrPath) { + return template(release, chartOrPath, Map.of()); + } + + public String template(String release, String chartOrPath, Map args) { + return helm(List.of("template", release, chartOrPath), args); + } + + public String uninstall(String release, String namespace) { + String[] command = {"helm", "uninstall", release, "--namespace", namespace}; + return commandExecutor.execute(command).getStdOut(); + } + + private String helm(List verbAndParams) { + return helm(verbAndParams, Map.of()); + } + + private String helm(List verbAndParams, Map args) { + List command = new ArrayList<>(); + command.add("helm"); + command.addAll(verbAndParams); + + if (args != null) { + for (Map.Entry entry : args.entrySet()) { + String key = entry.getKey(); + Object value = entry.getValue(); + command.add("--" + key); + command.add(value != null ? value.toString() : ""); + } + } + + log.trace("Executing helm command: {}", String.join(" ", command)); + return commandExecutor.execute(command.toArray(new String[0])).getStdOut(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManager.java b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManager.java new file mode 100644 index 000000000..d862825f9 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManager.java @@ -0,0 +1,78 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class GlobalPropertyManager { + + private final JenkinsApiClient apiClient; + + public void setGlobalProperty(String key, String value) { + String script = """ + instance = Jenkins.getInstance() + globalNodeProperties = instance.getGlobalNodeProperties() + envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) + + def newEnvVarsNodeProperty + def envVars + + if ( envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0 ) { + newEnvVarsNodeProperty = new hudson.slaves.EnvironmentVariablesNodeProperty() + globalNodeProperties.add(newEnvVarsNodeProperty) + envVars = newEnvVarsNodeProperty.getEnvVars() + } else { + envVars = envVarsNodePropertyList.get(0).getEnvVars() + + } + + envVars.put('%KEY%', '%VALUE%') + + instance.save() + print("Done") + """; + + script = script.replace("%KEY%", escapeString(key)).replace("%VALUE%", escapeString(value)); + + String result = apiClient.runScript(script); + if (!"Done".equals(result)) { + throw new IllegalStateException("Could not create global property: " + result); + } + } + + public void deleteGlobalProperty(String key) { + String script = """ + def instance = Jenkins.getInstance() + def globalNodeProperties = instance.getGlobalNodeProperties() + def envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) + + if (envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0) { + print("Nothing to do") + return + } + + envVars = envVarsNodePropertyList.get(0).getEnvVars() + envVars.remove('%KEY%') + print("Done") + """; + + script = script.replace("%KEY%", escapeString(key)); + + String result = apiClient.runScript(script); + if (!"Nothing to do".equals(result) && !"Done".equals(result)) { + throw new IllegalStateException("Could not delete global property: " + result); + } + } + + private static String escapeString(String str) { + if (str.contains("\\")) { + // We don't want to get in trouble with escaping, + // e.g. `foo\'foo` => `foo\\'foo`. Now we would have a backslash followed by an unescaped + // quote. + throw new IllegalArgumentException("Backslashes within the escaped variables are forbidden."); + } + + return str.replace("'", "\\'"); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClient.java b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClient.java new file mode 100644 index 000000000..013bc19ae --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClient.java @@ -0,0 +1,189 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.config.Config; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import jakarta.inject.Named; +import jakarta.inject.Singleton; +import lombok.AccessLevel; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; +import okhttp3.Credentials; +import okhttp3.FormBody; +import okhttp3.OkHttpClient; +import okhttp3.Request; +import okhttp3.RequestBody; +import okhttp3.Response; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.function.Supplier; + +@Singleton +@Slf4j +public class JenkinsApiClient { + + private static final ObjectMapper objectMapper = new ObjectMapper(); + private static final int HTTP_OK = 200; + private static final int HTTP_UNAUTHORIZED = 401; + private static final int HTTP_FORBIDDEN = 403; + private static final int DEFAULT_MAX_RETRIES = 180; + private static final int DEFAULT_WAIT_PERIOD_MS = 2000; + + private final Config config; + private final OkHttpClient client; + private ResolvedCredentials runtimeCredentials; + + // Number of retries is uncommonly high, because we might have to outlive an unexpected Jenkins restart + // Here no constant is directly used because in uni tests we need to overwrite the maxRetries + @Setter(AccessLevel.PROTECTED) + private int maxRetries = DEFAULT_MAX_RETRIES; + + @Setter(AccessLevel.PROTECTED) + private int waitPeriodInMs = DEFAULT_WAIT_PERIOD_MS; + + public JenkinsApiClient(Config config, @Named("jenkins") OkHttpClient client) { + this.config = config; + + if (config.getApplication() != null && config.getApplication().getInsecure()) { + this.client = client.newBuilder().hostnameVerifier((hostname, session) -> true).build(); + } else { + this.client = client; + } + } + + public void setRuntimeCredentials(ResolvedCredentials runtimeCredentials) { + this.runtimeCredentials = runtimeCredentials; + } + + public String runScript(String code) { + log.trace("Running groovy script in Jenkins: {}", code); + try (Response response = postRequestWithCrumb( + "scriptText", new FormBody.Builder().add("script", code) + .build() + )) { + if (response.code() != HTTP_OK) { + throw new IllegalStateException("Could not run script. Status code " + response.code()); + } + return response.body().string(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to run Jenkins script", e); + } + } + + public Response postRequestWithCrumb(String url) { + return postRequestWithCrumb(url, null); + } + + public Response postRequestWithCrumb(String url, RequestBody postData) { + return sendRequestWithRetries(() -> { + Request.Builder request = buildRequest(url).header("Jenkins-Crumb", getCrumb()); + + if (postData != null) { + request.method("POST", postData); + } else { + // Explicitly set empty body, Otherwise okhttp sends GET + RequestBody emptyBody = RequestBody.create("", null); + request.method("POST", emptyBody); + } + + return request.build(); + }); + } + + private String getCrumb() { + log.trace("Getting Crumb for Jenkins"); + // Single attempt: this is called from within postRequestWithCrumb()'s own retry loop, which + // already waits and retries up to maxRetries times. Retrying here too would multiply into maxRetries^2 + // attempts. + try (Response response = sendRequestWithRetries(() -> buildRequest("crumbIssuer/api/json").build(), 1)) { + if (response.code() != HTTP_OK) { + throw new IllegalStateException("Could not create crumb. Status code " + response.code()); + } + + JsonNode json = objectMapper.readTree(response.body().byteStream()); + + if (json == null || !json.has("crumb")) { + throw new IllegalStateException("Could not create crumb. Invalid json."); + } + + return json.get("crumb").asText(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to retrieve Jenkins crumb", e); + } + } + + private Request.Builder buildRequest(String url) { + String username = runtimeCredentials == null + ? config.getJenkins().getUsername() + : runtimeCredentials.username(); + String password = runtimeCredentials == null + ? config.getJenkins().getPassword() + : runtimeCredentials.password(); + + return new Request.Builder().url(config.getJenkins().getUrl() + "/" + url) + .header("Authorization", Credentials.basic(username, password)); + } + + // We pass a supplier, so that we actually refetch a new crumb for a failed request + // The Jenkins ApiClient has its own retry logic on top of RetryInterceptor, because of crumb + // lifetime and restarts + private Response sendRequestWithRetries(Supplier requestSupplier) { + return sendRequestWithRetries(requestSupplier, maxRetries); + } + + private Response sendRequestWithRetries(Supplier requestSupplier, int retries) { + int retry = 0; + Response response = null; + do { + closeQuietly(response); + response = attemptRequest(requestSupplier, retry, retries); + if (response != null && !shouldRetryRequest(response)) { + break; + } + waitBeforeRetry(retry, retries, response); + retry++; + } while (retry < retries); + + if (response == null) { + throw new IllegalStateException("Failed to send request after " + retries + " retries"); + } + return response; + } + + private Response attemptRequest(Supplier requestSupplier, int retry, int retries) { + try { + Request request = requestSupplier.get(); + return client.newCall(request).execute(); + } catch (Exception e) { + log.trace("Jenkins request failed, retrying... (try {}/{})", retry, retries, e); + return null; + } + } + + private void waitBeforeRetry(int retry, int retries, Response response) { + if (retry + 1 >= retries) { + return; + } + try { + Thread.sleep(waitPeriodInMs); + } catch (InterruptedException e) { + closeQuietly(response); + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting for retry", e); + } + } + + private static void closeQuietly(Response response) { + if (response != null) { + response.close(); + } + } + + private static boolean shouldRetryRequest(Response response) { + // We might run into a 403 due to an invalid crumb from a previous session before jenkins was + // restarted. Here in the ApiClient, we simply retry all 401 and 403 including fetching a new crumb + return response.code() == HTTP_UNAUTHORIZED || response.code() == HTTP_FORBIDDEN; + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/JobManager.java b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/JobManager.java new file mode 100644 index 000000000..5538d5789 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/JobManager.java @@ -0,0 +1,125 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.fasterxml.jackson.databind.ObjectMapper; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import okhttp3.FormBody; +import okhttp3.MediaType; +import okhttp3.RequestBody; +import okhttp3.Response; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.LinkedHashMap; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class JobManager { + + private static final int HTTP_OK = 200; + + private static final ObjectMapper objectMapper = new ObjectMapper(); + + private final JenkinsApiClient apiClient; + + public void createCredential(String jobName, String id, String username, String password, String description) { + try { + Map innerMap = new LinkedHashMap<>(); + innerMap.put("scope", "GLOBAL"); + innerMap.put("id", id); + innerMap.put("username", username); + innerMap.put("password", password); + innerMap.put("description", description); + innerMap.put("$class", "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"); + + Map payloadMap = new LinkedHashMap<>(); + payloadMap.put("credentials", innerMap); + + String jsonPayload = objectMapper.writeValueAsString(payloadMap); + + try (Response response = apiClient.postRequestWithCrumb( + "job/" + jobName + "/credentials/store/folder/domain/_/createCredentials", + new FormBody.Builder().add("json", jsonPayload) + .build() + )) { + if (response.code() != HTTP_OK) { + throw new IllegalStateException("Could not create credential id=" + id + ",job=" + jobName + ". StatusCode: " + response.code()); + } + } + } catch (IOException e) { + throw new UncheckedIOException("Failed to serialize or send credential request", e); + } + } + + /** + * @return true, if created; false if job already exists and nothing was changed. + */ + public boolean createJob(String name, String serverUrl, String jobNamespace, String credentialsId) { + if (jobExists(name)) { + log.warn("Job '{}' already exists, ignoring.", name); + return false; + } + createJobViaApi(name, serverUrl, jobNamespace, credentialsId); + return true; + } + + private void createJobViaApi(String name, String serverUrl, String jobNamespace, String credentialsId) { + try { + // Note for development: the XML representation of an existing job can be exporting by + // adding /config.xml to the URL + String payloadXml = new TemplatingEngine().template( + new File("argocd/cluster-resources/apps/jenkins/templates/namespaceJobTemplate.xml.ftl"), + Map.of( + "SCMM_NAMESPACE_JOB_SERVER_URL", + serverUrl, + "SCMM_NAMESPACE_JOB_NAMESPACE", + jobNamespace, + "SCMM_NAMESPACE_JOB_CREDENTIALS_ID", + credentialsId + ) + ); + + RequestBody body = RequestBody.create(payloadXml, MediaType.get("text/xml")); + + try (Response response = apiClient.postRequestWithCrumb("createItem?name=" + name, body)) { + if (response.code() != HTTP_OK) { + throw new IllegalStateException("Could not create job '" + name + "'. StatusCode: " + response.code()); + } + } + } catch (IOException | freemarker.template.TemplateException e) { + throw new RuntimeException("Failed to prepare or deploy Helm chart / template XML", e); + } + } + + public boolean jobExists(String name) { + try (Response response = apiClient.postRequestWithCrumb("job/" + name)) { + return response.code() == HTTP_OK; + } + } + + public void deleteJob(String name) { + if (name.contains("'")) { + throw new IllegalArgumentException("Job name cannot contain quotes."); + } + + String script = "print(Jenkins.instance.getItem('" + name + "')?.delete())"; + String result = apiClient.runScript(script); + + if (!"null".equals(result)) { + throw new IllegalStateException("Could not delete job " + name); + } + } + + public void startJob(String jobName) { + try (Response response = apiClient.postRequestWithCrumb("job/" + jobName + "/build?delay=0sec")) { + if (response.code() != HTTP_OK) { + throw new IllegalStateException("Could not trigger build of Jenkins job: " + jobName + ". StatusCode: " + response.code()); + } + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/PrometheusConfigurator.java b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/PrometheusConfigurator.java new file mode 100644 index 000000000..c58a41262 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/PrometheusConfigurator.java @@ -0,0 +1,25 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class PrometheusConfigurator { + private final JenkinsApiClient apiClient; + + public void enableAuthentication() { + String result = apiClient.runScript(""" + import org.jenkinsci.plugins.prometheus.config.* + + def config = Jenkins.instance.getDescriptor(PrometheusConfiguration) + config.setUseAuthenticatedEndpoint(true) + + print(config.useAuthenticatedEndpoint) + """); + + if (!"true".equals(result)) { + throw new IllegalStateException("Cannot enable authentication for prometheus: " + result); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/UserManager.java b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/UserManager.java new file mode 100644 index 000000000..73eb9c921 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/jenkins/UserManager.java @@ -0,0 +1,115 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.util.List; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class UserManager { + + private final JenkinsApiClient apiClient; + + public void createUser(String username, String password) { + log.debug("Add user {} to jenkins", username); + + String script = """ + def realm = Jenkins.getInstance().getSecurityRealm() + def user = realm.createAccount('%USERNAME%', '%PASSWORD%') + + print(user) + """; + + script = script.replace("%USERNAME%", escapeString(username)).replace("%PASSWORD%", escapeString(password)); + + String result = apiClient.runScript(script); + + if (!username.equals(result)) { + throw new IllegalStateException("Error when creating user: " + result); + } + } + + public void grantPermission(String username, Permissions permission) { + if (!isUsingMatrixBasedPermissions()) { + log.debug("Is not using matrix based permission. Does not need to add permission."); + return; + } + + log.debug("Grant user {} permission {}", username, permission); + + String script = """ + import org.jenkinsci.plugins.matrixauth.PermissionEntry + import org.jenkinsci.plugins.matrixauth.AuthorizationType + + def permissions = Jenkins.getInstance().getAuthorizationStrategy().getGrantedPermissionEntries() + permissions.computeIfAbsent(%PERMISSION%) { + new HashSet<>() + } + print(permissions[%PERMISSION%].add(new PermissionEntry(AuthorizationType.USER, '%USERNAME%'))) + """; + + script = script.replace("%PERMISSION%", permission.toJenkinsPermissionEnum()) + .replace("%USERNAME%", escapeString(username)); + + String result = apiClient.runScript(script); + + if (!"true".equals(result) && !"false".equals(result)) { + // Both are valid return values for Set.add(). true == was already in set, false == was not + // already in set + throw new IllegalStateException("Failed to add permission " + permission + " to " + username + ": " + result); + } + } + + public boolean isUsingMatrixBasedPermissions() { + String result = apiClient.runScript("print(Jenkins.getInstance().getAuthorizationStrategy().class)"); + + if (!result.startsWith("class ")) { + throw new IllegalStateException("Error when trying to determine authorization strategy: " + result); + } + + return "class hudson.security.GlobalMatrixAuthorizationStrategy".equals(result) || "class hudson.security.ProjectMatrixAuthorizationStrategy".equals( + result); + } + + public boolean isUsingSecurityRealmWithoutLocalUserCreation() { + String result = apiClient.runScript("print(Jenkins.getInstance().getSecurityRealm().class)"); + + if (!result.startsWith("class ")) { + throw new IllegalStateException("Error when trying to determine security realm: " + result); + } + + return List.of( + "class org.jenkinsci.plugins.cas.CasSecurityRealm", + "class org.jenkinsci.plugins.oic.OicSecurityRealm" + ) + .contains(result); + } + + private static String escapeString(String str) { + if (str.contains("\\")) { + // We don't want to get in trouble with escaping, + // e.g. `foo\'foo` => `foo\\'foo`. Now we would have a backslash followed by an unescaped + // quote. + throw new IllegalArgumentException("Backslashes within the escaped variables are forbidden."); + } + + return str.replace("'", "\\'"); + } + + public enum Permissions { + METRICS_VIEW("jenkins.metrics.api.Metrics.VIEW"); + + private final String value; + + Permissions(String value) { + this.value = value; + } + + public String toJenkinsPermissionEnum() { + return value; + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClient.java b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClient.java new file mode 100644 index 000000000..a48ae06ad --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClient.java @@ -0,0 +1,1470 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.api; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.utils.MapUtils; +import com.cloudogu.gitops.utils.Tuple; +import com.fasterxml.jackson.core.type.TypeReference; +import io.fabric8.kubernetes.api.model.ConfigMap; +import io.fabric8.kubernetes.api.model.ConfigMapBuilder; +import io.fabric8.kubernetes.api.model.GenericKubernetesResource; +import io.fabric8.kubernetes.api.model.GenericKubernetesResourceBuilder; +import io.fabric8.kubernetes.api.model.GenericKubernetesResourceList; +import io.fabric8.kubernetes.api.model.HasMetadata; +import io.fabric8.kubernetes.api.model.IntOrString; +import io.fabric8.kubernetes.api.model.NamedContext; +import io.fabric8.kubernetes.api.model.Namespace; +import io.fabric8.kubernetes.api.model.NamespaceBuilder; +import io.fabric8.kubernetes.api.model.Node; +import io.fabric8.kubernetes.api.model.NodeAddress; +import io.fabric8.kubernetes.api.model.NodeList; +import io.fabric8.kubernetes.api.model.Pod; +import io.fabric8.kubernetes.api.model.PodBuilder; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.api.model.Service; +import io.fabric8.kubernetes.api.model.ServiceBuilder; +import io.fabric8.kubernetes.api.model.ServicePort; +import io.fabric8.kubernetes.api.model.ServicePortBuilder; +import io.fabric8.kubernetes.client.ConfigBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import io.fabric8.kubernetes.client.dsl.NonDeletingOperation; +import io.fabric8.kubernetes.client.dsl.Resource; +import io.fabric8.kubernetes.client.dsl.base.PatchContext; +import io.fabric8.kubernetes.client.dsl.base.ResourceDefinitionContext; +import io.fabric8.kubernetes.client.utils.Serialization; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.function.Supplier; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +/** + * Kubernetes client using Fabric8 Kubernetes Client. + */ +@Singleton +@SuppressWarnings("java:S3776") +@Slf4j +public class K8sClient { + + private static final TypeReference> MAP_TYPE = new TypeReference<>() { + }; + + private static final String DEFAULT_NAMESPACE = "default"; + private static final String INTERNAL_IP_TYPE = "InternalIP"; + private static final String DOCKER_CONFIG_JSON_TYPE = "kubernetes.io/dockerconfigjson"; + private static final String DOCKER_CONFIG_JSON_KEY = ".dockerconfigjson"; + private static final String NOT_FOUND_IN_NAMESPACE = " not found in namespace "; + private static final String APPLIED_PREFIX = "Applied "; + private static final ResourceDefinitionContext OPENSHIFT_PROJECT_CONTEXT = new ResourceDefinitionContext.Builder() + .withGroup("project.openshift.io") + .withVersion("v1") + .withKind("Project") + .withPlural("projects") + .withNamespaced(false) + .build(); + + private static final int DEFAULT_TIMEOUT_SECONDS = 60; + private static final int DEFAULT_CHECK_INTERVAL_SECONDS = 1; + private static final int FABRIC8_REQUEST_TIMEOUT_MILLIS = 60_000; + private static final int FABRIC8_CONNECTION_TIMEOUT_MILLIS = 10_000; + private static final int MILLIS_PER_SECOND = 1000; + private static final int DEFAULT_SLEEP_TIME_MILLIS = MILLIS_PER_SECOND; + private static final int DEFAULT_RETRIES = 120; + + protected int sleepTimeMillis = DEFAULT_SLEEP_TIME_MILLIS; + protected int defaultRetries = DEFAULT_RETRIES; + + /** + * -- GETTER -- + * Returns the underlying fabric8 client. + *

+ *

+ * -- SETTER -- + * Replaces the underlying fabric8 client, mainly for tests. + * + * @return the fabric8 client + * @param client the fabric8 client to use + */ + @Setter + @Getter + private KubernetesClient client; + /** + * -- SETTER -- + * Sets the GitOps Playground config after construction. + * + * @param gopConfig the GitOps Playground config; may be null + */ + @Setter + private com.cloudogu.gitops.config.Config gopConfig; + + /** + * Creates a client with default fabric8 configuration and no playground config. + */ + public K8sClient() { + this(null); + } + + /** + * Creates a client with default fabric8 configuration. + * + * @param gopConfig the GitOps Playground config, used e.g. to detect OpenShift mode; may be null + */ + public K8sClient(com.cloudogu.gitops.config.Config gopConfig) { + io.fabric8.kubernetes.client.Config config = new ConfigBuilder().withRequestTimeout( + FABRIC8_REQUEST_TIMEOUT_MILLIS) + .withConnectionTimeout( + FABRIC8_CONNECTION_TIMEOUT_MILLIS) + .build(); + + this.client = new KubernetesClientBuilder().withConfig(config).build(); + this.gopConfig = gopConfig; + } + + /** + * Waits for the first node in the cluster to become available. + * + * @return The name of the first available node + */ + public String waitForNode() { + log.debug("Waiting for first node of the cluster to become ready"); + + String nodeName = waitForResourceWithRetry( + "node", () -> { + NodeList nodes = client.nodes().list(); + if (nodes != null && nodes.getItems() != null && !nodes.getItems().isEmpty()) { + return nodes.getItems().get(0).getMetadata().getName(); + } + return null; + } + ); + + log.debug("First node of the cluster is ready: {}", nodeName); + return nodeName; + } + + /** + * Waits for and retrieves the internal IP address of the first node. + * + * @return the internal IP address of the first node + */ + public String waitForInternalNodeIp() { + String nodeName = waitForNode(); + log.debug("Waiting for internal IP of node {}", nodeName); + + String internalIp = waitForResourceWithRetry( + "internal IP of node " + nodeName, + () -> findInternalNodeIp(nodeName) + ); + + log.debug("Internal IP of node {}: {}", nodeName, internalIp); + return internalIp; + } + + private String findInternalNodeIp(String nodeName) { + Node node = client.nodes().withName(nodeName).get(); + if (node != null && node.getStatus() != null && node.getStatus().getAddresses() != null) { + for (NodeAddress address : node.getStatus().getAddresses()) { + if (INTERNAL_IP_TYPE.equals(address.getType())) { + return address.getAddress(); + } + } + } + return null; + } + + /** + * Waits for a service's NodePort to become available. + * + * @param serviceName name of the service to inspect + * @param namespace namespace of the service; empty means the default namespace + * @return the NodePort of the service's first port + */ + public String waitForNodePort(String serviceName, String namespace) { + log.debug("Getting node port for service {}, ns={}", serviceName, namespace); + + String nodePort = waitForResourceWithRetry( + "node port for service " + serviceName, + () -> findServiceNodePort(serviceName, namespace) + ); + + log.debug("Node port for service {}, ns={}: {}", serviceName, namespace, nodePort); + return nodePort; + } + + private String findServiceNodePort(String serviceName, String namespace) { + Service service = client.services().inNamespace(namespace).withName(serviceName).get(); + if (service != null && service.getSpec() != null && service.getSpec().getPorts() != null && !service.getSpec() + .getPorts() + .isEmpty()) { + Integer port = service.getSpec().getPorts().get(0).getNodePort(); + return port != null ? port.toString() : null; + } + return null; + } + + /** + * Creates a NodePort service (idempotent). + * + * @param name name of the service to create + * @param tcp port mapping in the form {@code port[:targetPort]} + * @param nodePort fixed node port to expose; empty for auto-assignment + * @param namespace target namespace; empty means the default namespace + */ + public void createServiceNodePort(String name, String tcp, String nodePort, String namespace) { + log.debug("Creating NodePort service {} in namespace {}", name, namespace); + + String[] ports = tcp.split(":"); + int port = Integer.parseInt(ports[0]); + int targetPort = ports.length > 1 ? Integer.parseInt(ports[1]) : port; + + ServicePort servicePort = new ServicePortBuilder().withPort(port) + .withTargetPort(new IntOrString(targetPort)) + .build(); + if (nodePort != null && !nodePort.isEmpty()) { + servicePort.setNodePort(Integer.parseInt(nodePort)); + } + + Service service = new ServiceBuilder().withNewMetadata() + .withName(name) + .withNamespace(resolveNamespace(namespace)) + .endMetadata() + .withNewSpec() + .withType("NodePort") + .withPorts(servicePort) + .endSpec() + .build(); + + executeWithErrorHandling( + "create NodePort service " + name, () -> { + client.services() + .inNamespace(resolveNamespace(namespace)) + .resource(service) + .createOr(NonDeletingOperation::update); + return null; + } + ); + + log.debug("NodePort service {} created/updated successfully", name); + } + + /** + * Patches the nodePort of a specific port in a service. + * + * @param serviceName name of the service to patch + * @param namespace namespace of the service + * @param portName name of the port entry whose nodePort is replaced + * @param newNodePort new node port value + */ + public void patchServiceNodePort(String serviceName, String namespace, String portName, int newNodePort) { + K8sClientHelper.validateServiceNodePortPatch(serviceName, namespace, portName, newNodePort); + + log.debug("Patching service {} port {} with nodePort {}", serviceName, portName, newNodePort); + + Service service = client.services().inNamespace(namespace).withName(serviceName).get(); + + if (service == null) { + throw new IllegalStateException("Service " + serviceName + NOT_FOUND_IN_NAMESPACE + namespace); + } + + List ports = service.getSpec().getPorts(); + int portIndex = -1; + for (int i = 0; i < ports.size(); i++) { + if (portName.equals(ports.get(i).getName())) { + portIndex = i; + break; + } + } + + if (portIndex == -1) { + throw new IllegalStateException("Port with name " + portName + " not found in service " + serviceName + "."); + } + + // Create JSON patch + List> patch = List.of(Map.of( + "op", + "replace", + "path", + "/spec/ports/" + portIndex + "/nodePort", + "value", + newNodePort + )); + + String patchJson = Serialization.asJson(patch); + PatchContext patchContext = new PatchContext.Builder().withPatchType(io.fabric8.kubernetes.client.dsl.base.PatchType.JSON) + .build(); + + executeWithErrorHandling( + "patch service " + serviceName, () -> { + client.services().inNamespace(namespace).withName(serviceName).patch(patchContext, patchJson); + return null; + } + ); + + log.debug( + "Service {} in namespace {} successfully patched with nodePort {} for port {}.", + serviceName, + namespace, + newNodePort, + portName + ); + } + + /** + * Creates a namespace (or an OpenShift project) if it does not already exist (idempotent). + * + * @param name name of the namespace to create + */ + public void createNamespace(String name) { + K8sClientHelper.validateNamespaceName(name); + + if (!namespaceExists(name)) { + log.debug("Namespace {} does not exist, proceeding to create.", name); + + if (runInOpenshift()) { + GenericKubernetesResource project = new GenericKubernetesResourceBuilder() + .withApiVersion("project.openshift.io/v1") + .withKind("Project") + .withNewMetadata() + .withName(name) + .endMetadata() + .build(); + executeWithErrorHandling( + "create project " + name, () -> { + client.genericKubernetesResources(OPENSHIFT_PROJECT_CONTEXT).resource(project).create(); + return null; + } + ); + log.debug("Project {} created successfully.", name); + } else { + Namespace namespace = new NamespaceBuilder().withNewMetadata().withName(name).endMetadata().build(); + + executeWithErrorHandling( + "create namespace " + name, () -> { + client.namespaces().resource(namespace).create(); + return null; + } + ); + + log.debug("Namespace {} created successfully.", name); + } + } + } + + /** + * Creates multiple namespaces. + * + * @param names names of the namespaces to create + */ + public void createNamespaces(List names) { + if (names == null) { + throw new IllegalArgumentException("Namespaces must be provided and cannot be null."); + } + for (String name : names) { + createNamespace(name); + } + } + + /** + * Checks if a namespace exists. + * + * @param namespace name of the namespace to check + * @return true if the namespace exists + */ + public boolean namespaceExists(String namespace) { + try { + Namespace ns = client.namespaces().withName(namespace).get(); + if (ns != null) { + log.debug("Namespace {} already exists.", namespace); + return true; + } + } catch (Exception e) { + log.trace("Namespace {} does not exist: {}", namespace, e.getMessage()); + } + return false; + } + + /** + * Creates or updates an empty secret in the default namespace (idempotent). + * + * @param type secret type, e.g. {@code generic} + * @param name name of the secret + */ + public void createSecret(String type, String name) { + createSecret(type, name, "", new Tuple[0]); + } + + /** + * Creates or updates an empty secret (idempotent). + * + * @param type secret type, e.g. {@code generic} + * @param name name of the secret + * @param namespace target namespace; empty means the default namespace + */ + public void createSecret(String type, String name, String namespace) { + createSecret(type, name, namespace, new Tuple[0]); + } + + /** + * Creates or updates a generic secret (idempotent). + * + * @param type secret type; {@code generic} is mapped to {@code Opaque} + * @param name name of the secret + * @param namespace target namespace; empty means the default namespace + * @param literals key-value pairs stored as string data + */ + public void createSecret(String type, String name, String namespace, Tuple... literals) { + log.debug("Creating secret {} of type {} in namespace {}", name, type, namespace); + + Map data = new HashMap<>(); + if (literals != null) { + for (Tuple tuple : literals) { + data.put(String.valueOf(tuple.getFirst()), String.valueOf(tuple.getSecond())); + } + } + + String resolvedType = "generic".equals(type) ? "Opaque" : type; + Secret secret = new SecretBuilder().withNewMetadata() + .withName(name) + .withNamespace(resolveNamespace(namespace)) + .endMetadata() + .withType(resolvedType) + .withStringData(data) + .build(); + + executeWithErrorHandling( + "create secret " + name, () -> { + client.secrets() + .inNamespace(resolveNamespace(namespace)) + .resource(secret) + .createOr(NonDeletingOperation::update); + return null; + } + ); + + log.debug("Secret {} created/updated successfully", name); + } + + /** + * Creates or updates an image pull secret in the default namespace (idempotent). + * + * @param name name of the secret + * @param host registry host the credentials belong to + * @param user registry username + * @param password registry password + */ + public void createImagePullSecret(String name, String host, String user, String password) { + createImagePullSecret(name, "", host, user, password); + } + + /** + * Creates or updates an image pull secret (idempotent). + * + * @param name name of the secret + * @param namespace target namespace; empty means the default namespace + * @param host registry host the credentials belong to + * @param user registry username + * @param password registry password + */ + public void createImagePullSecret(String name, String namespace, String host, String user, String password) { + log.debug("Creating image pull secret {} in namespace {}", name, namespace); + + String auth = Base64.getEncoder().encodeToString((user + ":" + password).getBytes(StandardCharsets.UTF_8)); + String dockerConfig = Serialization.asJson( + Map.of("auths", Map.of(host, Map.of("username", user, "password", password, "auth", auth))) + ); + + Secret secret = new SecretBuilder().withNewMetadata() + .withName(name) + .withNamespace(resolveNamespace(namespace)) + .endMetadata() + .withType(DOCKER_CONFIG_JSON_TYPE) + .addToStringData(DOCKER_CONFIG_JSON_KEY, dockerConfig) + .build(); + + executeWithErrorHandling( + "create image pull secret " + name, () -> { + client.secrets() + .inNamespace(resolveNamespace(namespace)) + .resource(secret) + .createOr(NonDeletingOperation::update); + return null; + } + ); + + log.debug("Image pull secret {} created/updated successfully", name); + } + + /** + * Retrieves the {@code namespaces} data from an ArgoCD secret, waiting for the secret to appear. + * + * @param name name of the secret + * @param namespace namespace of the secret; empty means the default namespace + * @return the base64-encoded {@code namespaces} value of the secret + */ + public String getArgoCDNamespacesSecret(String name, String namespace) { + log.debug("Getting Secret {} from namespace {}", name, namespace); + + return waitForResourceWithRetry( + "secret " + name, () -> { + Secret secret = client.secrets().inNamespace(resolveNamespace(namespace)).withName(name).get(); + + return (secret != null && secret.getData() != null && secret.getData() + .containsKey("namespaces")) ? secret.getData() + .get( + "namespaces") : null; + } + ); + } + + /** + * Extracts credentials from a secret using the default keys {@code username} and {@code + * password}. + * + * @param secretname name of the secret + * @param namespace namespace of the secret + * @return the decoded credentials + */ + public Credentials getCredentialsFromSecret(String secretname, String namespace) { + return getCredentialsFromSecret(secretname, namespace, "username", "password"); + } + + /** + * Extracts credentials from a Kubernetes secret. + * + * @param secretname name of the secret + * @param namespace namespace of the secret + * @param usernameKey data key holding the username + * @param passwordKey data key holding the password + * @return the decoded credentials + */ + public Credentials getCredentialsFromSecret( + String secretname, + String namespace, + String usernameKey, + String passwordKey) { + return executeWithErrorHandling( + "get credentials from secret " + secretname, + () -> resolveCredentialsFromSecret(secretname, namespace, usernameKey, passwordKey) + ); + } + + private Credentials resolveCredentialsFromSecret( + String secretname, + String namespace, + String usernameKey, + String passwordKey) { + Secret secret = client.secrets().inNamespace(namespace).withName(secretname).get(); + if (secret == null || secret.getData() == null) { + throw new IllegalStateException("Secret " + secretname + NOT_FOUND_IN_NAMESPACE + namespace); + } + + Map secretData = secret.getData(); + String username = new String(Base64.getDecoder().decode(secretData.get(usernameKey)), StandardCharsets.UTF_8); + String password = new String(Base64.getDecoder().decode(secretData.get(passwordKey)), StandardCharsets.UTF_8); + return new Credentials(username, password); + } + + /** + * Extracts credentials from a Kubernetes secret using a Credentials object as input. + * + * @param credentials reference describing secret name, namespace and data keys + * @return a copy of the input with username and password resolved from the secret + */ + public Credentials getCredentialsFromSecret(Credentials credentials) { + return executeWithErrorHandling( + "get credentials from secret " + credentials.getSecretName(), + () -> resolveCredentialsFromSecret(credentials) + ); + } + + private Credentials resolveCredentialsFromSecret(Credentials credentials) { + Secret secret = client.secrets() + .inNamespace(credentials.getSecretNamespace()) + .withName(credentials.getSecretName()) + .get(); + if (secret == null || secret.getData() == null) { + throw new IllegalStateException("Secret " + credentials.getSecretName() + NOT_FOUND_IN_NAMESPACE + credentials.getSecretNamespace()); + } + + Map secretData = secret.getData(); + String usernameEncoded = secretData.get(credentials.getUsernameKey()); + String username = usernameEncoded != null ? new String( + Base64.getDecoder() + .decode(usernameEncoded), StandardCharsets.UTF_8 + ) : credentials.getUsername(); + String password = new String( + Base64.getDecoder() + .decode(secretData.get(credentials.getPasswordKey())), StandardCharsets.UTF_8 + ); + + Credentials credentialsNew = new Credentials(credentials); + credentialsNew.setUsername(username); + credentialsNew.setPassword(password); + + return credentialsNew; + } + + /** + * Creates or updates a ConfigMap from a file (idempotent). + * + * @param name name of the ConfigMap + * @param namespace target namespace; empty means the default namespace + * @param filePath path of the file whose content becomes the ConfigMap data + */ + public void createConfigMapFromFile(String name, String namespace, String filePath) { + log.debug("Creating ConfigMap {} from file {} in namespace {}", name, filePath, namespace); + + File file = new File(filePath); + if (!file.exists()) { + throw new IllegalStateException("File not found: " + filePath); + } + + String fileContent; + try { + fileContent = Files.readString(file.toPath()); + } catch (IOException e) { + throw new UncheckedIOException("Failed to read file: " + filePath, e); + } + + Map data = Map.of(file.getName(), fileContent); + + ConfigMap configMap = new ConfigMapBuilder().withNewMetadata() + .withName(name) + .withNamespace(resolveNamespace(namespace)) + .endMetadata() + .withData(data) + .build(); + + executeWithErrorHandling( + "create ConfigMap " + name + " from file", () -> { + client.configMaps() + .inNamespace(resolveNamespace(namespace)) + .resource(configMap) + .createOr(NonDeletingOperation::update); + return null; + } + ); + + log.debug("ConfigMap {} created/updated successfully", name); + } + + /** + * Retrieves a value from a ConfigMap in the current namespace. + * + * @param mapName name of the ConfigMap + * @param key data key to read + * @return the value stored under the given key + */ + public String getConfigMap(String mapName, String key) { + String namespace = getCurrentNamespace(); + + log.debug("Getting ConfigMap {}/{}, key: {}", namespace, mapName, key); + + ConfigMap configMap = client.configMaps().inNamespace(namespace).withName(mapName).get(); + + if (configMap == null) { + throw new IllegalStateException("Could not fetch configmap " + mapName + " from namespace " + namespace); + } + + if (configMap.getData() == null || !configMap.getData().containsKey(key)) { + throw new IllegalStateException("Could not fetch " + key + " within config-map " + mapName + " from namespace " + namespace); + } + + return configMap.getData().get(key); + } + + /** + * Applies YAML resources from a URL, file or directory (recursively). + * + * @param yamlLocation http(s) URL, file path or directory path containing YAML resources + * @return a summary of how many resources were applied + */ + public String applyYaml(String yamlLocation) { + log.debug("Applying YAML from {}", yamlLocation); + + if (yamlLocation.startsWith("http://") || yamlLocation.startsWith("https://")) { + try { + int appliedResources = applyYamlStream(URI.create(yamlLocation).toURL().openStream(), yamlLocation); + return APPLIED_PREFIX + appliedResources + " resource(s) from " + yamlLocation; + } catch (IOException | IllegalArgumentException e) { + throw new UncheckedIOException("Failed to apply YAML from URL: " + yamlLocation, new IOException(e)); + } + } + + File location = new File(yamlLocation); + + if (!location.exists()) { + throw new IllegalStateException("File or directory not found: " + yamlLocation); + } + + if (location.isDirectory()) { + List yamlFiles; + try (Stream stream = Files.walk(location.toPath())) { + yamlFiles = stream.filter(Files::isRegularFile) + .map(Path::toFile) + .filter(file -> file.getName().endsWith(".yaml") || file.getName().endsWith(".yml")) + .collect(Collectors.toCollection(ArrayList::new)); + } catch (IOException e) { + throw new UncheckedIOException("Failed to list YAML files in directory: " + yamlLocation, e); + } + + yamlFiles.sort(Comparator.comparing(File::getAbsolutePath)); + + int appliedResources = 0; + for (File file : yamlFiles) { + try { + appliedResources += applyYamlStream(Files.newInputStream(file.toPath()), file.getAbsolutePath()); + } catch (IOException e) { + throw new UncheckedIOException("Failed to apply YAML file: " + file.getAbsolutePath(), e); + } + } + + return APPLIED_PREFIX + appliedResources + " resource(s) from directory " + yamlLocation; + } + + try { + int appliedResources = applyYamlStream(Files.newInputStream(location.toPath()), yamlLocation); + return APPLIED_PREFIX + appliedResources + " resource(s) from " + yamlLocation; + } catch (IOException e) { + throw new UncheckedIOException("Failed to apply YAML file: " + yamlLocation, e); + } + } + + private int applyYamlStream(InputStream stream, String sourceDescription) { + List resources = executeWithErrorHandling( + "load YAML from " + sourceDescription, + () -> loadYamlItems(stream, sourceDescription) + ); + + for (HasMetadata resource : resources) { + executeWithErrorHandling( + "apply resource from " + sourceDescription, () -> { + applyResource(resource); + return null; + } + ); + } + + return resources.size(); + } + + private void applyResource(HasMetadata resource) { + if (resource instanceof GenericKubernetesResource genericResource) { + applyGenericResource(genericResource); + return; + } + + String namespace = resource.getMetadata() != null ? resource.getMetadata().getNamespace() : null; + if (namespace != null && !namespace.isBlank()) { + client.resource(resource).inNamespace(namespace).createOr(NonDeletingOperation::update); + return; + } + + client.resource(resource).createOr(NonDeletingOperation::update); + } + + private void applyGenericResource(GenericKubernetesResource resource) { + ResourceDefinitionContext context = K8sClientHelper.resolveResourceDefinitionContext(client, resource.getKind()); + var resourceClient = client.genericKubernetesResources(context); + String namespace = resource.getMetadata() != null ? resource.getMetadata().getNamespace() : null; + + if (namespace != null && !namespace.isBlank()) { + resourceClient.inNamespace(namespace).resource(resource).createOr(NonDeletingOperation::update); + return; + } + + resourceClient.resource(resource).createOr(NonDeletingOperation::update); + } + + private List loadYamlItems(InputStream stream, String sourceDescription) { + try (stream) { + return client.load(stream).items(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to close YAML input stream for " + sourceDescription, e); + } + } + + /** + * Adds or removes labels on a resource in the default namespace. + * + * @param resource resource type, e.g. {@code node} + * @param name resource name; {@code --all} applies to all nodes + * @param keyValues labels to set; a key ending in {@code -} removes that label + */ + public void label(String resource, String name, Tuple... keyValues) { + label(resource, name, "", keyValues); + } + + /** + * Adds or removes labels on a resource. + * + * @param resource resource type, e.g. {@code node} + * @param name resource name; {@code --all} applies to all nodes + * @param namespace namespace of the resource; empty means the default namespace + * @param keyValues labels to set; a key ending in {@code -} removes that label + */ + public void label(String resource, String name, String namespace, Tuple... keyValues) { + if (keyValues == null || keyValues.length == 0) { + throw new IllegalArgumentException("Missing key-value-pairs"); + } + + if ("--all".equals(name)) { + NodeList nodes = client.nodes().list(); + if (nodes != null && nodes.getItems() != null) { + for (Node node : nodes.getItems()) { + label(resource, node.getMetadata().getName(), namespace, keyValues); + } + } + return; + } + + log.debug("Labeling {}/{} in namespace {}", resource, name, namespace); + + Map labelsToAdd = new HashMap<>(); + List labelsToRemove = new ArrayList<>(); + + for (Tuple tuple : keyValues) { + String key = String.valueOf(tuple.getFirst()); + String value = String.valueOf(tuple.getSecond()); + + if (key.endsWith("-")) { + labelsToRemove.add(key.substring(0, key.length() - 1)); + } else { + labelsToAdd.put(key, value); + } + } + + executeWithErrorHandling( + "label " + resource + "/" + name, () -> { + Resource resourceClient = K8sClientHelper.getResourceClient( + client, + resource, + name, + resolveNamespace(namespace) + ); + applyLabelChanges(resourceClient, resource, name, labelsToAdd, labelsToRemove); + return null; + } + ); + + log.debug("Labels updated successfully"); + } + + /** + * Fetches the resource behind {@code resourceClient}, applies the given label additions/removals + * and writes it back. Kept as a generic helper (rather than inline in {@link #label}) because + * {@code io.fabric8.kubernetes.client.dsl.Resource#replace} requires the exact type returned by + * {@code Resource#get}; a wildcard-typed local variable can't satisfy that across two separate + * calls due to Java's per-expression wildcard capture, whereas a type variable bound once for the + * whole method invocation can. + */ + private static void applyLabelChanges( + Resource resourceClient, + String resource, + String name, + Map labelsToAdd, + List labelsToRemove) { + T existingResource = resourceClient.get(); + + if (existingResource == null) { + throw new IllegalStateException("Resource " + resource + "/" + name + " not found"); + } + + Map existingLabels = existingResource.getMetadata().getLabels(); + if (existingLabels == null) { + existingLabels = new HashMap<>(); + } else { + existingLabels = new HashMap<>(existingLabels); // ensure mutable + } + + for (String key : labelsToRemove) { + existingLabels.remove(key); + } + existingLabels.putAll(labelsToAdd); + + existingResource.getMetadata().setLabels(existingLabels); + resourceClient.patch(existingResource); + } + + /** + * Removes the given labels from a resource. + * + * @param resource resource type, e.g. {@code node} + * @param name resource name; {@code --all} applies to all nodes + * @param namespace namespace of the resource; empty means the default namespace + * @param keys label keys to remove + */ + public void labelRemove(String resource, String name, String namespace, String... keys) { + Tuple[] tuples = new Tuple[keys.length]; + for (int i = 0; i < keys.length; i++) { + tuples[i] = new Tuple<>(keys[i] + "-", ""); + } + label(resource, name, namespace, tuples); + } + + /** + * Patches a resource in the default namespace using the default patch type. + * + * @param resource resource type, e.g. {@code service} + * @param name resource name + * @param yaml patch content as nested map + */ + public void patch(String resource, String name, Map yaml) { + patch(resource, name, "", "", yaml); + } + + /** + * Patches a resource using the default patch type. + * + * @param resource resource type, e.g. {@code service} + * @param name resource name + * @param namespace namespace of the resource; empty means the default namespace + * @param yaml patch content as nested map + */ + public void patch(String resource, String name, String namespace, Map yaml) { + patch(resource, name, namespace, "", yaml); + } + + /** + * Patches a resource. + * + * @param resource resource type, e.g. {@code service} + * @param name resource name + * @param namespace namespace of the resource; empty means the default namespace + * @param type patch type: {@code merge}, {@code json-merge}, {@code strategic} or {@code json} + * @param yaml patch content as nested map + */ + public void patch(String resource, String name, String namespace, String type, Map yaml) { + log.debug("Patching {}/{} in namespace {}", resource, name, namespace); + + PatchContext patchContext = K8sClientHelper.createPatchContext(type); + String patchJson = Serialization.asJson(yaml); + log.trace("Patch JSON: {}", patchJson); + + executeWithErrorHandling( + "patch " + resource + "/" + name, () -> { + Resource resourceClient = K8sClientHelper.getResourceClient( + client, + resource, + name, + resolveNamespace(namespace) + ); + resourceClient.patch(patchContext, patchJson); + return null; + } + ); + + log.debug("Resource {}/{} patched successfully", resource, name); + } + + /** + * Deletes resources by label selectors in the default namespace, see {@link #delete(String, + * String, Tuple...)}. + * + * @param resource resource type, e.g. {@code secret} + */ + public void delete(String resource) { + delete(resource, "", new Tuple[0]); + } + + /** + * Deletes resources by label selectors, see {@link #delete(String, String, Tuple...)}. + * + * @param resource resource type, e.g. {@code secret} + * @param namespace namespace to delete in; empty means the default namespace + */ + public void delete(String resource, String namespace) { + delete(resource, namespace, new Tuple[0]); + } + + /** + * Deletes all resources of a type matching the given label selectors. Failures are logged, not + * thrown, since the resources may not exist. + * + * @param resource resource type, e.g. {@code secret} + * @param namespace namespace to delete in; empty means the default namespace + * @param selectors label key-value pairs the resources must match + */ + public void delete(String resource, String namespace, Tuple... selectors) { + log.debug("Deleting {} in namespace {} with selectors", resource, namespace); + + Map labels = new HashMap<>(); + if (selectors != null) { + for (Tuple tuple : selectors) { + labels.put(String.valueOf(tuple.getFirst()), String.valueOf(tuple.getSecond())); + } + } + + try { + K8sClientHelper.deleteResourcesByType(client, resource, resolveNamespace(namespace), labels); + log.debug("Resources deleted successfully"); + } catch (Exception e) { + log.warn("Failed to delete resources (may not exist): {}", e.getMessage()); + } + } + + /** + * Deletes a single resource by name. Failures are logged, not thrown, since the resource may not + * exist. + * + * @param resource resource type, e.g. {@code secret} + * @param namespace namespace of the resource; empty means the default namespace + * @param name resource name + */ + public void delete(String resource, String namespace, String name) { + log.debug("Deleting {}/{} in namespace {}", resource, name, namespace); + + try { + Resource resourceClient = K8sClientHelper.getResourceClient( + client, + resource, + name, + resolveNamespace(namespace) + ); + resourceClient.delete(); + log.debug("Resource {}/{} deleted successfully", resource, name); + } catch (Exception e) { + log.warn("Failed to delete resource (may not exist): {}", e.getMessage()); + } + } + + /** + * Runs a pod in the default namespace, see {@link #run(String, String, String, Map, String...)}. + * + * @param name name of the pod + * @param image container image to run + * @return a status message or, with {@code --rm}-style params, the pod output + */ + public String run(String name, String image) { + return run(name, image, "", Map.of(), new String[0]); + } + + /** + * Runs a pod, see {@link #run(String, String, String, Map, String...)}. + * + * @param name name of the pod + * @param image container image to run + * @param namespace target namespace; empty means the default namespace + * @return a status message or, with {@code --rm}-style params, the pod output + */ + public String run(String name, String image, String namespace) { + return run(name, image, namespace, Map.of(), new String[0]); + } + + /** + * Runs a pod with pod-spec overrides, see {@link #run(String, String, String, Map, String...)}. + * + * @param name name of the pod + * @param image container image to run + * @param namespace target namespace; empty means the default namespace + * @param overrides pod spec fields to override, analogous to {@code kubectl run --overrides} + * @return a status message or, with {@code --rm}-style params, the pod output + */ + public String run(String name, String image, String namespace, Map overrides) { + return run(name, image, namespace, overrides, new String[0]); + } + + /** + * Runs a pod with kubectl-run-style params, see {@link #run(String, String, String, Map, + * String...)}. + * + * @param name name of the pod + * @param image container image to run + * @param namespace target namespace; empty means the default namespace + * @param params kubectl-run-style flags such as {@code --rm} or {@code --restart=Never} + * @return a status message or, with {@code --rm}-style params, the pod output + */ + public String run(String name, String image, String namespace, String... params) { + return run(name, image, namespace, Map.of(), params); + } + + /** + * Runs a pod, analogous to {@code kubectl run}. + * + * @param name name of the pod + * @param image container image to run + * @param namespace target namespace; empty means the default namespace + * @param overrides pod spec fields to override, analogous to {@code kubectl run --overrides} + * @param params kubectl-run-style flags such as {@code --rm} or {@code --restart=Never} + * @return a status message or, when the params request output collection, the pod output + */ + public String run(String name, String image, String namespace, Map overrides, String... params) { + log.debug("Running pod {} with image {} in namespace {}", name, image, namespace); + String resolvedNamespace = resolveNamespace(namespace); + List runParams = params != null ? Arrays.asList(params) : Collections.emptyList(); + + Pod pod = new PodBuilder().withNewMetadata() + .withName(name) + .withNamespace(resolvedNamespace) + .endMetadata() + .withNewSpec() + .addNewContainer() + .withName(name) + .withImage(image) + .endContainer() + .endSpec() + .build(); + + K8sClientHelper.applyRunParams(pod, runParams); + + if (overrides != null && !overrides.isEmpty()) { + log.debug("Applying overrides: {}", overrides); + pod = K8sClientHelper.applyPodOverrides(pod, overrides); + } + + final Pod finalPod = pod; + Pod createdPod = executeWithErrorHandling( + "run pod " + name, () -> client.pods() + .inNamespace(resolvedNamespace) + .resource(finalPod) + .create() + ); + + log.debug("Pod {} created successfully", name); + if (K8sClientHelper.shouldReturnPodOutput(runParams)) { + return K8sClientHelper.collectPodRunOutput( + client, + createdPod.getMetadata() + .getName(), + resolvedNamespace, + K8sClientHelper.shouldRemovePod(runParams), + defaultRetries, + sleepTimeMillis, + this + ); + } + + return "pod/" + createdPod.getMetadata().getName() + " created"; + } + + /** + * Lists custom resources of the given type across all namespaces. + * + * @param resource custom resource type, resolved via API discovery + * @return namespace/name pairs of all found resources; empty when the type is unknown or listing + * fails + */ + public List getCustomResource(String resource) { + log.debug("Getting custom resources of type {}", resource); + + try { + Map match = K8sClientHelper.findApiResourceViaDiscovery( + client, + resource.toLowerCase(Locale.ROOT), + resource + ); + ResourceDefinitionContext context = new ResourceDefinitionContext.Builder().withGroup((String) match.get( + "group")) + .withVersion((String) match.get( + "version")) + .withKind((String) match.get( + "kind")) + .withPlural((String) match.get( + "plural")) + .withNamespaced((Boolean) match.get( + "namespaced")) + .build(); + + // `apiClient`'s type is a long nested generic (MixedOperation>); spelling it out + // would hurt readability more than `var` costs, so it's kept as `var` deliberately. + var apiClient = client.genericKubernetesResources(context); + GenericKubernetesResourceList resourceList = apiClient.inAnyNamespace().list(); + + if (resourceList == null || resourceList.getItems() == null) { + return Collections.emptyList(); + } + + return resourceList.getItems().stream().map(K8sClient::toCustomResource).toList(); + } catch (Exception e) { + log.warn("Failed to get custom resources: {}", e.getMessage()); + return Collections.emptyList(); + } + } + + private static CustomResource toCustomResource(GenericKubernetesResource item) { + Map metadata = item.getMetadata() != null ? Serialization.unmarshal( + Serialization.asJson(item.getMetadata()), + MAP_TYPE + ) : Collections.emptyMap(); + String ns = metadata.containsKey("namespace") ? String.valueOf(metadata.get("namespace")) : ""; + String name = metadata.containsKey("name") ? String.valueOf(metadata.get("name")) : ""; + return new CustomResource(ns, name); + } + + /** + * Reads an annotation from a resource in the default namespace. + * + * @param resource resource type, e.g. {@code service} + * @param name resource name + * @param key annotation key to read + * @return the annotation value; may be null when the annotation is not set + */ + public String getAnnotation(String resource, String name, String key) { + return getAnnotation(resource, name, key, ""); + } + + /** + * Reads an annotation from a resource. + * + * @param resource resource type, e.g. {@code service} + * @param name resource name + * @param key annotation key to read + * @param namespace namespace of the resource; empty means the default namespace + * @return the annotation value; may be null when the annotation is not set + */ + public String getAnnotation(String resource, String name, String key, String namespace) { + log.debug("Getting annotation {} from {}/{} in namespace {}", key, resource, name, namespace); + + Resource resourceClient = K8sClientHelper.getResourceClient( + client, + resource, + name, + resolveNamespace(namespace) + ); + HasMetadata k8sResource = resourceClient.get(); + + if (k8sResource == null) { + throw new IllegalStateException("Resource " + resource + "/" + name + " not found"); + } + + Map annotations = k8sResource.getMetadata().getAnnotations(); + if (annotations == null) { + throw new IllegalStateException("No annotations found on resource " + resource + "/" + name); + } + + String value = annotations.get(key); + log.debug("getAnnotation returns = {}", value); + return value; + } + + /** + * Returns the name of the current kubeconfig context. + * + * @return the context name, or a placeholder when no context is set + */ + public String getCurrentContext() { + try { + NamedContext currentContext = client.getConfiguration().getCurrentContext(); + String context = currentContext != null ? currentContext.getName() : null; + return context != null ? context : "(current context not set)"; + } catch (Exception e) { + log.trace("Failed to get current context: {}", e.getMessage()); + return "(current context not set)"; + } + } + + /** + * Waits for a resource to reach a phase using default timeout and check interval. + * + * @param resourceType resource type, e.g. {@code pod} + * @param resourceName resource name + * @param namespace namespace of the resource; empty means the default namespace + * @param desiredPhase phase to wait for, e.g. {@code Running} + */ + public void waitForResourcePhase(String resourceType, String resourceName, String namespace, String desiredPhase) { + waitForResourcePhase( + resourceType, + resourceName, + namespace, + desiredPhase, + DEFAULT_TIMEOUT_SECONDS, + DEFAULT_CHECK_INTERVAL_SECONDS + ); + } + + /** + * Waits for a resource to reach a phase, polling in fixed intervals until the timeout expires. + * + * @param resourceType resource type, e.g. {@code pod} + * @param resourceName resource name + * @param namespace namespace of the resource; empty means the default namespace + * @param desiredPhase phase to wait for, e.g. {@code Running} + * @param timeoutSeconds maximum time to wait before failing + * @param checkIntervalSeconds pause between phase checks + */ + public void waitForResourcePhase( + String resourceType, + String resourceName, + String namespace, + String desiredPhase, + int timeoutSeconds, + int checkIntervalSeconds) { + K8sClientHelper.validateWaitForResourcePhaseParams( + resourceType, + resourceName, + namespace, + desiredPhase, + timeoutSeconds, + checkIntervalSeconds + ); + + log.debug("Waiting for {}/{} to reach phase {}", resourceType, resourceName, desiredPhase); + + long startTime = System.currentTimeMillis(); + long endTime = startTime + ((long) timeoutSeconds * MILLIS_PER_SECOND); + + while (System.currentTimeMillis() < endTime) { + if (hasReachedPhase(resourceType, resourceName, namespace, desiredPhase)) { + log.debug( + "Resource {}/{} in namespace {} reached the desired phase: {}", + resourceType, + resourceName, + namespace, + desiredPhase + ); + return; + } + + try { + Thread.sleep((long) checkIntervalSeconds * MILLIS_PER_SECOND); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting for resource phase", e); + } + } + + throw new IllegalStateException("Timeout reached. Resource " + resourceType + "/" + resourceName + " in namespace " + namespace + " did not reach the desired phase: " + desiredPhase + " within " + timeoutSeconds + " seconds."); + } + + private boolean hasReachedPhase(String resourceType, String resourceName, String namespace, String desiredPhase) { + try { + Resource resourceClient = K8sClientHelper.getResourceClient( + client, + resourceType, + resourceName, + resolveNamespace(namespace) + ); + HasMetadata resource = resourceClient.get(); + if (resource == null) { + return false; + } + + String phase = extractPhase(resource); + if (desiredPhase.equals(phase)) { + return true; + } + + log.debug("Current phase: {}. Waiting for phase: {}...", phase, desiredPhase); + return false; + } catch (Exception e) { + log.trace("Error checking resource phase: {}", e.getMessage()); + return false; + } + } + + private static String extractPhase(HasMetadata resource) { + if (resource instanceof Pod pod) { + return pod.getStatus() != null ? pod.getStatus().getPhase() : null; + } + + // Generic / Custom Resources + Map status = Serialization.unmarshal(Serialization.asJson(resource), MAP_TYPE); + Map statusMap = MapUtils.asStringObjectMap(status.get("status")); + return statusMap != null ? (String) statusMap.get("phase") : null; + } + + private T waitForResourceWithRetry(String resourceDescription, Supplier fetchSupplier) { + int tryCount = 0; + T result = null; + + while (result == null && tryCount < defaultRetries) { + try { + result = fetchSupplier.get(); + } catch (Exception e) { + log.trace("Error fetching {}: {}", resourceDescription, e.getMessage()); + } + + if (result == null) { + tryCount++; + log.debug("Still waiting for {}... (try {}/{})", resourceDescription, tryCount, defaultRetries); + try { + Thread.sleep(sleepTimeMillis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting", e); + } + } + } + + if (result == null) { + throw new IllegalStateException("Failed to retrieve " + resourceDescription + " after " + defaultRetries + " retries"); + } + + return result; + } + + private static T executeWithErrorHandling(String operation, Supplier supplier) { + try { + return supplier.get(); + } catch (Exception e) { + throw new RuntimeException("Failed to " + operation + ": " + e.getMessage(), e); + } + } + + private static String resolveNamespace(String namespace) { + return namespace != null && !namespace.isEmpty() ? namespace : DEFAULT_NAMESPACE; + } + + /** + * Returns the namespace the client currently operates in. + * + * @return the current namespace from the kubeconfig context + */ + public String getCurrentNamespace() { + return this.client.getNamespace(); + } + + private boolean runInOpenshift() { + return this.gopConfig != null && this.gopConfig.getApplication() != null && this.gopConfig.getApplication() + .getOpenshift(); + } + + /** + * Namespace/name coordinate of a custom resource as returned by {@link #getCustomResource}. + * + * @param namespace namespace the resource lives in; empty for cluster-scoped resources + * @param name name of the resource + */ + public record CustomResource( + String namespace, + + String name + ) { + } + + /** + * Thrown when a custom resource type cannot be resolved via Kubernetes API discovery. + */ + public static class KubernetesApiResourceNotFoundException extends RuntimeException { + /** + * Creates the exception for the given unresolvable type. + * + * @param resourceType the custom resource type that could not be found + */ + public KubernetesApiResourceNotFoundException(String resourceType) { + super("No API resource found for custom resource type '" + resourceType + "'"); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientHelper.java b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientHelper.java new file mode 100644 index 000000000..65f2035df --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientHelper.java @@ -0,0 +1,497 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.api; + +import com.cloudogu.gitops.utils.MapUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import io.fabric8.kubernetes.api.model.APIGroup; +import io.fabric8.kubernetes.api.model.APIGroupList; +import io.fabric8.kubernetes.api.model.APIResource; +import io.fabric8.kubernetes.api.model.APIResourceList; +import io.fabric8.kubernetes.api.model.GroupVersionForDiscovery; +import io.fabric8.kubernetes.api.model.HasMetadata; +import io.fabric8.kubernetes.api.model.Pod; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.dsl.base.PatchContext; +import io.fabric8.kubernetes.client.dsl.base.PatchType; +import io.fabric8.kubernetes.client.dsl.base.ResourceDefinitionContext; +import io.fabric8.kubernetes.client.utils.Serialization; +import io.micronaut.core.util.StringUtils; +import lombok.extern.slf4j.Slf4j; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; + +@Slf4j +class K8sClientHelper { + + private static final TypeReference> MAP_TYPE = new TypeReference<>() { + }; + + private static final String GROUP_KEY = "group"; + private static final String VERSION_KEY = "version"; + private static final String KIND_KEY = "kind"; + private static final String PLURAL_KEY = "plural"; + private static final String NAMESPACED_KEY = "namespaced"; + + private K8sClientHelper() { + } + + static Pod applyPodOverrides(Pod pod, Map overrides) { + Map podAsMap = Serialization.unmarshal(Serialization.asJson(pod), MAP_TYPE); + Map normalizedOverrides = MapUtils.asStringObjectMap(normalizeOverrideValue(overrides)); + Map mergedPod = MapUtils.deepMerge(normalizedOverrides, podAsMap); + return Serialization.unmarshal(Serialization.asJson(mergedPod), Pod.class); + } + + static Object normalizeOverrideValue(Object value) { + if (value instanceof CharSequence) { + return value.toString(); + } + + if (value instanceof Map) { + Map result = new LinkedHashMap<>(); + ((Map) value).forEach((k, v) -> result.put(k.toString(), normalizeOverrideValue(v))); + return result; + } + + if (value instanceof Collection) { + List result = new ArrayList<>(); + for (Object entry : (Collection) value) { + result.add(normalizeOverrideValue(entry)); + } + return result; + } + + return value; + } + + static void applyRunParams(Pod pod, List params) { + String restartPolicy = null; + for (String param : params) { + if (param.startsWith("--restart=")) { + restartPolicy = param.substring("--restart=".length()); + break; + } + } + if (restartPolicy != null) { + pod.getSpec().setRestartPolicy(restartPolicy); + } + } + + static boolean shouldReturnPodOutput(List params) { + return params.contains("--rm") || params.contains("-i") || params.contains("-it") || params.contains("-ti"); + } + + static boolean shouldRemovePod(List params) { + return params.contains("--rm"); + } + + static String collectPodRunOutput( + KubernetesClient client, + String podName, + String namespace, + boolean removePod, + int defaultRetries, + int sleepTime, + K8sClient k8sClient) { + String phase; + try { + phase = waitForPodCompletion(client, podName, namespace, defaultRetries, sleepTime); + String logOutput = client.pods().inNamespace(namespace).withName(podName).getLog(); + if (logOutput == null) { + logOutput = ""; + } + + if ("Failed".equals(phase)) { + throw new IllegalStateException("Pod " + podName + " failed:\n" + logOutput); + } + + return logOutput; + } finally { + if (removePod) { + k8sClient.delete("pod", namespace, podName); + } + } + } + + static String waitForPodCompletion( + KubernetesClient client, + String podName, + String namespace, + int defaultRetries, + int sleepTime) { + int tryCount = 0; + + while (tryCount < defaultRetries) { + Pod pod = client.pods().inNamespace(namespace).withName(podName).get(); + + String phase = (pod != null && pod.getStatus() != null) ? pod.getStatus().getPhase() : null; + if ("Succeeded".equals(phase) || "Failed".equals(phase)) { + return phase; + } + + tryCount++; + log.debug("Still waiting for pod/{} to complete... (try {}/{})", podName, tryCount, defaultRetries); + try { + Thread.sleep(sleepTime); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted while waiting for pod completion", e); + } + } + + throw new IllegalStateException("Failed to retrieve completed pod/" + podName + " after " + defaultRetries + " retries"); + } + + static PatchContext createPatchContext(String type) { + PatchType patchType = type == null || type.isEmpty() ? PatchType.JSON_MERGE : switch (type.toLowerCase( + Locale.ROOT)) { + case "merge", "json-merge" -> PatchType.JSON_MERGE; + case "strategic" -> PatchType.STRATEGIC_MERGE; + case "json" -> PatchType.JSON; + default -> throw new IllegalArgumentException("Unsupported patch type: " + type); + }; + + return new PatchContext.Builder().withPatchType(patchType).build(); + } + + static void validateNamespaceName(String name) { + if (name == null || name.trim().isEmpty()) { + throw new IllegalArgumentException("Namespace name must be provided and cannot be null or empty."); + } + } + + static void validateServiceNodePortPatch(String serviceName, String namespace, String portName, int newNodePort) { + if (StringUtils.isEmpty(serviceName) || StringUtils.isEmpty(namespace) || StringUtils.isEmpty(portName) || newNodePort <= 0) { + throw new IllegalArgumentException("Service name, namespace, port name, and valid nodePort must be provided"); + } + } + + static void validateWaitForResourcePhaseParams( + String resourceType, + String resourceName, + String namespace, + String desiredPhase, + int timeoutSeconds, + int checkIntervalSeconds) { + if (StringUtils.isEmpty(resourceType) || StringUtils.isEmpty(resourceName) || StringUtils.isEmpty(namespace) || StringUtils.isEmpty( + desiredPhase)) { + throw new IllegalArgumentException("Resource type, name, namespace, and desired phase must be provided"); + } + if (timeoutSeconds <= 0 || checkIntervalSeconds <= 0) { + throw new IllegalArgumentException("Timeout and check interval must be greater than zero"); + } + } + + @SuppressWarnings("unchecked") + static io.fabric8.kubernetes.client.dsl.Resource getResourceClient( + KubernetesClient client, + String resourceType, + String name, + String resolvedNamespace) { + return (io.fabric8.kubernetes.client.dsl.Resource) resolveResourceClient( + client, + resourceType, + name, + resolvedNamespace + ); + } + + private static io.fabric8.kubernetes.client.dsl.Resource resolveResourceClient( + KubernetesClient client, + String resourceType, + String name, + String resolvedNamespace) { + return switch (resourceType.toLowerCase(Locale.ROOT)) { + case "pod", "pods" -> client.pods().inNamespace(resolvedNamespace).withName(name); + case "service", "services", "svc" -> client.services().inNamespace(resolvedNamespace).withName(name); + case "deployment", "deployments" -> + client.apps().deployments().inNamespace(resolvedNamespace).withName(name); + case "configmap", "configmaps", "cm" -> client.configMaps().inNamespace(resolvedNamespace).withName(name); + case "secret", "secrets" -> client.secrets().inNamespace(resolvedNamespace).withName(name); + case "namespace", "namespaces", "ns" -> client.namespaces().withName(name); + case "node", "nodes" -> client.nodes().withName(name); + case "serviceaccount", "serviceaccounts" -> + client.serviceAccounts().inNamespace(resolvedNamespace).withName(name); + default -> { + log.debug( + "Searching API resource via discovery for resourceType={}, name={}, ns={}", + resourceType, + name, + resolvedNamespace + ); + yield getCustomResourceClient(client, resourceType, name, resolvedNamespace); + } + }; + } + + static io.fabric8.kubernetes.client.dsl.Resource getCustomResourceClient( + KubernetesClient client, + String resourceType, + String name, + String namespace) { + String normalized = resourceType.toLowerCase(Locale.ROOT); + + Map match = findApiResourceViaDiscovery(client, normalized, resourceType); + + if (match.isEmpty()) { + throw new K8sClient.KubernetesApiResourceNotFoundException(resourceType); + } + + log.debug( + "Resolved '{}' via discovery to {}/{} kind={} plural={} namespaced={}", + resourceType, + match.get(GROUP_KEY), + match.get(VERSION_KEY), + match.get(KIND_KEY), + match.get(PLURAL_KEY), + match.get(NAMESPACED_KEY) + ); + + ResourceDefinitionContext context = toResourceDefinitionContext(match); + boolean namespaced = Boolean.TRUE.equals(match.get(NAMESPACED_KEY)); + + // type is MixedOperation>; kept as `var` deliberately. + var resourceClient = client.genericKubernetesResources(context); + return namespaced ? resourceClient.inNamespace(namespace).withName(name) : resourceClient.withName(name); + } + + private static ResourceDefinitionContext toResourceDefinitionContext(Map match) { + return new ResourceDefinitionContext.Builder().withGroup((String) match.get(GROUP_KEY)) + .withVersion((String) match.get(VERSION_KEY)) + .withKind((String) match.get(KIND_KEY)) + .withPlural((String) match.get(PLURAL_KEY)) + .withNamespaced(Boolean.TRUE.equals(match.get(NAMESPACED_KEY))) + .build(); + } + + static Map findApiResourceViaDiscovery( + KubernetesClient client, + String normalized, + String original) { + for (APIGroup group : fetchApiGroups(client)) { + Map match = findApiResourceInGroup(client, group, normalized, original); + if (!match.isEmpty()) { + return match; + } + } + return Collections.emptyMap(); + } + + static ResourceDefinitionContext resolveResourceDefinitionContext( + KubernetesClient client, + String resourceType) { + Map match = findApiResourceViaDiscovery( + client, + resourceType.toLowerCase(Locale.ROOT), + resourceType + ); + + if (!match.isEmpty()) { + return toResourceDefinitionContext(match); + } + + ResourceDefinitionContext context = resolveResourceDefinitionContextViaCrd(client, resourceType); + if (context != null) { + return context; + } + + throw new K8sClient.KubernetesApiResourceNotFoundException(resourceType); + } + + private static ResourceDefinitionContext resolveResourceDefinitionContextViaCrd( + KubernetesClient client, + String resourceType) { + try { + var crdList = client.apiextensions().v1().customResourceDefinitions().list(); + if (crdList == null || crdList.getItems() == null) { + return null; + } + + for (var crd : crdList.getItems()) { + var spec = crd.getSpec(); + if (spec == null || spec.getNames() == null || spec.getVersions() == null) { + continue; + } + + var names = spec.getNames(); + boolean matches = resourceType.equalsIgnoreCase(names.getKind()) + || resourceType.equalsIgnoreCase(names.getPlural()) + || resourceType.equalsIgnoreCase(names.getSingular()); + if (!matches) { + continue; + } + + String version = null; + for (var candidate : spec.getVersions()) { + if (Boolean.TRUE.equals(candidate.getServed()) && version == null) { + version = candidate.getName(); + } + if (Boolean.TRUE.equals(candidate.getServed()) && Boolean.TRUE.equals(candidate.getStorage())) { + version = candidate.getName(); + break; + } + } + + if (version == null) { + continue; + } + + log.debug( + "Resolved '{}' from CRD because API discovery did not return it", + resourceType + ); + return new ResourceDefinitionContext.Builder().withGroup(spec.getGroup()) + .withVersion(version) + .withKind(names.getKind()) + .withPlural(names.getPlural()) + .withNamespaced("Namespaced".equalsIgnoreCase(spec.getScope())) + .build(); + } + } catch (Exception e) { + log.trace("Failed to resolve resource '{}' from CRDs: {}", resourceType, e.getMessage()); + } + + return null; + } + + private static List fetchApiGroups(KubernetesClient client) { + try { + APIGroupList groupList = client.getApiGroups(); + return groupList != null ? groupList.getGroups() : Collections.emptyList(); + } catch (Exception e) { + log.warn("Failed to discover API groups: {}", e.getMessage()); + return Collections.emptyList(); + } + } + + private static Map findApiResourceInGroup( + KubernetesClient client, + APIGroup group, + String normalized, + String original) { + for (String version : groupVersions(group)) { + APIResource resolved = findMatchingResourceInVersion(client, group, version, normalized, original); + if (resolved != null) { + return toResourceMatch(group, version, resolved); + } + } + return Collections.emptyMap(); + } + + private static List groupVersions(APIGroup group) { + List versions = new ArrayList<>(); + if (group.getPreferredVersion() != null && group.getPreferredVersion().getVersion() != null) { + versions.add(group.getPreferredVersion().getVersion()); + } + if (group.getVersions() != null) { + for (GroupVersionForDiscovery v : group.getVersions()) { + if (v.getVersion() != null && !versions.contains(v.getVersion())) { + versions.add(v.getVersion()); + } + } + } + return versions; + } + + private static APIResource findMatchingResourceInVersion( + KubernetesClient client, + APIGroup group, + String version, + String normalized, + String original) { + for (APIResource res : fetchApiResources(client, group, version)) { + if (isTopLevelResource(res) && matchesResource(res, normalized, original)) { + return res; + } + } + return null; + } + + private static List fetchApiResources(KubernetesClient client, APIGroup group, String version) { + try { + APIResourceList resourceList = client.getApiResources(group.getName() + "/" + version); + return resourceList != null ? resourceList.getResources() : Collections.emptyList(); + } catch (Exception e) { + log.trace("Failed to fetch {}/{}: {}", group.getName(), version, e.getMessage()); + return Collections.emptyList(); + } + } + + private static boolean isTopLevelResource(APIResource res) { + return res.getName() != null && !res.getName().contains("/"); + } + + private static boolean matchesResource(APIResource res, String normalized, String original) { + boolean match = res.getKind().equalsIgnoreCase(original) || res.getName() + .equalsIgnoreCase(normalized) || (res.getSingularName() != null && res.getSingularName() + .equalsIgnoreCase( + normalized)); + if (match || res.getShortNames() == null) { + return match; + } + for (String shortName : res.getShortNames()) { + if (shortName.equalsIgnoreCase(normalized)) { + return true; + } + } + return false; + } + + private static Map toResourceMatch(APIGroup group, String version, APIResource resolved) { + Map map = new HashMap<>(); + map.put(GROUP_KEY, group.getName()); + map.put(VERSION_KEY, version); + map.put(KIND_KEY, resolved.getKind()); + map.put(PLURAL_KEY, resolved.getName()); + map.put(NAMESPACED_KEY, resolved.getNamespaced()); + return map; + } + + static void deleteResourcesByType( + KubernetesClient client, + String resource, + String namespace, + Map labels) { + switch (resource.toLowerCase(Locale.ROOT)) { + case "secret", "secrets": + client.secrets().inNamespace(namespace).withLabels(labels).delete(); + break; + + case "pod", "pods": + client.pods().inNamespace(namespace).withLabels(labels).delete(); + break; + + case "service", "services", "svc": + client.services().inNamespace(namespace).withLabels(labels).delete(); + break; + + case "deployment", "deployments": + client.apps().deployments().inNamespace(namespace).withLabels(labels).delete(); + break; + + case "configmap", "configmaps", "cm": + client.configMaps().inNamespace(namespace).withLabels(labels).delete(); + break; + + default: + Map match = findApiResourceViaDiscovery( + client, + resource.toLowerCase(Locale.ROOT), + resource + ); + if (!match.isEmpty()) { + ResourceDefinitionContext context = toResourceDefinitionContext(match); + client.genericKubernetesResources(context).inNamespace(namespace).withLabels(labels).delete(); + } else { + log.warn("Failed to find resource definition for deletion of {}", resource); + } + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinition.java b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinition.java new file mode 100644 index 000000000..37556627d --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinition.java @@ -0,0 +1,104 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.rbac; + +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.utils.TemplatingEngine; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +@RequiredArgsConstructor +@Slf4j +public class RbacDefinition { + + private final Role.Variant variant; + private String name; + private String namespace; + private List serviceAccounts = new ArrayList<>(); + private String subfolder = "rbac"; + private GitRepo repo; + private Map config; + + private final TemplatingEngine templater = new TemplatingEngine(); + + public RbacDefinition withName(String name) { + this.name = name; + return this; + } + + public RbacDefinition withNamespace(String namespace) { + this.namespace = namespace; + return this; + } + + public RbacDefinition withServiceAccounts(List accounts) { + this.serviceAccounts = new ArrayList<>(accounts); + return this; + } + + public RbacDefinition withServiceAccountsFrom(String saNamespace, List saNames) { + return withServiceAccounts(ServiceAccountRef.fromNames(saNamespace, saNames)); + } + + public RbacDefinition withSubfolder(String subfolder) { + this.subfolder = subfolder; + return this; + } + + public RbacDefinition withRepo(GitRepo repo) { + this.repo = repo; + return this; + } + + public RbacDefinition withTemplateConfig(Map templateConfig) { + this.config = templateConfig; + return this; + } + + public void generate() { + if (repo == null) { + throw new IllegalStateException("SCMM repo must be set using withRepo() before calling generate()"); + } + + log.trace("Generating RBAC for name='{}', namespace='{}', subfolder='{}'", name, namespace, subfolder); + + File outputDir = Path.of(repo.getAbsoluteLocalRepoTmpDir(), subfolder).toFile(); + outputDir.mkdirs(); + + generateRole(outputDir); + generateRoleBinding(outputDir); + } + + private void generateRole(File outputDir) { + if (variant == Role.Variant.CLUSTER_ADMIN) { + log.trace("Skipping creation of ClusterRole cluster-admin"); + return; + } + + Role role = new Role(name, namespace, variant, config); + + try { + templater.template(role.getTemplateFile(), role.getOutputFile(outputDir), role.toTemplateParams()); + } catch (Exception e) { + throw new RuntimeException("Failed to generate role template", e); + } + } + + private void generateRoleBinding(File outputDir) { + String roleName = name; + if (variant == Role.Variant.CLUSTER_ADMIN) { + roleName = "cluster-admin"; + } + RoleBinding binding = new RoleBinding(name, namespace, roleName, serviceAccounts); + + try { + templater.template(binding.getTemplateFile(), binding.getOutputFile(outputDir), binding.toTemplateParams()); + } catch (Exception e) { + throw new RuntimeException("Failed to generate role binding template", e); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/Role.java b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/Role.java new file mode 100644 index 000000000..4006d80fd --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/Role.java @@ -0,0 +1,64 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.rbac; + +import java.io.File; +import java.util.Map; + +public record Role( + String name, + + String namespace, + + Variant variant, + + Map config +) { + + public Role { + if (name == null || name.trim().isEmpty()) { + throw new IllegalArgumentException("Role name must not be blank"); + } + if (namespace == null || namespace.trim().isEmpty()) { + throw new IllegalArgumentException("Role namespace must not be blank"); + } + if (variant == null) { + throw new IllegalArgumentException("Role variant must not be null"); + } + if (config == null) { + throw new IllegalArgumentException("Config must not be null"); + } + } + + public enum Variant { + ARGOCD("templates/kubernetes/rbac/argocd-role.ftl.yaml"), + CLUSTER_ADMIN(""); + + private final String templatePath; + + Variant(String templatePath) { + this.templatePath = templatePath; + } + + public String getTemplatePath() { + return templatePath; + } + } + + public Map toTemplateParams() { + return Map.of("name", name, "namespace", namespace, "config", config); + } + + public File getTemplateFile() { + if (variant == Variant.CLUSTER_ADMIN) { + throw new IllegalStateException("cluster-admin role shall not be created"); + } + return new File(variant.getTemplatePath()); + } + + public File getOutputFile(File outputDir) { + if (variant == Variant.CLUSTER_ADMIN) { + throw new IllegalStateException("cluster-admin role shall not be created"); + } + String filename = "role-" + name + "-" + namespace + ".yaml"; + return new File(outputDir, filename); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RoleBinding.java b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RoleBinding.java new file mode 100644 index 000000000..e1114625b --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RoleBinding.java @@ -0,0 +1,78 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.rbac; + +import lombok.Getter; + +import java.io.File; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +@Getter +public class RoleBinding { + private final String name; + private final String kind; + private final String namespace; + private final String roleName; + private final String roleKind; + private final List serviceAccounts; + + public RoleBinding(String name, String namespace, String roleName, List serviceAccounts) { + if (name == null || name.trim().isEmpty()) { + throw new IllegalArgumentException("RoleBinding name must not be blank"); + } + if (namespace == null || namespace.trim().isEmpty()) { + throw new IllegalArgumentException("RoleBinding namespace must not be blank"); + } + if (roleName == null || roleName.trim().isEmpty()) { + throw new IllegalArgumentException("Role name must not be blank"); + } + if (serviceAccounts == null || serviceAccounts.isEmpty()) { + throw new IllegalArgumentException("At least one service account is required"); + } + + this.name = name; + this.namespace = namespace; + this.roleName = roleName; + this.serviceAccounts = new ArrayList<>(serviceAccounts); + + if (roleName.equals("cluster-admin")) { + this.kind = "ClusterRoleBinding"; + this.roleKind = "ClusterRole"; + } else { + this.kind = "RoleBinding"; + this.roleKind = "Role"; + } + } + + public Map toTemplateParams() { + return Map.of( + "name", + name, + "kind", + kind, + "namespace", + namespace, + "roleName", + roleName, + "roleKind", + roleKind, + "serviceAccounts", + serviceAccounts.stream() + .map(ServiceAccountRef::toMap) + .toList() + ); + } + + public String getTemplatePath() { + return "templates/kubernetes/rbac/rolebinding.ftl.yaml"; + } + + public File getTemplateFile() { + return new File(getTemplatePath()); + } + + public File getOutputFile(File outputDir) { + String filename = "rolebinding-" + name + "-" + namespace + ".yaml"; + return new File(outputDir, filename); + } +} diff --git a/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/ServiceAccountRef.java b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/ServiceAccountRef.java new file mode 100644 index 000000000..fbb429fe1 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/ServiceAccountRef.java @@ -0,0 +1,42 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.rbac; + +import lombok.Getter; + +import java.util.List; +import java.util.Map; + +@Getter +public class ServiceAccountRef { + private final String name; + private final String namespace; + + public ServiceAccountRef(String name, String namespace) { + if (name == null || name.trim().isEmpty()) { + throw new IllegalArgumentException("ServiceAccount name must not be blank"); + } + if (namespace == null || namespace.trim().isEmpty()) { + throw new IllegalArgumentException("ServiceAccount namespace must not be blank"); + } + this.name = name; + this.namespace = namespace; + } + + public static List fromNames(String namespace, List names) { + if (namespace == null || namespace.trim().isEmpty()) { + throw new IllegalArgumentException("Namespace must not be blank for service accounts"); + } + if (names == null) { + return List.of(); + } + + return names.stream() + .filter(name -> name != null && !name.trim().isEmpty()) + .distinct() + .map(name -> new ServiceAccountRef(name, namespace)) + .toList(); + } + + public Map toMap() { + return Map.of("name", name, "namespace", namespace); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/CertManager.java b/src/main/java/com/cloudogu/gitops/tools/CertManager.java new file mode 100644 index 000000000..809c78f56 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/CertManager.java @@ -0,0 +1,97 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.FileSystemUtils; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.extern.slf4j.Slf4j; + +import java.util.Map; + +@Singleton +@Order(160) +@Slf4j +public class CertManager extends AbstractMappedTool { + + public static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/cert-manager/templates/values.ftl.yaml"; + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TOOL_NAME = "cert-manager"; + private static final String CERT_MANAGER_APP_PATH = "apps/cert-manager"; + + private final ImagePullSecretCreator imagePullSecretCreator; + private String namespace; + + public CertManager( + FileSystemUtils fileSystemUtils, + Deployer deployer, + AirGappedUtils airGappedUtils, + GitHandler gitHandler, + ImagePullSecretCreator imagePullSecretCreator, + CertManagerToolConfigMapper configMapper) { + super(configMapper); + this.fileSystemUtils = fileSystemUtils; + this.deployer = deployer; + this.airGappedUtils = airGappedUtils; + this.gitHandler = gitHandler; + this.imagePullSecretCreator = imagePullSecretCreator; + } + + @Override + protected boolean isEnabled(CertManagerToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + this.namespace = activeNamespace(toolConfig()); + + createImagePullSecret(); + prepareCertManagerApp(repositoryWorkspace.getClusterResourcesRepository()); + replaceCertManagerTemplates(repositoryWorkspace.getClusterResourcesRepository()); + } + + @Override + protected void deploy() { + addHelmValuesData("config", toolConfig().templateConfig()); + deployHelmChart(TOOL_NAME, TOOL_NAME, namespace, toolConfig().helm(), HELM_VALUES_PATH, context); + } + + @Override + protected void publishChanges() { + publishClusterResourcesChanges(TOOL_NAME); + } + + @Override + protected String activeNamespace(CertManagerToolConfig config) { + return config.namespace(); + } + + @Override + public String getNamespace() { + return namespace; + } + + private void createImagePullSecret() { + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + } + + private void prepareCertManagerApp(GitRepo clusterResourcesRepo) { + log.debug("Preparing cert-manager repository content in {}", clusterResourcesRepo.getRepoTarget()); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, CERT_MANAGER_APP_PATH) + ); + } + + private void replaceCertManagerTemplates(GitRepo clusterResourcesRepo) { + clusterResourcesRepo.replaceTemplates(Map.of("config", toolConfig().templateConfig())); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/CertManagerToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/CertManagerToolConfig.java new file mode 100644 index 000000000..9e568c26b --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/CertManagerToolConfig.java @@ -0,0 +1,22 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Map; + +@Builder +public record CertManagerToolConfig( + boolean active, + String namespace, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public CertManagerToolConfig { + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/CertManagerToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/CertManagerToolConfigMapper.java new file mode 100644 index 000000000..901ed27c9 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/CertManagerToolConfigMapper.java @@ -0,0 +1,49 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class CertManagerToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public CertManagerToolConfig map(DeploymentContext context) { + Config.CertManagerSchema certManager = config.getFeatures().getCertManager(); + return CertManagerToolConfig.builder() + .active(certManager.getActive()) + .namespace(config.getApplication().getNamePrefix() + certManager.getNamespace()) + .helm(ToolConfigMapperSupport.helmChart( + certManager.getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config)) + .build(); + } + + private static Map templateConfig(Config config) { + Config.CertManagerSchema certManager = config.getFeatures().getCertManager(); + Config.CertManagerSchema.CertManagerHelmSchema helm = certManager.getHelm(); + return new TemplateConfig() + .put("application.podResources", config.getApplication().getPodResources()) + .put("application.skipCrds", config.getApplication().getSkipCrds()) + .put("features.certManager.issuer", certManager.getIssuer()) + .put("features.certManager.helm.image", helm.getImage()) + .put("features.certManager.helm.webhookImage", helm.getWebhookImage()) + .put("features.certManager.helm.cainjectorImage", helm.getCainjectorImage()) + .put("features.certManager.helm.acmeSolverImage", helm.getAcmeSolverImage()) + .put("features.certManager.helm.startupAPICheckImage", helm.getStartupAPICheckImage()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .values(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperator.java b/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperator.java new file mode 100644 index 000000000..dca91c61c --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperator.java @@ -0,0 +1,91 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.FileSystemUtils; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +@Singleton +@Order(400) +@Slf4j +public class ExternalSecretsOperator extends AbstractMappedTool { + + public static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/external-secrets/templates/values.ftl.yaml"; + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TOOL_NAME = "external-secrets"; + private static final String RELEASE_NAME = "external-secrets"; + private static final String EXTERNAL_SECRETS_APP_PATH = "apps/external-secrets"; + + private final ImagePullSecretCreator imagePullSecretCreator; + + @Getter + @Setter + private String namespace; + + public ExternalSecretsOperator( + FileSystemUtils fileSystemUtils, + Deployer deployer, + AirGappedUtils airGappedUtils, + GitHandler gitHandler, + ImagePullSecretCreator imagePullSecretCreator, + ExternalSecretsOperatorToolConfigMapper configMapper) { + super(configMapper); + this.deployer = deployer; + this.fileSystemUtils = fileSystemUtils; + this.airGappedUtils = airGappedUtils; + this.gitHandler = gitHandler; + this.imagePullSecretCreator = imagePullSecretCreator; + } + + @Override + protected boolean isEnabled(ExternalSecretsOperatorToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + this.namespace = activeNamespace(toolConfig()); + + createImagePullSecret(); + prepareExternalSecretsApp(repositoryWorkspace.getClusterResourcesRepository()); + } + + @Override + protected void deploy() { + addHelmValuesData("config", toolConfig().templateConfig()); + deployHelmChart(TOOL_NAME, RELEASE_NAME, namespace, toolConfig().helm(), HELM_VALUES_PATH, context); + } + + @Override + protected void publishChanges() { + publishClusterResourcesChanges(TOOL_NAME); + } + + @Override + protected String activeNamespace(ExternalSecretsOperatorToolConfig config) { + return config.namespace(); + } + + private void createImagePullSecret() { + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + } + + private void prepareExternalSecretsApp(GitRepo clusterResourcesRepo) { + log.debug("Preparing external-secrets repository content in {}", clusterResourcesRepo.getRepoTarget()); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, EXTERNAL_SECRETS_APP_PATH) + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfig.java new file mode 100644 index 000000000..3720da452 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfig.java @@ -0,0 +1,22 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Map; + +@Builder +public record ExternalSecretsOperatorToolConfig( + boolean active, + String namespace, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public ExternalSecretsOperatorToolConfig { + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfigMapper.java new file mode 100644 index 000000000..13224248f --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfigMapper.java @@ -0,0 +1,48 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class ExternalSecretsOperatorToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public ExternalSecretsOperatorToolConfig map(DeploymentContext context) { + Config.SecretsSchema secrets = config.getFeatures().getSecrets(); + return ExternalSecretsOperatorToolConfig.builder() + .active(secrets.getActive()) + .namespace(config.getApplication().getNamePrefix() + secrets.getNamespace()) + .helm(ToolConfigMapperSupport.helmChart( + secrets.getExternalSecrets().getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config)) + .build(); + } + + private static Map templateConfig(Config config) { + Config.SecretsSchema.ESOSchema.ESOHelmSchema helm = config.getFeatures() + .getSecrets() + .getExternalSecrets() + .getHelm(); + return new TemplateConfig() + .put("application.podResources", config.getApplication().getPodResources()) + .put("application.skipCrds", config.getApplication().getSkipCrds()) + .put("features.secrets.externalSecrets.helm.image", helm.getImage()) + .put("features.secrets.externalSecrets.helm.certControllerImage", helm.getCertControllerImage()) + .put("features.secrets.externalSecrets.helm.webhookImage", helm.getWebhookImage()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .values(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/Ingress.java b/src/main/java/com/cloudogu/gitops/tools/Ingress.java new file mode 100644 index 000000000..65281ebee --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/Ingress.java @@ -0,0 +1,91 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.FileSystemUtils; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +@Singleton +@Order(150) +@Slf4j +public class Ingress extends AbstractMappedTool { + + public static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/traefik/templates/values.ftl.yaml"; + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TOOL_NAME = "traefik"; + private static final String RELEASE_NAME = "traefik"; + private static final String INGRESS_APP_PATH = "apps/traefik"; + + private final ImagePullSecretCreator imagePullSecretCreator; + + @Getter + @Setter + private String namespace; + + public Ingress( + FileSystemUtils fileSystemUtils, + Deployer deployer, + AirGappedUtils airGappedUtils, + GitHandler gitHandler, + ImagePullSecretCreator imagePullSecretCreator, + IngressToolConfigMapper configMapper) { + super(configMapper); + this.deployer = deployer; + this.fileSystemUtils = fileSystemUtils; + this.airGappedUtils = airGappedUtils; + this.gitHandler = gitHandler; + this.imagePullSecretCreator = imagePullSecretCreator; + } + + @Override + protected boolean isEnabled(IngressToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + this.namespace = activeNamespace(toolConfig()); + + createImagePullSecret(); + prepareIngressApp(repositoryWorkspace.getClusterResourcesRepository()); + } + + @Override + protected void deploy() { + addHelmValuesData("config", toolConfig().templateConfig()); + deployHelmChart(TOOL_NAME, RELEASE_NAME, namespace, toolConfig().helm(), HELM_VALUES_PATH, context); + } + + @Override + protected void publishChanges() { + publishClusterResourcesChanges(TOOL_NAME); + } + + @Override + protected String activeNamespace(IngressToolConfig config) { + return config.namespace(); + } + + private void createImagePullSecret() { + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + } + + private static void prepareIngressApp(GitRepo clusterResourcesRepo) { + log.debug("Preparing ingress repository content in {}", clusterResourcesRepo.getRepoTarget()); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, INGRESS_APP_PATH) + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/IngressToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/IngressToolConfig.java new file mode 100644 index 000000000..956dbd9bc --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/IngressToolConfig.java @@ -0,0 +1,22 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Map; + +@Builder +public record IngressToolConfig( + boolean active, + String namespace, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public IngressToolConfig { + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/IngressToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/IngressToolConfigMapper.java new file mode 100644 index 000000000..ffd4cd76c --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/IngressToolConfigMapper.java @@ -0,0 +1,45 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class IngressToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public IngressToolConfig map(DeploymentContext context) { + Config.IngressSchema ingress = config.getFeatures().getIngress(); + + return IngressToolConfig.builder() + .active(ingress.getActive()) + .namespace(config.getApplication().getNamePrefix() + ingress.getIngressNamespace()) + .helm(ToolConfigMapperSupport.helmChart( + ingress.getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config)) + .build(); + } + + private static Map templateConfig(Config config) { + return new TemplateConfig() + .put("application.namePrefix", config.getApplication().getNamePrefix()) + .put("application.netpols", config.getApplication().getNetpols()) + .put("features.ingress.helm.image", config.getFeatures().getIngress().getHelm().getImage()) + .put("features.monitoring.active", config.getFeatures().getMonitoring().getActive()) + .put("features.monitoring.namespace", config.getFeatures().getMonitoring().getNamespace()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .values(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/Monitoring.java b/src/main/java/com/cloudogu/gitops/tools/Monitoring.java new file mode 100644 index 000000000..b1459da27 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/Monitoring.java @@ -0,0 +1,389 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.cloudogu.gitops.utils.Tuple; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.net.MalformedURLException; +import java.net.URI; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; + +@Singleton +@Order(300) +@Slf4j +public class Monitoring extends AbstractMappedTool { + + public static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml"; + public static final String RBAC_NAMESPACE_ISOLATION_TEMPLATE = "argocd/cluster-resources/apps/monitoring/templates/rbac/namespace-isolation-rbac.ftl.yaml"; + public static final String NETWORK_POLICIES_PROMETHEUS_ALLOW_TEMPLATE = "argocd/cluster-resources/apps/monitoring/templates/netpols/prometheus-allow-scraping.ftl.yaml"; + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TOOL_NAME = "monitoring"; + private static final String RELEASE_NAME = "kube-prometheus-stack"; + private static final String MONITORING_APP_PATH = "apps/monitoring"; + private static final String PASSWORD_KEY = "password"; + private static final String GRAFANA_ADMIN_SECRET = "grafana-admin-credentials"; + private static final String GENERIC_SECRET_TYPE = "generic"; + private static final String NAMESPACE_KEY = "namespace"; + private static final String MONITORING_RBAC_PATH = MONITORING_APP_PATH + "/misc/rbac"; + private static final String MONITORING_NETPOLS_PATH = MONITORING_APP_PATH + "/misc/netpols"; + private static final String MONITORING_DASHBOARD_PATH = MONITORING_APP_PATH + "/misc/dashboard"; + + private final ImagePullSecretCreator imagePullSecretCreator; + private final K8sClient k8sClient; + private final CredentialsResolver credentialsResolver; + private ResolvedCredentials runtimeApplicationCredentials; + private ResolvedCredentials runtimeJenkinsMetricsCredentials; + private ResolvedCredentials runtimeSmtpCredentials; + + @Getter + @Setter + private String namespace; + + public Monitoring( + FileSystemUtils fileSystemUtils, + Deployer deployer, + K8sClient k8sClient, + AirGappedUtils airGappedUtils, + GitHandler gitHandler, + ImagePullSecretCreator imagePullSecretCreator, + MonitoringToolConfigMapper configMapper, + CredentialsResolver credentialsResolver) { + super(configMapper); + this.deployer = deployer; + this.fileSystemUtils = fileSystemUtils; + this.k8sClient = k8sClient; + this.airGappedUtils = airGappedUtils; + this.gitHandler = gitHandler; + this.imagePullSecretCreator = imagePullSecretCreator; + this.credentialsResolver = credentialsResolver; + } + + @Override + protected boolean isEnabled(MonitoringToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + this.namespace = activeNamespace(toolConfig()); + resolveRuntimeCredentials(); + + createImagePullSecret(); + prepareMonitoringHelmValues(); + + // Create secrets imperatively here instead of values.yaml, + // because we don't want credentials to be visible in the Git repo. + setupMonitoringSecrets(); + createMonitoringCrd(); + + prepareMonitoringApp(repositoryWorkspace.getClusterResourcesRepository()); + replaceMonitoringTemplates(repositoryWorkspace.getClusterResourcesRepository()); + writeMonitoringGitOpsArtifacts(repositoryWorkspace.getClusterResourcesRepository()); + } + + @Override + protected void deploy() { + addHelmValuesData("config", toolConfig().templateConfig()); + deployHelmChart(TOOL_NAME, RELEASE_NAME, namespace, toolConfig().helm(), HELM_VALUES_PATH, context); + } + + @Override + protected void publishChanges() { + // We always assume internal monitoring for deploying artifacts + publishClusterResourcesChanges(TOOL_NAME); + } + + @Override + protected String activeNamespace(MonitoringToolConfig config) { + return config.namespace(); + } + + private void createImagePullSecret() { + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + } + + private void prepareMonitoringHelmValues() { + String uid = ""; + if (toolConfig().openshift()) { + uid = findValidOpenShiftUid(); + } + + String grafanaUrl = toolConfig().grafanaUrl(); + String host = ""; + try { + if (grafanaUrl != null && !grafanaUrl.isEmpty()) { + host = URI.create(grafanaUrl).toURL().getHost(); + } + } catch (IllegalArgumentException | MalformedURLException e) { + throw new IllegalArgumentException("Failed to parse Grafana URL: " + grafanaUrl, e); + } + + addHelmValuesData(TOOL_NAME, Map.of("grafana", Map.of("host", host))); + addHelmValuesData( + "namespaces", toolConfig().activeNamespaces() + ); + addHelmValuesData("scm", scmConfigurationMetrics()); + addHelmValuesData("jenkins", jenkinsConfigurationMetrics()); + addHelmValuesData("uid", uid); + } + + private void prepareMonitoringApp(GitRepo clusterResourcesRepo) { + log.debug("Preparing Monitoring repository content in {}", clusterResourcesRepo.getRepoTarget()); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, MONITORING_APP_PATH) + ); + } + + private void replaceMonitoringTemplates(GitRepo clusterResourcesRepo) { + clusterResourcesRepo.replaceTemplates(Map.of("config", toolConfig().templateConfig())); + } + + private void writeMonitoringGitOpsArtifacts(GitRepo clusterResourcesRepo) { + if (toolConfig().namespaceIsolation()) { + generateNamespaceIsolationRBAC(clusterResourcesRepo); + } + + if (toolConfig().netpols()) { + generateNetpols(clusterResourcesRepo); + } + + // Remove dashboards for features that are not enabled + cleanupUnusedDashboards(clusterResourcesRepo); + } + + private void resolveRuntimeCredentials() { + runtimeApplicationCredentials = credentialsResolver.resolveReference( + toolConfig().applicationCredentials(), + toolConfig().applicationUsername(), + toolConfig().applicationPassword() + ); + + if (toolConfig().jenkinsActive()) { + runtimeJenkinsMetricsCredentials = credentialsResolver.resolveReference( + toolConfig().jenkinsMetricsCredentials(), + toolConfig().jenkinsMetricsUsername(), + toolConfig().jenkinsMetricsPassword() + ); + } + + runtimeSmtpCredentials = credentialsResolver.resolveReference( + toolConfig().smtpCredentials(), + toolConfig().smtpUser(), + toolConfig().smtpPassword() + ); + } + + private void setupMonitoringSecrets() { + k8sClient.createSecret( + GENERIC_SECRET_TYPE, GRAFANA_ADMIN_SECRET, namespace, + new Tuple<>("admin-user", runtimeApplicationCredentials.username()), + new Tuple<>("admin-password", runtimeApplicationCredentials.password()) + ); + + if (hasScmManagerMetricsEndpoint()) { + k8sClient.createSecret( + GENERIC_SECRET_TYPE, "prometheus-metrics-creds-scmm", namespace, new Tuple<>( + PASSWORD_KEY, gitHandler.getResourcesScm().getCredentials().getPassword() + ) + ); + } + + if (toolConfig().jenkinsActive()) { + k8sClient.createSecret( + GENERIC_SECRET_TYPE, "prometheus-metrics-creds-jenkins", namespace, new Tuple<>( + PASSWORD_KEY, runtimeJenkinsMetricsCredentials.password() + ) + ); + } + + if (isNotEmpty(runtimeSmtpCredentials.username()) || isNotEmpty(runtimeSmtpCredentials.password())) { + k8sClient.createSecret( + GENERIC_SECRET_TYPE, "grafana-email-secret", namespace, new Tuple<>( + "user", runtimeSmtpCredentials.username() + ), new Tuple<>( + PASSWORD_KEY, runtimeSmtpCredentials.password() + ) + ); + } + } + + private void generateNamespaceIsolationRBAC(GitRepo clusterResourcesRepo) { + for (String currentNamespace : toolConfig().activeNamespaces()) { + try { + String rbacYaml = new TemplatingEngine().template( + new File(RBAC_NAMESPACE_ISOLATION_TEMPLATE), Map.of( + NAMESPACE_KEY, + currentNamespace, + "namePrefix", + toolConfig().namePrefix(), + "config", + toolConfig().templateConfig() + ) + ); + + clusterResourcesRepo.writeFile(MONITORING_RBAC_PATH + "/" + currentNamespace + ".yaml", rbacYaml); + } catch (Exception e) { + throw new RuntimeException("Failed to generate namespace isolation RBAC for " + currentNamespace, e); + } + } + } + + private void generateNetpols(GitRepo clusterResourcesRepo) { + for (String currentNamespace : toolConfig().activeNamespaces()) { + try { + String netpolsYaml = new TemplatingEngine().template( + new File(NETWORK_POLICIES_PROMETHEUS_ALLOW_TEMPLATE), Map.of( + NAMESPACE_KEY, currentNamespace, "namePrefix", toolConfig().namePrefix() + ) + ); + + clusterResourcesRepo.writeFile(MONITORING_NETPOLS_PATH + "/" + currentNamespace + ".yaml", netpolsYaml); + } catch (Exception e) { + throw new RuntimeException("Failed to generate netpols allow template for " + currentNamespace, e); + } + } + } + + private Map scmConfigurationMetrics() { + URI uri = this.gitHandler.getResourcesScm().prometheusMetricsEndpoint(); + return uriComponents(uri); + } + + private static Map uriComponents(URI uri) { + if (uri == null) { + return Map.of("protocol", "", "host", "", "path", ""); + } + return Map.of( + "protocol", + Objects.requireNonNullElse(uri.getScheme(), ""), + "host", + Objects.requireNonNullElse(uri.getAuthority(), ""), + "path", + Objects.requireNonNullElse(uri.getPath(), "") + ); + } + + protected void createMonitoringCrd() { + if (!toolConfig().skipCrds()) { + String serviceMonitorCrdYaml; + if (toolConfig().airgapped()) { + serviceMonitorCrdYaml = Path.of( + toolConfig().helm().localHelmChartFolder() + "/" + toolConfig().helm().chart(), + "charts/crds/crds/crd-servicemonitors.yaml" + ) + .toString(); + } else { + serviceMonitorCrdYaml = "https://raw.githubusercontent.com/prometheus-community/helm-charts/" + "kube-prometheus-stack-" + toolConfig().helm().version() + "/" + "charts/kube-prometheus-stack/charts/crds/crds/crd-servicemonitors.yaml"; + } + + log.debug( + "Applying ServiceMonitor CRD; Argo CD fails if it is not there. Chicken-egg-problem.\n" + "Applying from path {}", + serviceMonitorCrdYaml + ); + k8sClient.applyYaml(serviceMonitorCrdYaml); + } + } + + private Map jenkinsConfigurationMetrics() { + URI uri = baseUriJenkins(toolConfig()).resolve("prometheus"); + Map components = new HashMap<>(uriComponents(uri)); + components.put( + "metricsUsername", runtimeJenkinsMetricsCredentials != null + && runtimeJenkinsMetricsCredentials.username() != null + ? runtimeJenkinsMetricsCredentials.username() + : "" + ); + return components; + } + + private static URI baseUriJenkins(MonitoringToolConfig config) { + try { + if (config.jenkinsInternal()) { + return new URI("http://jenkins." + config.namePrefix() + config.jenkinsNamespace() + ".svc.cluster.local/"); + } + String urlString = config.jenkinsUrl() != null ? config.jenkinsUrl().trim() : ""; + if (urlString.isEmpty()) { + throw new IllegalArgumentException("config.jenkins.url must be set when config.jenkins.internal = false"); + } + URI url = URI.create(urlString); + return url.toString().endsWith("/") ? url : URI.create(url.toString() + "/"); + } catch (Exception e) { + throw new RuntimeException("Failed to construct base Jenkins URI", e); + } + } + + private String findValidOpenShiftUid() { + String uidRange = k8sClient.getAnnotation(NAMESPACE_KEY, namespace, "openshift.io/sa.scc.uid-range"); + + if (uidRange != null && !uidRange.isEmpty()) { + log.debug("found UID={}", uidRange); + return uidRange.split("/")[0]; + } else { + throw new IllegalStateException("Could not find a valid UID! Really running on OpenShift?"); + } + } + + protected void cleanupUnusedDashboards(GitRepo clusterResourcesRepo) { + String repoRoot = clusterResourcesRepo.getAbsoluteLocalRepoTmpDir(); + String dashboardRoot = repoRoot + "/" + MONITORING_DASHBOARD_PATH; + + if (!toolConfig().ingressActive()) { + FileSystemUtils.deleteFile(dashboardRoot + "/traefik-dashboard.yaml"); + FileSystemUtils.deleteFile(dashboardRoot + "/traefik-dashboard-requests-handling.yaml"); + } + + if (!toolConfig().jenkinsActive()) { + FileSystemUtils.deleteFile(dashboardRoot + "/jenkins-dashboard.yaml"); + } + + if (!hasScmManagerMetricsEndpoint()) { + FileSystemUtils.deleteFile(dashboardRoot + "/scmm-dashboard.yaml"); + } + } + + private boolean hasScmManagerMetricsEndpoint() { + if (toolConfig().scmProviderType() != ScmProviderType.SCM_MANAGER) { + return false; + } + + URI uri = this.gitHandler.getResourcesScm().prometheusMetricsEndpoint(); + + if (uri == null) { + return false; + } + + return hasText(uri.getScheme()) || hasText(uri.getAuthority()) || hasText(uri.getPath()); + } + + private static boolean isNotEmpty(String value) { + return value != null && !value.isEmpty(); + } + + private static boolean hasText(String value) { + return value != null && !value.trim().isEmpty(); + } + +} diff --git a/src/main/java/com/cloudogu/gitops/tools/MonitoringToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/MonitoringToolConfig.java new file mode 100644 index 000000000..a1fd4fb19 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/MonitoringToolConfig.java @@ -0,0 +1,49 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Collection; +import java.util.Map; + +@Builder +public record MonitoringToolConfig( + boolean active, + String namespace, + String namePrefix, + Collection activeNamespaces, + boolean namespaceIsolation, + boolean netpols, + boolean skipCrds, + boolean openshift, + boolean airgapped, + String applicationUsername, + String applicationPassword, + CredentialsReference applicationCredentials, + String jenkinsMetricsUsername, + String jenkinsMetricsPassword, + CredentialsReference jenkinsMetricsCredentials, + String smtpUser, + String smtpPassword, + CredentialsReference smtpCredentials, + String grafanaUrl, + boolean jenkinsInternal, + String jenkinsNamespace, + String jenkinsUrl, + ScmProviderType scmProviderType, + boolean ingressActive, + boolean jenkinsActive, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public MonitoringToolConfig { + activeNamespaces = ImmutableConfigData.copyList(activeNamespaces); + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/MonitoringToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/MonitoringToolConfigMapper.java new file mode 100644 index 000000000..051acb54e --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/MonitoringToolConfigMapper.java @@ -0,0 +1,112 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Collection; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class MonitoringToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public MonitoringToolConfig map(DeploymentContext context) { + Config.MonitoringSchema monitoring = config.getFeatures().getMonitoring(); + Collection activeNamespaces = config.getApplication().getNamespaces().getActiveNamespaces(); + return MonitoringToolConfig.builder() + .active(monitoring.getActive()) + .namespace(config.getApplication().getNamePrefix() + monitoring.getNamespace()) + .namePrefix(config.getApplication().getNamePrefix()) + .activeNamespaces(activeNamespaces) + .namespaceIsolation(config.getApplication().getNamespaceIsolation()) + .netpols(config.getApplication().getNetpols()) + .skipCrds(config.getApplication().getSkipCrds()) + .openshift(context.isOpenshift()) + .airgapped(context.isAirgapped()) + .applicationUsername(config.getApplication().getUsername()) + .applicationPassword(config.getApplication().getPassword()) + .applicationCredentials(CredentialsReference.from(config.getApplication().getCredentials())) + .jenkinsMetricsUsername(config.getJenkins().getMetricsUsername()) + .jenkinsMetricsPassword(config.getJenkins().getMetricsPassword()) + .jenkinsMetricsCredentials(CredentialsReference.from(config.getJenkins().getMetricsCredentials())) + .smtpUser(config.getFeatures().getMail().getSmtpUser()) + .smtpPassword(config.getFeatures().getMail().getSmtpPassword()) + .smtpCredentials(CredentialsReference.from(config.getFeatures().getMail().getCredentials())) + .grafanaUrl(monitoring.getGrafanaUrl()) + .jenkinsInternal(config.getJenkins().getInternal()) + .jenkinsNamespace(config.getJenkins().getNamespace()) + .jenkinsUrl(config.getJenkins().getUrl()) + .scmProviderType(config.getScm() == null ? null : config.getScm().getScmProviderType()) + .ingressActive(config.getFeatures().getIngress().getActive()) + .jenkinsActive(config.getJenkins().getActive()) + .helm(ToolConfigMapperSupport.helmChart( + monitoring.getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config, context)) + .build(); + } + + private static Map templateConfig(Config config, DeploymentContext context) { + Config.MonitoringSchema.MonitoringHelmSchema helm = config.getFeatures().getMonitoring().getHelm(); + String scmManagerNamespace = config.getScm() == null || config.getScm().getScmManager() == null + ? "scm-manager" + : config.getScm().getScmManager().getNamespace(); + return new TemplateConfig() + .put("application.namePrefix", config.getApplication().getNamePrefix()) + .put("application.namespaceIsolation", config.getApplication().getNamespaceIsolation()) + .put("application.openshift", context.isOpenshift()) + .put("application.podResources", config.getApplication().getPodResources()) + .put("application.skipCrds", config.getApplication().getSkipCrds()) + .put("features.certManager.active", config.getFeatures().getCertManager().getActive()) + .put("features.certManager.issuer", config.getFeatures().getCertManager().getIssuer()) + .put("features.mail.active", config.getFeatures().getMail().getActive()) + .put("features.mail.smtpAddress", config.getFeatures().getMail().getSmtpAddress()) + .put("features.mail.smtpCredentialsConfigured", smtpCredentialsConfigured(config)) + .put("features.mail.smtpPort", config.getFeatures().getMail().getSmtpPort()) + .put("features.monitoring.grafanaEmailFrom", config.getFeatures().getMonitoring().getGrafanaEmailFrom()) + .put("features.monitoring.grafanaEmailTo", config.getFeatures().getMonitoring().getGrafanaEmailTo()) + .put("features.monitoring.grafanaUrl", config.getFeatures().getMonitoring().getGrafanaUrl()) + .put("features.monitoring.namespace", config.getFeatures().getMonitoring().getNamespace()) + .put( + "features.monitoring.oidc", ToolConfigMapperSupport.oidc( + config.getFeatures().getMonitoring().getOidc()) + ) + .put("features.monitoring.helm.grafanaImage", helm.getGrafanaImage()) + .put("features.monitoring.helm.grafanaSidecarImage", helm.getGrafanaSidecarImage()) + .put("features.monitoring.helm.prometheusConfigReloaderImage", helm.getPrometheusConfigReloaderImage()) + .put("features.monitoring.helm.prometheusImage", helm.getPrometheusImage()) + .put("features.monitoring.helm.prometheusOperatorImage", helm.getPrometheusOperatorImage()) + .put("jenkins.active", config.getJenkins().getActive()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .put("scm.scmManager.namespace", scmManagerNamespace) + .put("scm.scmProviderType", config.getScm() == null ? null : config.getScm().getScmProviderType()) + .values(); + } + + private static boolean smtpCredentialsConfigured(Config config) { + return hasText(config.getFeatures().getMail().getSmtpUser()) + || hasText(config.getFeatures().getMail().getSmtpPassword()) + || hasMailSecretReference(config); + } + + private static boolean hasMailSecretReference(Config config) { + var credentials = config.getFeatures().getMail().getCredentials(); + return credentials != null + && (hasText(credentials.getSecretName()) || hasText(credentials.getSecretNamespace())); + } + + private static boolean hasText(String value) { + return value != null && !value.isEmpty(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/Registry.java b/src/main/java/com/cloudogu/gitops/tools/Registry.java new file mode 100644 index 000000000..3f9973332 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/Registry.java @@ -0,0 +1,124 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +import java.util.HashMap; +import java.util.Map; + +@Singleton +@Order(30) +@Slf4j +public class Registry extends AbstractMappedTool { + + /** + * Local container port of the registry within the pod + */ + public static final String CONTAINER_PORT = "5000"; + + private static final String TOOL_NAME = "registry"; + private static final String RELEASE_NAME = "docker-registry"; + + private final K8sClient k8sClient; + + @Getter + @Setter + private String namespace; + + public Registry( + FileSystemUtils fileSystemUtils, K8sClient k8sClient, AirGappedUtils airGappedUtils, + // Bootstrap with Helm first, then create an ArgoCD Application for GitOps management. + Deployer deployer, + RegistryToolConfigMapper configMapper) { + super(configMapper); + this.deployer = deployer; + this.fileSystemUtils = fileSystemUtils; + this.k8sClient = k8sClient; + this.airGappedUtils = airGappedUtils; + } + + @Override + protected boolean isEnabled(RegistryToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + if (!isInternalRegistry()) { + return; + } + + this.namespace = activeNamespace(toolConfig()); + + prepareRegistryHelmValues(); + } + + @Override + protected void deploy() { + if (!isInternalRegistry()) { + return; + } + + deployInternalRegistry(); + createInternalRegistryNodePortIfRequired(); + } + + @Override + protected void publishChanges() { + if (!isInternalRegistry()) { + return; + } + + publishClusterResourcesChanges(TOOL_NAME); + } + + @Override + protected String activeNamespace(RegistryToolConfig config) { + return config.namespace(); + } + + private boolean isInternalRegistry() { + return toolConfig().internal(); + } + + private void prepareRegistryHelmValues() { + Map service = new HashMap<>(); + service.put("nodePort", toolConfig().bootstrapNodePort()); + service.put("type", "NodePort"); + addHelmValuesData("service", service); + } + + private void deployInternalRegistry() { + deployHelmChart( + TOOL_NAME, RELEASE_NAME, namespace, toolConfig().helm(), "", context, true + ); + } + + private void createInternalRegistryNodePortIfRequired() { + if (toolConfig().internalPort() == toolConfig().bootstrapNodePort()) { + return; + } + + /* + * Add additional node port. + * + * 30000 is needed as a static port by Docker via k3d port mapping, + * e.g. 32769 -> 30000 on the server-0 container. + * + * See "-p 30000" in init-cluster.sh. + * e.g. 32769 is needed so the kubelet can access the image inside the server-0 container. + */ + k8sClient.createServiceNodePort( + "docker-registry-internal-port", CONTAINER_PORT + ":" + CONTAINER_PORT, + toolConfig().internalPort().toString(), namespace + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/RegistryToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/RegistryToolConfig.java new file mode 100644 index 000000000..ca6a7ba59 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/RegistryToolConfig.java @@ -0,0 +1,15 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import lombok.Builder; + +@Builder +public record RegistryToolConfig( + boolean active, + boolean internal, + String namespace, + int bootstrapNodePort, + Integer internalPort, + HelmChartConfig helm +) { +} diff --git a/src/main/java/com/cloudogu/gitops/tools/RegistryToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/RegistryToolConfigMapper.java new file mode 100644 index 000000000..4dfe11423 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/RegistryToolConfigMapper.java @@ -0,0 +1,35 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class RegistryToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public RegistryToolConfig map(DeploymentContext context) { + Config.RegistrySchema registry = config.getRegistry(); + String namespace = registry.getInternal() + ? config.getApplication().getNamePrefix() + registry.getNamespace() + : null; + + return RegistryToolConfig.builder() + .active(registry.getActive()) + .internal(registry.getInternal()) + .namespace(namespace) + .bootstrapNodePort(Config.DEFAULT_REGISTRY_PORT) + .internalPort(registry.getInternalPort()) + .helm(ToolConfigMapperSupport.helmChart( + registry.getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .build(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/Vault.java b/src/main/java/com/cloudogu/gitops/tools/Vault.java new file mode 100644 index 000000000..f8846253c --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/Vault.java @@ -0,0 +1,185 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.cloudogu.gitops.utils.Tuple; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.net.MalformedURLException; +import java.net.URI; +import java.nio.file.Path; +import java.util.Map; +import java.util.UUID; + +@Singleton +@Order(500) +@Slf4j +public class Vault extends AbstractMappedTool { + + public static final String VAULT_START_SCRIPT_PATH = "argocd/cluster-resources/apps/vault/templates/dev-post-start.ftl.sh"; + public static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/vault/templates/values.ftl.yaml"; + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TOOL_NAME = "vault"; + private static final String RELEASE_NAME = "vault"; + private static final String VAULT_APP_PATH = "apps/vault"; + private static final String VAULT_USER_CREDENTIALS_SECRET = "vault-user-credentials"; + + private final ImagePullSecretCreator imagePullSecretCreator; + private final K8sClient k8sClient; + private final CredentialsResolver credentialsResolver; + + @Getter + @Setter + private String namespace; + + public Vault( + FileSystemUtils fileSystemUtils, + Deployer deployer, + K8sClient k8sClient, + AirGappedUtils airGappedUtils, + GitHandler gitHandler, + ImagePullSecretCreator imagePullSecretCreator, + VaultToolConfigMapper configMapper, + CredentialsResolver credentialsResolver) { + super(configMapper); + this.deployer = deployer; + this.fileSystemUtils = fileSystemUtils; + this.k8sClient = k8sClient; + this.airGappedUtils = airGappedUtils; + this.gitHandler = gitHandler; + this.imagePullSecretCreator = imagePullSecretCreator; + this.credentialsResolver = credentialsResolver; + } + + @Override + protected boolean isEnabled(VaultToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + this.namespace = activeNamespace(toolConfig()); + + createImagePullSecret(); + prepareVaultApp(repositoryWorkspace.getClusterResourcesRepository()); + replaceVaultTemplates(repositoryWorkspace.getClusterResourcesRepository()); + prepareVaultHelmValues(); + prepareDevModeIfRequired(); + } + + @Override + protected void deploy() { + addHelmValuesData("config", toolConfig().templateConfig()); + deployHelmChart(TOOL_NAME, RELEASE_NAME, namespace, toolConfig().helm(), HELM_VALUES_PATH, context); + } + + @Override + protected void publishChanges() { + publishClusterResourcesChanges(TOOL_NAME); + } + + @Override + protected String activeNamespace(VaultToolConfig config) { + return config.namespace(); + } + + private void createImagePullSecret() { + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + } + + private void prepareVaultHelmValues() { + String url = toolConfig().url(); + try { + addHelmValuesData("host", (url != null && !url.isEmpty()) ? URI.create(url).toURL().getHost() : ""); + } catch (IllegalArgumentException | MalformedURLException e) { + throw new IllegalArgumentException("Failed to parse Vault URL: " + url, e); + } + } + + private void prepareDevModeIfRequired() { + if (!toolConfig().developmentMode()) { + return; + } + + log.debug("WARNING! Vault dev mode is enabled! In this mode, Vault runs entirely in-memory\n" + "and starts unsealed with a single unseal key. "); + + Path templatedFile = fileSystemUtils.copyToTempDir(fileSystemUtils.getRootDir() + "/" + VAULT_START_SCRIPT_PATH); + File postStartScript; + try { + postStartScript = new TemplatingEngine().replaceTemplate( + templatedFile.toFile(), Map.of( + "namePrefix", toolConfig().namePrefix() + ) + ); + } catch (Exception e) { + throw new RuntimeException("Failed to template Vault post-start script", e); + } + + log.debug("Creating namespace for vault, so it can add its secrets there"); + k8sClient.createNamespace(namespace); + + ResolvedCredentials applicationCredentials = credentialsResolver.resolveReference( + toolConfig().applicationCredentials(), + toolConfig().applicationUsername(), + toolConfig().applicationPassword() + ); + k8sClient.createSecret( + "generic", + VAULT_USER_CREDENTIALS_SECRET, + namespace, + new Tuple<>("username", applicationCredentials.username()), + new Tuple<>("password", applicationCredentials.password()) + ); + + // Create config map from init script. + // Init script creates/authorizes secrets, users, service accounts, etc. + String vaultPostStartConfigMap = "vault-dev-post-start"; + String vaultPostStartVolume = "dev-post-start"; + k8sClient.createConfigMapFromFile(vaultPostStartConfigMap, namespace, postStartScript.getAbsolutePath()); + + addHelmValuesData( + "dev", Map.of( + "rootToken", + UUID.randomUUID() + .toString(), + "vaultPostStartConfigMap", + vaultPostStartConfigMap, + "vaultPostStartVolume", + vaultPostStartVolume, + "userCredentialsSecret", + VAULT_USER_CREDENTIALS_SECRET, + "postStartScriptName", + postStartScript.getName() + ) + ); + } + + private void prepareVaultApp(GitRepo clusterResourcesRepo) { + log.debug("Preparing vault repository content in {}", clusterResourcesRepo.getRepoTarget()); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, VAULT_APP_PATH) + ); + } + + private void replaceVaultTemplates(GitRepo clusterResourcesRepo) { + clusterResourcesRepo.replaceTemplates(Map.of("config", toolConfig().templateConfig())); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/VaultToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/VaultToolConfig.java new file mode 100644 index 000000000..92f118619 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/VaultToolConfig.java @@ -0,0 +1,29 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Map; + +@Builder +public record VaultToolConfig( + boolean active, + String namespace, + String namePrefix, + String url, + String applicationUsername, + String applicationPassword, + CredentialsReference applicationCredentials, + boolean developmentMode, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public VaultToolConfig { + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/VaultToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/VaultToolConfigMapper.java new file mode 100644 index 000000000..48a8616ea --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/VaultToolConfigMapper.java @@ -0,0 +1,68 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class VaultToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public VaultToolConfig map(DeploymentContext context) { + Config.SecretsSchema secrets = config.getFeatures().getSecrets(); + return VaultToolConfig.builder() + .active(secrets.getActive()) + .namespace(config.getApplication().getNamePrefix() + secrets.getNamespace()) + .namePrefix(config.getApplication().getNamePrefix()) + .url(secrets.getVault().getUrl()) + .applicationUsername(config.getApplication().getUsername()) + .applicationPassword(config.getApplication().getPassword()) + .applicationCredentials(CredentialsReference.from(config.getApplication().getCredentials())) + .developmentMode(isDevelopmentMode(secrets.getVault().getMode())) + .helm(ToolConfigMapperSupport.helmChart( + secrets.getVault().getHelm(), config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config, context)) + .build(); + } + + private static Map templateConfig(Config config, DeploymentContext context) { + return new TemplateConfig() + .put("application.namePrefix", config.getApplication().getNamePrefix()) + .put("application.namespaceIsolation", config.getApplication().getNamespaceIsolation()) + .put("application.openshift", context.isOpenshift()) + .put("application.podResources", config.getApplication().getPodResources()) + .put("features.argocd.active", config.getFeatures().getArgocd().getActive()) + .put("features.certManager.active", config.getFeatures().getCertManager().getActive()) + .put("features.certManager.issuer", config.getFeatures().getCertManager().getIssuer()) + .put( + "features.secrets.vault.oidc", + ToolConfigMapperSupport.oidc(config.getFeatures().getSecrets().getVault().getOidc()) + ) + .put("features.secrets.vault.helm.image", config.getFeatures().getSecrets().getVault().getHelm().getImage()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .values(); + } + + private static boolean isDevelopmentMode(Config.VaultMode mode) { + if (mode == null) { + return false; + } + + return switch (mode) { + case DEV -> true; + case PROD -> false; + }; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/AbstractMappedTool.java b/src/main/java/com/cloudogu/gitops/tools/common/AbstractMappedTool.java new file mode 100644 index 000000000..16bd41e06 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/AbstractMappedTool.java @@ -0,0 +1,67 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; + +import java.util.Objects; + +/** + * Base class for tools that consume a focused, tool-specific configuration instead of the complete GOP config. + * + *

The mapper is invoked during technical execution preparation, before the lifecycle starts. Lifecycle methods + * access only the mapped configuration through {@link #toolConfig()}. + * + * @param immutable configuration view required by the concrete tool + */ +public abstract class AbstractMappedTool extends AbstractTool { + + private final ToolConfigMapper toolConfigMapper; + private T toolConfig; + + protected AbstractMappedTool(ToolConfigMapper toolConfigMapper) { + this.toolConfigMapper = Objects.requireNonNull( + toolConfigMapper, + "Tool config mapper must not be null" + ); + } + + protected abstract boolean isEnabled(T config); + + protected String activeNamespace(T config) { + return null; + } + + protected final T toolConfig() { + return Objects.requireNonNull( + toolConfig, + "Tool config is only available during and after execution preparation" + ); + } + + @Override + public final boolean isEnabled(DeploymentContext context) { + return isEnabled(mapConfig(context)); + } + + @Override + protected void prepareExecution(DeploymentContext context, RepositoryWorkspace workspace) { + this.toolConfig = null; + super.prepareExecution(context, workspace); + this.toolConfig = mapConfig(context); + } + + @Override + public final String getActiveNamespaceFromFeature(DeploymentContext context) { + T mappedConfig = mapConfig(context); + return isEnabled(mappedConfig) ? activeNamespace(mappedConfig) : null; + } + + private T mapConfig(DeploymentContext context) { + Objects.requireNonNull(context, "Deployment context must not be null"); + + return Objects.requireNonNull( + toolConfigMapper.map(context), + () -> "Tool config mapper returned null for " + getClass().getName() + ); + } +} \ No newline at end of file diff --git a/src/main/java/com/cloudogu/gitops/tools/common/AbstractTool.java b/src/main/java/com/cloudogu/gitops/tools/common/AbstractTool.java new file mode 100644 index 000000000..3561f0766 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/AbstractTool.java @@ -0,0 +1,241 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.MapUtils; +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import freemarker.template.Configuration; +import freemarker.template.DefaultObjectWrapperBuilder; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Path; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; + +@Slf4j +public abstract class AbstractTool { + + private static final ObjectMapper yamlMapper = new ObjectMapper(new YAMLFactory()); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + + protected FileSystemUtils fileSystemUtils; + protected Deployer deployer; + protected AirGappedUtils airGappedUtils; + protected GitHandler gitHandler; + protected DeploymentContext context; + protected RepositoryWorkspace repositoryWorkspace; + protected Map helmValuesTemplateData = new HashMap<>(); + + /** + * Activation check for the current deployment run. + * + *

This method must be side-effect free. Do not add deployment preparation, config mutation or + * workspace access here. + */ + public abstract boolean isEnabled(DeploymentContext context); + + /** + * Executes this tool along its internal lifecycle. + */ + public boolean execute(DeploymentContext context, RepositoryWorkspace workspace) { + prepareExecution(context, workspace); + + log.info("Installing Tool {}", getClass().getSimpleName()); + + validate(); + preDeploy(); + deploy(); + postDeploy(); + publishChanges(); + + log.info("Tool installed: {}", getClass().getSimpleName()); + return true; + } + + /** + * Technical initialization of runtime state. + * + *

This is not a lifecycle phase. AbstractTool-specific preparation belongs into preDeploy(). + */ + protected void prepareExecution(DeploymentContext context, RepositoryWorkspace workspace) { + this.context = context; + this.repositoryWorkspace = workspace; + this.helmValuesTemplateData = new HashMap<>(); + } + + /** + * Lifecycle phase: validate tool-specific configuration and prerequisites. + * + *

Throw a RuntimeException to stop the deployment immediately. + */ + public void validate() { + } + + /** + * Lifecycle phase: prepare deployment inputs and prerequisites. + */ + protected void preDeploy() { + } + + /** + * Lifecycle phase: deploy the tool. + */ + protected void deploy() { + } + + /** + * Lifecycle phase: run follow-up steps after deployment. + */ + protected void postDeploy() { + } + + /** + * Lifecycle phase: publish GitOps repository changes. + */ + protected void publishChanges() { + } + + protected void publishClusterResourcesChanges(String toolName) { + try { + repositoryWorkspace.commitAndPushClusterResourcesChanges("Update " + toolName + " GitOps resources"); + } catch (Exception e) { + throw new RuntimeException("Failed to publish cluster resources changes for " + toolName, e); + } + } + + protected void addHelmValuesData(String key, Object value) { + this.helmValuesTemplateData.put(key, value); + } + + public String getNamespace() { + return null; + } + + /** + * @param context deployment context used to resolve the namespace + */ + protected String activeNamespace(DeploymentContext context) { + return null; + } + + public String getActiveNamespaceFromFeature(DeploymentContext context) { + return isEnabled(context) ? activeNamespace(context) : null; + } + + public static Map templateToMap(String filePath, Map parameters) { + try { + String hydratedString = new TemplatingEngine().template(new File(filePath), parameters); + + if (hydratedString == null || hydratedString.trim().isEmpty()) { + // Otherwise empty array or exception, whereas we expect a Map + return Collections.emptyMap(); + } + return yamlMapper.readValue(hydratedString, YAML_MAP_TYPE); + } catch (Exception e) { + throw new RuntimeException("Failed to template file to map: " + filePath, e); + } + } + + protected void deployHelmChart( + String featureName, + String releaseName, + String namespace, + HelmChartConfig helmConfig, + String helmValuesTemplatePath, + DeploymentContext context) { + deployHelmChart(featureName, releaseName, namespace, helmConfig, helmValuesTemplatePath, context, false); + } + + protected void deployHelmChart( + String featureName, + String releaseName, + String namespace, + HelmChartConfig helmConfig, + String helmValuesTemplatePath, + DeploymentContext context, + boolean initByHelm) { + try { + this.addHelmValuesData( + "statics", new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build() + .getStaticModels() + ); + } catch (Exception e) { + throw new RuntimeException("Failed to retrieve Freemarker static models for template mapping", e); + } + + /* + * If we get a helmValuesTemplatePath we render the Template with the given Data. + * Some Features might not use a values template and thus passing no helmValuesTemplatePath, + * in that case we simply treat helmValuesTemplateData directly as helmValuesData. + */ + Map helmValuesData = this.helmValuesTemplateData; + if (helmValuesTemplatePath != null && !helmValuesTemplatePath.isEmpty()) { + if (helmValuesTemplatePath.contains(".ftl")) { + log.debug("Rendering helm values template from {}", helmValuesTemplatePath); + helmValuesData = templateToMap(helmValuesTemplatePath, this.helmValuesTemplateData); + } else { + log.debug("Reading plain helm values YAML from {}", helmValuesTemplatePath); + helmValuesData = fileSystemUtils.readYaml(Path.of(helmValuesTemplatePath)); + } + } + + helmValuesData = MapUtils.deepMerge(helmConfig.values(), helmValuesData); + + String repoURL = helmConfig.repoURL(); + String chartOrPath = helmConfig.chart(); + String version = helmConfig.version(); + RepoType repoType = RepoType.HELM; + + if (context.isAirgapped()) { + log.debug("Using a local, mirrored git repo as deployment source for feature {}", featureName); + + String repoNamespaceAndName = this.airGappedUtils.mirrorHelmRepoToGit(helmConfig); + repoURL = this.gitHandler.getResourcesScm().repoUrl(repoNamespaceAndName); + chartOrPath = "."; + repoType = RepoType.GIT; + try { + Map chartYaml = yamlMapper.readValue( + Path.of( + helmConfig.localHelmChartFolder(), helmConfig.chart(), "Chart.yaml" + ) + .toFile(), YAML_MAP_TYPE + ); + version = String.valueOf(chartYaml.get("version")); + } catch (IOException e) { + throw new UncheckedIOException("Failed to parse Chart.yaml for airgapped version mapping", e); + } + } + + log.debug("Starting deployment of feature {} from {}.", featureName, repoURL); + log.debug("helm values used: {}", helmValuesData); + + Path tempValuesPath = this.fileSystemUtils.writeTempFile(helmValuesData); + this.deployer.deployFeature( + repoURL, + featureName, + chartOrPath, + version, + namespace, + releaseName, + tempValuesPath, + repoType, + initByHelm, + context, + repositoryWorkspace + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/CommonToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/common/CommonToolConfig.java new file mode 100644 index 000000000..3c82c8937 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/CommonToolConfig.java @@ -0,0 +1,29 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.config.Config; + +public class CommonToolConfig implements ConfigLifecycleHook { + + @Override + public void preConfigInit(Config configToSet) { + validateConfig(configToSet); + } + + /** + * Make sure that config does not contain contradictory values. Throws RuntimeException with + * meaningful message, if invalid. + */ + public void validateConfig(Config configToSet) { + validateMirrorReposHelmChartFolderSet(configToSet); + } + + private static void validateMirrorReposHelmChartFolderSet(Config configToSet) { + if (configToSet.getApplication().getMirrorRepos() && (configToSet.getApplication() + .getLocalHelmChartFolder() == null || configToSet.getApplication() + .getLocalHelmChartFolder() + .isEmpty())) { + // This should only happen when run outside the image, i.e. during development + throw new IllegalArgumentException("Missing config for localHelmChartFolder.\n" + "Either run inside the official container image or setting env var " + "LOCAL_HELM_CHART_FOLDER='charts' after running 'scripts/downloadHelmCharts.sh' from the repo"); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/ConfigLifecycleHook.java b/src/main/java/com/cloudogu/gitops/tools/common/ConfigLifecycleHook.java new file mode 100644 index 000000000..4f827ee0a --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/ConfigLifecycleHook.java @@ -0,0 +1,18 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.config.Config; + +/** + * Optional participation in the global configuration initialization lifecycle. + * + *

This lifecycle is separate from the tool deployment lifecycle. Implement this interface only + * when a component needs access to the global {@link Config} before or after initialization. + */ +public interface ConfigLifecycleHook { + + default void preConfigInit(Config configToSet) { + } + + default void postConfigInit(Config configToSet) { + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/HelmChartConfig.java b/src/main/java/com/cloudogu/gitops/tools/common/HelmChartConfig.java new file mode 100644 index 000000000..d86f6476b --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/HelmChartConfig.java @@ -0,0 +1,19 @@ +package com.cloudogu.gitops.tools.common; + +import lombok.Builder; + +import java.util.Map; + +@Builder +public record HelmChartConfig( + String repoURL, + String chart, + String version, + Map values, + String localHelmChartFolder +) { + + public HelmChartConfig { + values = ImmutableConfigData.copyMap(values); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/ImagePullSecretConfig.java b/src/main/java/com/cloudogu/gitops/tools/common/ImagePullSecretConfig.java new file mode 100644 index 000000000..e6cd63904 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/ImagePullSecretConfig.java @@ -0,0 +1,21 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import lombok.Builder; + +@Builder +public record ImagePullSecretConfig( + boolean create, + String proxyUrl, + String url, + String proxyUsername, + String readOnlyUsername, + String username, + String proxyPassword, + String readOnlyPassword, + String password, + CredentialsReference proxyCredentials, + CredentialsReference readOnlyCredentials, + CredentialsReference credentials +) { +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/ImagePullSecretCreator.java b/src/main/java/com/cloudogu/gitops/tools/common/ImagePullSecretCreator.java new file mode 100644 index 000000000..8e8418297 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/ImagePullSecretCreator.java @@ -0,0 +1,91 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +/** + * Creates the registry image pull secret for tools that deploy workloads into Kubernetes. + * + *

The creator is intentionally not part of the AbstractTool base class. Tools call it explicitly + * in their setup flow when an image pull secret is relevant for their namespace. + */ +@Singleton +@RequiredArgsConstructor +@Slf4j +public class ImagePullSecretCreator { + + private static final String IMAGE_PULL_SECRET_NAME = "proxy-registry"; + + private final K8sClient k8sClient; + private final CredentialsResolver credentialsResolver; + + public void createIfRequired(ImagePullSecretConfig config, String namespace) { + if (!config.create()) { + return; + } + + if (namespace == null || namespace.isEmpty()) { + throw new IllegalArgumentException("Namespace must be set before creating an image pull secret."); + } + + log.trace("Creating image pull secret '{}' in namespace {}", IMAGE_PULL_SECRET_NAME, namespace); + + String url = firstNonBlank(config.proxyUrl(), config.url()); + ResolvedCredentials credentials = resolveCredentials(config); + + k8sClient.createNamespace(namespace); + k8sClient.createImagePullSecret( + IMAGE_PULL_SECRET_NAME, + namespace, + url, + credentials.username(), + credentials.password() + ); + } + + private ResolvedCredentials resolveCredentials(ImagePullSecretConfig config) { + if (hasConfiguredReference(config.proxyCredentials()) + || hasCompletePlainCredentials(config.proxyUsername(), config.proxyPassword())) { + return credentialsResolver.resolveReference( + config.proxyCredentials(), config.proxyUsername(), config.proxyPassword() + ); + } + if (hasConfiguredReference(config.readOnlyCredentials()) + || hasCompletePlainCredentials(config.readOnlyUsername(), config.readOnlyPassword())) { + return credentialsResolver.resolveReference( + config.readOnlyCredentials(), config.readOnlyUsername(), config.readOnlyPassword() + ); + } + if (hasConfiguredReference(config.credentials()) + || hasCompletePlainCredentials(config.username(), config.password())) { + return credentialsResolver.resolveReference(config.credentials(), config.username(), config.password()); + } + + return new ResolvedCredentials( + firstNonBlank(config.proxyUsername(), firstNonBlank(config.readOnlyUsername(), config.username())), + firstNonBlank(config.proxyPassword(), firstNonBlank(config.readOnlyPassword(), config.password())) + ); + } + + private static boolean hasCompletePlainCredentials(String username, String password) { + return hasText(username) && hasText(password); + } + + private static boolean hasConfiguredReference(CredentialsReference reference) { + return reference != null + && (hasText(reference.secretName()) || hasText(reference.secretNamespace())); + } + + private static boolean hasText(String value) { + return value != null && !value.isEmpty(); + } + + private static String firstNonBlank(String preferred, String fallback) { + return hasText(preferred) ? preferred : fallback; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/ImmutableConfigData.java b/src/main/java/com/cloudogu/gitops/tools/common/ImmutableConfigData.java new file mode 100644 index 000000000..149ae118f --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/ImmutableConfigData.java @@ -0,0 +1,73 @@ +package com.cloudogu.gitops.tools.common; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Creates defensive, immutable copies of configuration data while retaining insertion order and + * allowing {@code null} values. + * + *

{@link Map#copyOf(Map)} and {@link List#copyOf(Collection)} are intentionally not used here: + * freely configurable Helm values may contain {@code null} values, which both factory methods + * reject. + */ +public final class ImmutableConfigData { + + private ImmutableConfigData() { + } + + public static Map copyMap(Map source) { + if (source == null || source.isEmpty()) { + return Collections.emptyMap(); + } + + Map copy = new LinkedHashMap<>(); + for (Map.Entry entry : source.entrySet()) { + copy.put(entry.getKey(), copyValue(entry.getValue())); + } + return Collections.unmodifiableMap(copy); + } + + public static List copyList(Collection source) { + if (source == null || source.isEmpty()) { + return Collections.emptyList(); + } + + List copy = new ArrayList<>(source.size()); + for (T value : source) { + copy.add(copyValue(value)); + } + return Collections.unmodifiableList(copy); + } + + @SuppressWarnings("unchecked") + private static T copyValue(T value) { + if (value instanceof Map map) { + return (T) copyMap(map); + } + if (value instanceof List list) { + return (T) copyList(list); + } + if (value instanceof Set set) { + Set copy = new LinkedHashSet<>(); + for (Object element : set) { + copy.add(copyValue(element)); + } + return (T) Collections.unmodifiableSet(copy); + } + if (value instanceof Collection collection) { + List copy = new ArrayList<>(collection.size()); + for (Object element : collection) { + copy.add(copyValue(element)); + } + return (T) Collections.unmodifiableCollection(copy); + } + return value; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/TemplateConfig.java b/src/main/java/com/cloudogu/gitops/tools/common/TemplateConfig.java new file mode 100644 index 000000000..31a49e905 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/TemplateConfig.java @@ -0,0 +1,28 @@ +package com.cloudogu.gitops.tools.common; + +import java.util.HashMap; +import java.util.Map; + +/** + * Builds a focused template configuration so FreeMarker receives only + * the values required by a template instead of the complete application Config. + */ +public final class TemplateConfig { + + private final Map values = new HashMap<>(); + + public TemplateConfig put(String path, Object value) { + String[] segments = path.split("\\."); + Map current = values; + for (int index = 0; index < segments.length - 1; index++) { + Object nested = current.computeIfAbsent(segments[index], ignored -> new HashMap()); + current = (Map) nested; + } + current.put(segments[segments.length - 1], value); + return this; + } + + public Map values() { + return ImmutableConfigData.copyMap(values); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/ToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/common/ToolConfigMapper.java new file mode 100644 index 000000000..3090e0106 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/ToolConfigMapper.java @@ -0,0 +1,9 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.context.DeploymentContext; + +@FunctionalInterface +public interface ToolConfigMapper { + + T map(DeploymentContext context); +} diff --git a/src/main/java/com/cloudogu/gitops/tools/common/ToolConfigMapperSupport.java b/src/main/java/com/cloudogu/gitops/tools/common/ToolConfigMapperSupport.java new file mode 100644 index 000000000..c9d8b22cb --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/common/ToolConfigMapperSupport.java @@ -0,0 +1,83 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Map; + +public final class ToolConfigMapperSupport { + + private ToolConfigMapperSupport() { + } + + public static HelmChartConfig helmChart( + Config.HelmConfigWithValues helmConfig, + String localHelmChartFolder) { + return HelmChartConfig.builder() + .repoURL(helmConfig.getRepoURL()) + .chart(helmConfig.getChart()) + .version(helmConfig.getVersion()) + .values(helmConfig.getValues()) + .localHelmChartFolder(localHelmChartFolder) + .build(); + } + + public static ImagePullSecretConfig imagePullSecret(Config.RegistrySchema registry) { + return ImagePullSecretConfig.builder() + .create(registry.getCreateImagePullSecrets()) + .proxyUrl(registry.getProxyUrl()) + .url(registry.getUrl()) + .proxyUsername(registry.getProxyUsername()) + .readOnlyUsername(registry.getReadOnlyUsername()) + .username(registry.getUsername()) + .proxyPassword(registry.getProxyPassword()) + .readOnlyPassword(registry.getReadOnlyPassword()) + .password(registry.getPassword()) + .proxyCredentials(CredentialsReference.from(registry.getProxyCredentials())) + .readOnlyCredentials(CredentialsReference.from(registry.getReadOnlyCredentials())) + .credentials(CredentialsReference.from(registry.getCredentials())) + .build(); + } + + /** + * Projects the central OIDC schema into plain template data. This prevents tool DTOs from + * retaining central Config schema objects through their template view. + */ + public static Map oidc(Config.OidcSchema oidc) { + if (oidc == null) { + return Map.of(); + } + + return new TemplateConfig() + .put("providerName", oidc.getProviderName()) + .put("issuerUrl", oidc.getIssuerUrl()) + .put("clientId", oidc.getClientId()) + .put("clientSecret", oidc.getClientSecret()) + .put("scopes", oidc.getScopes()) + .put("adminGroupName", oidc.getAdminGroupName()) + .put("enabled", oidc.isEnabled()) + .values(); + } + + /** + * Projects only the Helm repository URL needed by ArgoCD templates. This keeps the tool view + * focused and prevents central Config schema objects from crossing the DTO boundary. + */ + public static List> helmReleaseRepositories( + Collection helmReleases) { + if (helmReleases == null || helmReleases.isEmpty()) { + return List.of(); + } + + List> result = new ArrayList<>(); + for (Config.ContentSchema.HelmReleaseSchema release : helmReleases) { + if (release != null) { + result.add(new TemplateConfig().put("repoURL", release.getRepoURL()).values()); + } + } + return ImmutableConfigData.copyList(result); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/Jenkins.java b/src/main/java/com/cloudogu/gitops/tools/core/Jenkins.java new file mode 100644 index 000000000..6db5e21a5 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/Jenkins.java @@ -0,0 +1,518 @@ +package com.cloudogu.gitops.tools.core; + +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.jenkins.GlobalPropertyManager; +import com.cloudogu.gitops.infrastructure.jenkins.JenkinsApiClient; +import com.cloudogu.gitops.infrastructure.jenkins.JobManager; +import com.cloudogu.gitops.infrastructure.jenkins.PrometheusConfigurator; +import com.cloudogu.gitops.infrastructure.jenkins.UserManager; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.CommandExecutor; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.NetworkingUtils; +import com.cloudogu.gitops.utils.Tuple; +import io.micronaut.core.annotation.Order; +import io.micronaut.core.util.StringUtils; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Random; + +@Singleton +@Order(200) +@Slf4j +public class Jenkins extends AbstractMappedTool { + + public static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/jenkins/templates/values.ftl.yaml"; + + private static final List OIDC_BOOT_PLUGIN_NAMES = Arrays.asList( + "oic-auth", + "json-path-api", + "matrix-auth" + ); + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TOOL_NAME = "jenkins"; + private static final String ETC_GROUP_PATH = "/etc/group"; + private static final String JENKINS_APP_PATH = "apps/jenkins"; + private static final int PLUGIN_NAME_SPLIT_LIMIT = 2; + private static final int GID_GREPPER_POD_SUFFIX_BOUND = 10_000; + private static final int ETC_GROUP_MIN_FIELDS = 3; + private static final int ETC_GROUP_GID_FIELD_INDEX = 2; + // Not security-sensitive: only used to make a temporary pod name unique. + private static final Random RANDOM = new Random(); + + @Getter + @Setter + private String namespace; + private final CommandExecutor commandExecutor; + private final GlobalPropertyManager globalPropertyManager; + private final JobManager jobManager; + private final UserManager userManager; + private final PrometheusConfigurator prometheusConfigurator; + + private final ImagePullSecretCreator imagePullSecretCreator; + private final K8sClient k8sClient; + private final NetworkingUtils networkingUtils; + private final JenkinsConfigUpdater configUpdater; + private final CredentialsResolver credentialsResolver; + private final JenkinsApiClient jenkinsApiClient; + private String runtimeUrl; + private ResolvedCredentials runtimeCredentials; + private ResolvedCredentials runtimeMetricsCredentials; + private ResolvedCredentials runtimeRegistryCredentials; + private ResolvedCredentials runtimeProxyRegistryCredentials; + + public Jenkins( + CommandExecutor commandExecutor, + FileSystemUtils fileSystemUtils, + GlobalPropertyManager globalPropertyManager, + JobManager jobManager, + UserManager userManager, + PrometheusConfigurator prometheusConfigurator, + Deployer deployer, + K8sClient k8sClient, + NetworkingUtils networkingUtils, + AirGappedUtils airGappedUtils, + GitHandler gitHandler, + ImagePullSecretCreator imagePullSecretCreator, + JenkinsToolConfigMapper configMapper, + JenkinsConfigUpdater configUpdater, + CredentialsResolver credentialsResolver, + JenkinsApiClient jenkinsApiClient) { + super(configMapper); + this.commandExecutor = commandExecutor; + this.fileSystemUtils = fileSystemUtils; + this.globalPropertyManager = globalPropertyManager; + this.jobManager = jobManager; + this.userManager = userManager; + this.prometheusConfigurator = prometheusConfigurator; + this.deployer = deployer; + this.k8sClient = k8sClient; + this.networkingUtils = networkingUtils; + this.airGappedUtils = airGappedUtils; + this.gitHandler = gitHandler; + this.imagePullSecretCreator = imagePullSecretCreator; + this.configUpdater = configUpdater; + this.credentialsResolver = credentialsResolver; + this.jenkinsApiClient = jenkinsApiClient; + } + + @Override + protected boolean isEnabled(JenkinsToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + resolveRuntimeCredentials(); + this.runtimeUrl = toolConfig().server().url(); + if (!isInternalJenkins()) { + return; + } + + this.namespace = activeNamespace(toolConfig()); + + createImagePullSecret(); + createJenkinsNamespace(); + labelJenkinsNode(); + createJenkinsCredentialsSecret(); + prepareJenkinsHelmValues(); + prepareJenkinsApp(repositoryWorkspace.getClusterResourcesRepository()); + } + + @Override + protected void deploy() { + if (!isInternalJenkins()) { + return; + } + + deployInternalJenkins(); + } + + @Override + protected void postDeploy() { + if (isInternalJenkins()) { + updateJenkinsUrl(); + } + + runSetupScript(); + } + + @Override + protected void publishChanges() { + if (!isInternalJenkins()) { + return; + } + + publishClusterResourcesChanges(TOOL_NAME); + } + + private void resolveRuntimeCredentials() { + runtimeRegistryCredentials = null; + runtimeProxyRegistryCredentials = null; + runtimeCredentials = credentialsResolver.resolveReference( + toolConfig().server().credentials(), + toolConfig().server().username(), + toolConfig().server().password() + ); + runtimeMetricsCredentials = credentialsResolver.resolveReference( + toolConfig().server().metricsCredentials(), + toolConfig().server().metricsUsername(), + toolConfig().server().metricsPassword() + ); + jenkinsApiClient.setRuntimeCredentials(runtimeCredentials); + } + + private void createImagePullSecret() { + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + } + + private void createJenkinsNamespace() { + k8sClient.createNamespace(namespace); + } + + private void labelJenkinsNode() { + // Mark the first node for Jenkins and agents. See jenkins/values.ftl.yaml "agent.workingDir" + // for details. + // Remove first in case new nodes were added. + k8sClient.labelRemove("node", "--all", "", "node"); + + String nodeName = k8sClient.waitForNode().replace("node/", ""); + k8sClient.label("node", nodeName, new Tuple<>("node", TOOL_NAME)); + } + + private void createJenkinsCredentialsSecret() { + k8sClient.createSecret( + "generic", "jenkins-credentials", namespace, new Tuple<>( + "jenkins-admin-user", runtimeCredentials.username() + ), new Tuple<>( + "jenkins-admin-password", runtimeCredentials.password() + ) + ); + } + + private void prepareJenkinsHelmValues() { + addHelmValuesData("dockerGid", findDockerGid()); + addHelmValuesData( + "jenkinsBootPlugins", + jenkinsOidcConfigured() ? getJenkinsOidcBootPlugins() : Collections.emptyList() + ); + } + + @Override + protected String activeNamespace(JenkinsToolConfig config) { + return config.namespace(); + } + + private boolean isInternalJenkins() { + return toolConfig().internal(); + } + + private void deployInternalJenkins() { + addHelmValuesData("config", toolConfig().templateConfig()); + deployHelmChart(TOOL_NAME, TOOL_NAME, namespace, toolConfig().helm(), HELM_VALUES_PATH, context, true); + } + + private void updateJenkinsUrl() { + // Defined here: + // https://github.com/jenkinsci/helm-charts/blob/jenkins-5.8.1/charts/jenkins/templates/_helpers.tpl#L46-L57 + String serviceName = TOOL_NAME; + + // Update jenkins.url after it is deployed and ports are known. + if (toolConfig().application().runningInsideK8s()) { + log.debug("Setting jenkins url to k8s service, since installation is running inside k8s"); + runtimeUrl = networkingUtils.createUrl(serviceName + "." + namespace + ".svc.cluster.local", "80"); + configUpdater.updateUrl(runtimeUrl); + } else { + log.debug( + "Setting jenkins configs for local single node cluster with internal jenkins. Waiting for NodePort..."); + String port = k8sClient.waitForNodePort(serviceName, namespace); + String clusterBindAddress = networkingUtils.findClusterBindAddress(); + runtimeUrl = networkingUtils.createUrl(clusterBindAddress, port); + configUpdater.updateUrl(runtimeUrl); + } + } + + private void prepareJenkinsApp(GitRepo clusterResourcesRepo) { + log.debug("Preparing Jenkins repository content in {}", clusterResourcesRepo.getRepoTarget()); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, JENKINS_APP_PATH) + ); + } + + private void runSetupScript() { + Map scriptParams = new HashMap<>(); + scriptParams.put("TRACE", toolConfig().application().trace()); + scriptParams.put("INTERNAL_JENKINS", toolConfig().internal()); + scriptParams.put("JENKINS_HELM_CHART_VERSION", toolConfig().helm().version()); + scriptParams.put("JENKINS_URL", runtimeUrl); + scriptParams.put("JENKINS_USERNAME", runtimeCredentials.username()); + scriptParams.put("JENKINS_PASSWORD", runtimeCredentials.password()); + scriptParams.put("SCM_URL", this.gitHandler.getTenant().getUrl()); + scriptParams.put("PREFIXED_SCM_URL", this.gitHandler.getTenant().repoPrefix()); + scriptParams.put("SCM_PASSWORD", this.gitHandler.getTenant().getCredentials().getPassword()); + scriptParams.put("SCM_PROVIDER", toolConfig().scm().providerType()); + scriptParams.put("INSTALL_ARGOCD", toolConfig().argocdActive()); + scriptParams.put("NAME_PREFIX", toolConfig().application().namePrefix()); + scriptParams.put("INSECURE", toolConfig().application().insecure()); + scriptParams.put("SKIP_RESTART", toolConfig().server().skipRestart()); + scriptParams.put("SKIP_PLUGINS", toolConfig().server().skipPlugins()); + + commandExecutor.execute(fileSystemUtils.getRootDir() + "/scripts/jenkins/init-jenkins.sh", scriptParams); + + configureGlobalProperties(); + configureMetricsUser(); + } + + private void configureGlobalProperties() { + setPrefixedGlobalProperty("SCM_URL", this.gitHandler.getTenant().getUrl()); + setPrefixedGlobalProperty("PREFIXED_SCM_URL", this.gitHandler.getTenant().repoPrefix()); + + if (!toolConfig().server().additionalEnvironments().isEmpty()) { + for (Map.Entry entry : toolConfig().server().additionalEnvironments().entrySet()) { + globalPropertyManager.setGlobalProperty(entry.getKey(), entry.getValue()); + } + } + + setPrefixedGlobalPropertyIfNotEmpty("REGISTRY_URL", toolConfig().registry().url()); + setPrefixedGlobalPropertyIfNotEmpty("REGISTRY_PATH", toolConfig().registry().path()); + + if (toolConfig().registry().twoRegistries()) { + setPrefixedGlobalProperty("REGISTRY_PROXY_URL", toolConfig().registry().proxyUrl()); + setPrefixedGlobalProperty("REGISTRY_PROXY_PATH", toolConfig().registry().proxyPath()); + } + + setPrefixedGlobalPropertyIfNotEmpty("MAVEN_CENTRAL_MIRROR", toolConfig().server().mavenCentralMirror()); + + setPrefixedGlobalProperty("K8S_VERSION", toolConfig().kubernetesVersion()); + } + + private void configureMetricsUser() { + if (userManager.isUsingSecurityRealmWithoutLocalUserCreation()) { + log.trace("Using a security realm without local user creation. Must not create user."); + } else { + userManager.createUser( + runtimeMetricsCredentials.username(), runtimeMetricsCredentials.password() + ); + } + + userManager.grantPermission( + runtimeMetricsCredentials.username(), UserManager.Permissions.METRICS_VIEW + ); + + if (toolConfig().monitoringActive() && toolConfig().internal()) { + // An external Jenkins can likely not be monitored + prometheusConfigurator.enableAuthentication(); + } + } + + private void setPrefixedGlobalProperty(String name, String value) { + globalPropertyManager.setGlobalProperty(toolConfig().application().environmentPrefix() + name, value); + } + + private void setPrefixedGlobalPropertyIfNotEmpty(String name, String value) { + if (StringUtils.isNotEmpty(value)) { + setPrefixedGlobalProperty(name, value); + } + } + + public void createJenkinsjob(String namespace, String repoName) { + String credentialId = "scm-user"; + String prefixedNamespace = toolConfig().application().namePrefix() + namespace; + String jobName = toolConfig().application().namePrefix() + repoName; + + jobManager.createJob(jobName, this.gitHandler.getTenant().getUrl(), prefixedNamespace, credentialId); + + var scmCredentials = gitHandler.getTenant().getCredentials(); + + if (toolConfig().scm().providerType() == ScmProviderType.SCM_MANAGER) { + jobManager.createCredential( + jobName, + credentialId, + toolConfig().application().namePrefix() + "gitops", + scmCredentials.getPassword(), + "credentials for accessing scm-manager" + ); + } + + if (toolConfig().scm().providerType() == ScmProviderType.GITLAB) { + jobManager.createCredential( + jobName, + credentialId, + scmCredentials.getUsername(), + scmCredentials.getPassword(), + "credentials for accessing gitlab" + ); + } + ResolvedCredentials registryCredentials = registryCredentials(); + jobManager.createCredential( + jobName, + "registry-user", + registryCredentials.username(), + registryCredentials.password(), + "credentials for accessing the docker-registry for writing images built on jenkins" + ); + + if (toolConfig().registry().twoRegistries()) { + ResolvedCredentials proxyRegistryCredentials = proxyRegistryCredentials(); + jobManager.createCredential( + jobName, + "registry-proxy-user", + proxyRegistryCredentials.username(), + proxyRegistryCredentials.password(), + "credentials for accessing the docker-registry that contains 3rd party or base images" + ); + } + + jobManager.startJob(jobName); + } + + private ResolvedCredentials registryCredentials() { + if (runtimeRegistryCredentials == null) { + runtimeRegistryCredentials = credentialsResolver.resolveReference( + toolConfig().registry().credentials(), + toolConfig().registry().username(), + toolConfig().registry().password() + ); + } + return runtimeRegistryCredentials; + } + + private ResolvedCredentials proxyRegistryCredentials() { + if (runtimeProxyRegistryCredentials == null) { + runtimeProxyRegistryCredentials = credentialsResolver.resolveReference( + toolConfig().registry().proxyCredentials(), + toolConfig().registry().proxyUsername(), + toolConfig().registry().proxyPassword() + ); + } + return runtimeProxyRegistryCredentials; + } + + private boolean jenkinsOidcConfigured() { + return toolConfig().server().oidcConfigured(); + } + + private List getJenkinsOidcBootPlugins() { + File pluginsFile = new File(fileSystemUtils.getRootDir() + "/scripts/jenkins/plugins/plugins.txt"); + Map pinnedPlugins = new HashMap<>(); + + try { + List lines = Files.readAllLines(pluginsFile.toPath()); + for (String line : lines) { + String pluginDefinition = line.trim(); + if (pluginDefinition.isEmpty() || pluginDefinition.startsWith("#")) { + continue; + } + String pluginName = pluginDefinition.split(":", PLUGIN_NAME_SPLIT_LIMIT)[0]; + if (OIDC_BOOT_PLUGIN_NAMES.contains(pluginName)) { + pinnedPlugins.put(pluginName, pluginDefinition); + } + } + } catch (IOException e) { + throw new UncheckedIOException("Failed to read plugins file: " + pluginsFile, e); + } + + List missingPlugins = OIDC_BOOT_PLUGIN_NAMES.stream() + .filter(name -> !pinnedPlugins.containsKey(name)) + .toList(); + + if (!missingPlugins.isEmpty()) { + throw new IllegalStateException("Required Jenkins OIDC boot plugins missing from " + pluginsFile + ": " + String.join( + ", ", + missingPlugins + )); + } + + List result = new ArrayList<>(); + for (String name : OIDC_BOOT_PLUGIN_NAMES) { + result.add(pinnedPlugins.get(name)); + } + return result; + } + + protected String findDockerGid() { + String gid = ""; + String etcGroup = k8sClient.run( + "tmp-docker-gid-grepper-" + RANDOM.nextInt(GID_GREPPER_POD_SUFFIX_BOUND), + "irrelevant" /* Redundant, but mandatory param */, + namespace, + createGidGrepperOverrides(), + "--restart=Never", + "-ti", + "--rm", + "--quiet" + ); + + if (etcGroup != null) { + String[] lines = etcGroup.split("\n"); + for (String line : lines) { + String[] parts = line.split(":"); + if (parts.length >= ETC_GROUP_MIN_FIELDS && "docker".equals(parts[0])) { + gid = parts[ETC_GROUP_GID_FIELD_INDEX]; + break; + } + } + } + + if (gid.isEmpty()) { + log.warn( + """ + Unable to determine Docker Group ID (GID). Jenkins Agent pods will run as root user (UID 0)! + Group docker not found in /etc/group: + {}""", etcGroup + ); + return ""; + } else { + log.debug("Using Docker Group ID (GID) {} for Jenkins Agent pods", gid); + return gid; + } + } + + Map createGidGrepperOverrides() { + return Map.of( + "spec", Map.of( + "containers", + List.of(Map.of( + "name", + "tmp-docker-gid-grepper", + "image", + toolConfig().server().internalBashImage(), + "args", + List.of("cat", ETC_GROUP_PATH), + "volumeMounts", + List.of(Map.of("name", "group", "mountPath", ETC_GROUP_PATH, "readOnly", true)) + )), + "nodeSelector", + Map.of("node", TOOL_NAME), + "volumes", + List.of(Map.of("name", "group", "hostPath", Map.of("path", ETC_GROUP_PATH))) + ) + ); + } + +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/JenkinsConfigUpdater.java b/src/main/java/com/cloudogu/gitops/tools/core/JenkinsConfigUpdater.java new file mode 100644 index 000000000..34ef807eb --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/JenkinsConfigUpdater.java @@ -0,0 +1,16 @@ +package com.cloudogu.gitops.tools.core; + +import com.cloudogu.gitops.config.Config; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class JenkinsConfigUpdater { + + private final Config config; + + public void updateUrl(String url) { + config.getJenkins().setUrl(url); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/JenkinsToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/core/JenkinsToolConfig.java new file mode 100644 index 000000000..71f56e0b0 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/JenkinsToolConfig.java @@ -0,0 +1,86 @@ +package com.cloudogu.gitops.tools.core; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Map; + +@Builder +public record JenkinsToolConfig( + boolean active, + boolean internal, + String namespace, + Application application, + Server server, + Scm scm, + Registry registry, + boolean argocdActive, + boolean monitoringActive, + String kubernetesVersion, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public JenkinsToolConfig { + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } + + @Builder + public record Application( + String namePrefix, + String environmentPrefix, + boolean runningInsideK8s, + boolean trace, + boolean insecure + ) { + } + + @Builder + public record Server( + String url, + String username, + String password, + CredentialsReference credentials, + String metricsUsername, + String metricsPassword, + CredentialsReference metricsCredentials, + boolean skipRestart, + boolean skipPlugins, + String mavenCentralMirror, + String internalBashImage, + boolean oidcConfigured, + Map additionalEnvironments + ) { + + public Server { + additionalEnvironments = ImmutableConfigData.copyMap(additionalEnvironments); + } + } + + @Builder + public record Scm( + ScmProviderType providerType + ) { + } + + @Builder + public record Registry( + String url, + String path, + String username, + String password, + CredentialsReference credentials, + boolean twoRegistries, + String proxyUrl, + String proxyPath, + String proxyUsername, + String proxyPassword, + CredentialsReference proxyCredentials + ) { + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/JenkinsToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/core/JenkinsToolConfigMapper.java new file mode 100644 index 000000000..5af845e28 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/JenkinsToolConfigMapper.java @@ -0,0 +1,100 @@ +package com.cloudogu.gitops.tools.core; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class JenkinsToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public JenkinsToolConfig map(DeploymentContext context) { + Config.JenkinsSchema jenkins = config.getJenkins(); + ScmProviderType scmProviderType = config.getScm() == null ? null : config.getScm().getScmProviderType(); + + JenkinsToolConfig.Application applicationConfig = JenkinsToolConfig.Application.builder() + .namePrefix(config.getApplication().getNamePrefix()) + .environmentPrefix(config.getApplication().getNamePrefixForEnvVars()) + .runningInsideK8s(config.getApplication().getRunningInsideK8s()) + .trace(config.getApplication().getTrace()) + .insecure(config.getApplication().getInsecure()) + .build(); + JenkinsToolConfig.Server serverConfig = JenkinsToolConfig.Server.builder() + .url(jenkins.getUrl()) + .username(jenkins.getUsername()) + .password(jenkins.getPassword()) + .credentials(CredentialsReference.from(jenkins.getCredentials())) + .metricsUsername(jenkins.getMetricsUsername()) + .metricsPassword(jenkins.getMetricsPassword()) + .metricsCredentials(CredentialsReference.from(jenkins.getMetricsCredentials())) + .skipRestart(jenkins.getSkipRestart()) + .skipPlugins(jenkins.getSkipPlugins()) + .mavenCentralMirror(jenkins.getMavenCentralMirror()) + .internalBashImage(jenkins.getInternalBashImage()) + .oidcConfigured(jenkins.getOidc() != null && jenkins.getOidc().isEnabled()) + .additionalEnvironments(jenkins.getAdditionalEnvs()) + .build(); + JenkinsToolConfig.Scm scmConfig = JenkinsToolConfig.Scm.builder() + .providerType(scmProviderType) + .build(); + JenkinsToolConfig.Registry registryConfig = JenkinsToolConfig.Registry.builder() + .url(config.getRegistry().getUrl()) + .path(config.getRegistry().getPath()) + .username(config.getRegistry().getUsername()) + .password(config.getRegistry().getPassword()) + .credentials(CredentialsReference.from(config.getRegistry().getCredentials())) + .twoRegistries(config.getRegistry().getTwoRegistries()) + .proxyUrl(config.getRegistry().getProxyUrl()) + .proxyPath(config.getRegistry().getProxyPath()) + .proxyUsername(config.getRegistry().getProxyUsername()) + .proxyPassword(config.getRegistry().getProxyPassword()) + .proxyCredentials(CredentialsReference.from(config.getRegistry().getProxyCredentials())) + .build(); + + return JenkinsToolConfig.builder() + .active(jenkins.getActive()) + .internal(jenkins.getInternal()) + .namespace(jenkins.getInternal() ? config.getApplication().getNamePrefix() + jenkins.getNamespace() : null) + .application(applicationConfig) + .server(serverConfig) + .scm(scmConfig) + .registry(registryConfig) + .argocdActive(config.getFeatures().getArgocd().getActive()) + .monitoringActive(config.getFeatures().getMonitoring().getActive()) + .kubernetesVersion(Config.K8S_VERSION) + .helm(ToolConfigMapperSupport.helmChart( + jenkins.getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config)) + .build(); + } + + private static Map templateConfig(Config config) { + return new TemplateConfig() + .put("application.baseUrl", config.getApplication().getBaseUrl()) + .put("features.certManager.active", config.getFeatures().getCertManager().getActive()) + .put("features.certManager.issuer", config.getFeatures().getCertManager().getIssuer()) + .put("jenkins.helm.version", config.getJenkins().getHelm().getVersion()) + .put("jenkins.ingress", config.getJenkins().getIngress()) + .put("jenkins.internalBashImage", config.getJenkins().getInternalBashImage()) + .put("jenkins.internalDockerClientVersion", config.getJenkins().getInternalDockerClientVersion()) + .put("jenkins.jenkinsImage", config.getJenkins().getJenkinsImage()) + .put("jenkins.oidc", ToolConfigMapperSupport.oidc(config.getJenkins().getOidc())) + .put("jenkins.url", config.getJenkins().getUrl()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .values(); + } +} \ No newline at end of file diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCD.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCD.java new file mode 100644 index 000000000..c89befd22 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCD.java @@ -0,0 +1,305 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ConfigLifecycleHook; +import com.cloudogu.gitops.tools.core.argocd.mode.DeploymentMode; +import com.cloudogu.gitops.tools.core.argocd.mode.DeploymentModeFactory; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.MapUtils; +import com.cloudogu.gitops.utils.Tuple; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.extern.slf4j.Slf4j; +import org.springframework.security.crypto.bcrypt.BCrypt; + +import java.io.File; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +@Singleton +@Order(100) +@Slf4j +public class ArgoCD extends AbstractMappedTool implements ConfigLifecycleHook { + + private static final int BCRYPT_LOG_ROUNDS = 4; + private static final String TOOL_NAME = "argocd"; + private static final String SECRET_RESOURCE = "secret"; + + private final K8sClient k8sClient; + private final HelmClient helmClient; + private final DeploymentModeFactory deploymentModeFactory; + private final CredentialsResolver credentialsResolver; + + private String password; + private String namespace; + private ArgoCDRepoSetup repoSetup; + private ArgoCDRepoLayout clusterResourcesRepo; + private DeploymentMode deploymentMode; + + public ArgoCD( + K8sClient k8sClient, + HelmClient helmClient, + FileSystemUtils fileSystemUtils, + GitHandler gitHandler, + DeploymentModeFactory deploymentModeFactory, + ArgoCDToolConfigMapper configMapper, + CredentialsResolver credentialsResolver) { + super(configMapper); + this.k8sClient = k8sClient; + this.helmClient = helmClient; + this.fileSystemUtils = fileSystemUtils; + this.gitHandler = gitHandler; + this.deploymentModeFactory = deploymentModeFactory; + this.credentialsResolver = credentialsResolver; + } + + @Override + protected boolean isEnabled(ArgoCDToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + this.namespace = activeNamespace(toolConfig()); + ResolvedCredentials applicationCredentials = credentialsResolver.resolveReference( + toolConfig().credentials(), + toolConfig().username(), + toolConfig().password() + ); + this.password = applicationCredentials.password(); + + this.repoSetup = ArgoCDRepoSetup.create(fileSystemUtils, gitHandler, repositoryWorkspace, toolConfig()); + + this.clusterResourcesRepo = repoSetup.clusterRepoLayout(); + + this.deploymentMode = deploymentModeFactory.create( + toolConfig(), + k8sClient, + gitHandler, + repositoryWorkspace, + repoSetup, + clusterResourcesRepo, + namespace + ); + + log.debug("Preparing ArgoCD repository content"); + repoSetup.prepareRepositories(); + + log.debug("Creating namespaces"); + k8sClient.createNamespaces(new ArrayList<>(toolConfig().activeNamespaces())); + + deploymentMode.createSCMCredentialsSecret(); + createNotificationSecretIfRequired(); + + if (toolConfig().operator()) { + deploymentMode.generateRBAC(); + } else { + mergeHelmValuesIfConfigured(); + } + } + + @Override + protected void deploy() { + log.debug("Installing Argo CD"); + + if (toolConfig().operator()) { + deployWithOperator(); + } else { + deployWithHelm(); + } + } + + @Override + protected void postDeploy() { + deploymentMode.applyBootstrapResources(); + deleteHelmArgoSecrets(); + } + + @Override + protected void publishChanges() { + try { + repositoryWorkspace.commitAndPushClusterResourcesAndTenantBootstrapChanges( + "Update ArgoCD repository content"); + } catch (Exception e) { + throw new RuntimeException("Failed to publish ArgoCD changes", e); + } + } + + @Override + protected String activeNamespace(ArgoCDToolConfig config) { + return config.namespace(); + } + + @Override + public String getNamespace() { + return namespace; + } + + @Override + public void postConfigInit(Config configToSet) { + // Exit early if not in operator mode or if env list is empty + if (!configToSet.getFeatures().getArgocd().getOperator() || configToSet.getFeatures() + .getArgocd() + .getEnv() == null) { + log.debug("Skipping features.argocd.env validation: operator mode is disabled or env list is empty."); + return; + } + + List env = configToSet.getFeatures().getArgocd().getEnv(); + + log.info("Validating env list in features.argocd.env with {} entries.", env.size()); + + for (Object entry : env) { + if (entry instanceof Map map && map.get("name") instanceof String && map.get("value") instanceof String) { + continue; + } + + throw new IllegalArgumentException( + "Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: " + (entry instanceof Map map ? formatMap( + map) : entry)); + } + + log.info("Env list validation for features.argocd.env completed successfully."); + } + + private static String formatMap(Map map) { + if (map == null) { + return "null"; + } + return map.entrySet() + .stream() + .map(entry -> entry.getKey() + ":" + entry.getValue()) + .collect(Collectors.joining(", ", "[", "]")); + } + + private void createNotificationSecretIfRequired() { + ResolvedCredentials smtpCredentials = credentialsResolver.resolveReference( + toolConfig().smtpCredentials(), + toolConfig().smtpUser(), + toolConfig().smtpPassword() + ); + String smtpUser = smtpCredentials.username(); + String smtpPassword = smtpCredentials.password(); + if ((smtpUser != null && !smtpUser.isEmpty()) || (smtpPassword != null && !smtpPassword.isEmpty())) { + k8sClient.createSecret( + "generic", + "argocd-notifications-secret", + namespace, + new Tuple<>("email-username", smtpUser), + new Tuple<>("email-password", smtpPassword) + ); + } + } + + private void mergeHelmValuesIfConfigured() { + Map values = toolConfig().values(); + if (values == null || values.isEmpty()) { + return; + } + + mergeAndWriteYamlValues(clusterResourcesRepo.helmValuesFile(), values, "values.yaml"); + } + + private void mergeAndWriteYamlValues(String configPath, Map values, String logLabel) { + log.debug("extend Argocd {} with {}", logLabel, values); + + Map argocdYaml = fileSystemUtils.readYaml(Path.of(configPath)); + Map result = MapUtils.deepMerge(values, argocdYaml); + + fileSystemUtils.writeYaml(result, new File(configPath)); + log.debug("Argocd {} contains {}", logLabel, result); + } + + private void deleteHelmArgoSecrets() { + // Delete helm-argo secrets to decouple from helm. + // This does not delete Argo from the cluster, but you can no longer modify argo directly with + // helm. + // For development keeping it in helm makes it easier, e.g. for helm uninstall. + k8sClient.delete(SECRET_RESOURCE, namespace, new Tuple<>("owner", "helm"), new Tuple<>("name", TOOL_NAME)); + } + + private void deployWithOperator() { + String argocdConfigPath = clusterResourcesRepo.operatorConfigFile(); + Map values = toolConfig().values(); + + if (values != null && !values.isEmpty()) { + mergeAndWriteYamlValues(argocdConfigPath, values, "argocd.yaml for operator"); + } + + k8sClient.applyYaml(argocdConfigPath); + + // ArgoCD is not installed until the ArgoCD-Operator did his job. + // This can take some time, so we wait for the status of the custom resource to become + // "Available" + k8sClient.waitForResourcePhase(TOOL_NAME, TOOL_NAME, namespace, "Available"); + + updateAdminPasswordForOperator(); + + deploymentMode.updateManagedNamespaces(); + + log.debug("Apply RBAC permissions for ArgoCD in all managed namespaces imperatively"); + k8sClient.applyYaml(clusterResourcesRepo.operatorRbacDir()); + } + + private void updateAdminPasswordForOperator() { + log.debug("Setting new argocd admin password"); + + // Set admin password imperatively here instead of operator/argocd.yaml, because we don't want + // it to show in git repo. + // The Operator uses an extra secret to store the admin Password, which is not bcrypted. + k8sClient.patch( + SECRET_RESOURCE, + "argocd-cluster", + namespace, + Map.of("stringData", Map.of("admin.password", password)) + ); + + // In newer Versions ArgoCD Operator uses the password in argocd-cluster secret only as + // generated initial password, + // but we want to set our own admin password so we set the password in both Secrets for + // consistency. + updateBcryptAdminPassword(); + } + + private void deployWithHelm() { + String umbrellaChartPath = clusterResourcesRepo.helmDir(); + + // Even if the Chart.lock already contains the repo, we need to add it before resolving it. + // See https://github.com/helm/helm/issues/8036#issuecomment-872502901 + Map chartYaml = fileSystemUtils.readYaml(Path.of(clusterResourcesRepo.chartYaml())); + List> helmDependencies = (List>) chartYaml.get("dependencies"); + String repository = (String) helmDependencies.get(0).get("repository"); + + helmClient.addRepo("argo", repository); + helmClient.dependencyBuild(umbrellaChartPath); + helmClient.upgrade(TOOL_NAME, umbrellaChartPath, Map.of("namespace", namespace)); + + updateBcryptAdminPassword(); + } + + private void updateBcryptAdminPassword() { + log.debug("Setting new argocd admin password"); + + String bcryptArgoCDPassword = BCrypt.hashpw(password, BCrypt.gensalt(BCRYPT_LOG_ROUNDS)); + + k8sClient.patch( + SECRET_RESOURCE, + "argocd-secret", + namespace, + Map.of("stringData", Map.of("admin.password", bcryptArgoCDPassword)) + ); + } + + protected ArgoCDRepoSetup getRepoSetup() { + return this.repoSetup; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoLayout.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoLayout.java new file mode 100644 index 000000000..c4c72b389 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoLayout.java @@ -0,0 +1,76 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import java.nio.file.Path; + +public record ArgoCDRepoLayout(String repoRootDir) { + + private static final String APPS_ARGOCD_DIR = "apps/argocd"; + + private static final String APPLICATIONS_DIR = "applications"; + private static final String HELM_DIR = "argocd"; + private static final String MULTITENANT_DIR = "multiTenant"; + private static final String OPERATOR_DIR = "operator"; + private static final String PROJECTS_DIR = "projects"; + + private static final String NETPOL_YAML = "templates/allow-namespaces.yaml"; + + public String rootDir() { + return repoRootDir; + } + + public String argocdRoot() { + return Path.of(repoRootDir, APPS_ARGOCD_DIR).toString(); + } + + public String operatorDir() { + return Path.of(argocdRoot(), OPERATOR_DIR).toString(); + } + + public String operatorRbacDir() { + return Path.of(operatorDir(), "rbac").toString(); + } + + public String operatorConfigFile() { + return Path.of(operatorDir(), "argocd.yaml").toString(); + } + + public String multiTenantDir() { + return Path.of(argocdRoot(), MULTITENANT_DIR).toString(); + } + + public String applicationsDir() { + return Path.of(argocdRoot(), APPLICATIONS_DIR).toString(); + } + + public String projectsDir() { + return Path.of(argocdRoot(), PROJECTS_DIR).toString(); + } + + public String helmDir() { + return Path.of(argocdRoot(), HELM_DIR).toString(); + } + + public String helmValuesFile() { + return Path.of(helmDir(), "values.yaml").toString(); + } + + public String chartYaml() { + return Path.of(helmDir(), "Chart.yaml").toString(); + } + + public String netpolFile() { + return Path.of(helmDir(), NETPOL_YAML).toString(); + } + + public static String argocdSubdirRel() { + return APPS_ARGOCD_DIR; + } + + public static String operatorRbacSubfolder() { + return APPS_ARGOCD_DIR + "/" + OPERATOR_DIR + "/rbac"; + } + + public static String operatorRbacTenantSubfolder() { + return operatorRbacSubfolder() + "/tenant"; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetup.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetup.java new file mode 100644 index 000000000..3e943d674 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetup.java @@ -0,0 +1,194 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.utils.ClusterResourcesCopyFilter; +import com.cloudogu.gitops.utils.FileSystemUtils; +import freemarker.template.DefaultObjectWrapperBuilder; +import freemarker.template.TemplateModel; +import lombok.AccessLevel; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.MalformedURLException; +import java.net.URI; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.Map; + +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +@Slf4j +public class ArgoCDRepoSetup { + + private static final String CLUSTER_RESOURCES_SOURCE_DIR = "argocd/cluster-resources"; + private static final String TENANT_BOOTSTRAP_SOURCE_DIR = "argocd/cluster-resources/apps/argocd/multiTenant/tenant"; + private static final String ARGOCD_APP_PATH = ArgoCDRepoLayout.argocdSubdirRel(); + + private final FileSystemUtils fileSystemUtils; + private final GitHandler gitHandler; + private final RepositoryWorkspace repositoryWorkspace; + private final ArgoCDToolConfig config; + + public static ArgoCDRepoSetup create( + FileSystemUtils fileSystemUtils, + GitHandler gitHandler, + RepositoryWorkspace repositoryWorkspace, + ArgoCDToolConfig config) { + return new ArgoCDRepoSetup(fileSystemUtils, gitHandler, repositoryWorkspace, config); + } + + public ArgoCDRepoLayout clusterRepoLayout() { + return new ArgoCDRepoLayout(repositoryWorkspace.clusterResourcesRootDir()); + } + + public ArgoCDRepoLayout tenantRepoLayout() { + if (!repositoryWorkspace.hasTenantBootstrapRepository()) { + throw new IllegalStateException("tenantBootstrap repo is not initialized in single-instance mode."); + } + + return new ArgoCDRepoLayout(repositoryWorkspace.tenantBootstrapRootDir()); + } + + public void prepareRepositories() { + validateRepositoryWorkspace(); + + prepareClusterResourcesRepo(); + + if (config.multiTenant()) { + prepareTenantBootstrapRepo(); + } + } + + private void validateRepositoryWorkspace() { + if (!config.multiTenant()) { + return; + } + + if (!repositoryWorkspace.hasTenantBootstrapRepository()) { + throw new IllegalStateException("Dedicated Multi-Tenant mode requires a tenant bootstrap repository."); + } + + try { + String clusterRoot = new File(repositoryWorkspace.clusterResourcesRootDir()).getCanonicalPath(); + String tenantRoot = new File(repositoryWorkspace.tenantBootstrapRootDir()).getCanonicalPath(); + + if (clusterRoot.equals(tenantRoot)) { + throw new IllegalStateException("Dedicated Multi-Tenant mode requires separate local workspaces for " + "central cluster-resources and tenant bootstrap repositories. " + "Both resolved to: " + clusterRoot); + } + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + private void prepareClusterResourcesRepo() { + GitRepo clusterResourcesRepo = repositoryWorkspace.getClusterResourcesRepository(); + + log.debug( + "Preparing ArgoCD repository content in {} from {}/{}", + clusterResourcesRepo.getRepoTarget(), + CLUSTER_RESOURCES_SOURCE_DIR, + ARGOCD_APP_PATH + ); + + clusterResourcesRepo.copyDirectoryContents( + CLUSTER_RESOURCES_SOURCE_DIR, + ClusterResourcesCopyFilter.forSubDir(CLUSTER_RESOURCES_SOURCE_DIR, ARGOCD_APP_PATH) + ); + + clusterResourcesRepo.replaceTemplates(buildTemplateValues(clusterResourcesRepo)); + + prepareClusterResourcesLayout(); + } + + private void prepareTenantBootstrapRepo() { + GitRepo tenantBootstrapRepo = repositoryWorkspace.tenantBootstrapRepositoryOrFail(); + + log.debug( + "Preparing tenant bootstrap repo {} from {}", + tenantBootstrapRepo.getRepoTarget(), + TENANT_BOOTSTRAP_SOURCE_DIR + ); + + tenantBootstrapRepo.copyDirectoryContents(TENANT_BOOTSTRAP_SOURCE_DIR, allowAllFilter()); + + tenantBootstrapRepo.replaceTemplates(buildTemplateValues(tenantBootstrapRepo)); + } + + private void prepareClusterResourcesLayout() { + ArgoCDRepoLayout layout = clusterRepoLayout(); + + if (config.operator()) { + FileSystemUtils.deleteDir(layout.helmDir()); + } else { + FileSystemUtils.deleteDir(layout.operatorDir()); + } + + if (config.multiTenant()) { + log.debug( + "Deleting unnecessary non dedicated instances folders from argocd repo: " + "applications={}, projects={}, tenant={}/tenant", + layout.applicationsDir(), + layout.projectsDir(), + layout.multiTenantDir() + ); + + FileSystemUtils.deleteDir(layout.applicationsDir()); + FileSystemUtils.deleteDir(layout.projectsDir()); + + fileSystemUtils.moveDirectoryMergeOverwrite( + Path.of(layout.multiTenantDir(), "central"), + Path.of(layout.argocdRoot()) + ); + + FileSystemUtils.deleteDir(layout.multiTenantDir()); + } else { + FileSystemUtils.deleteDir(layout.multiTenantDir()); + } + + if (!config.netpols()) { + FileSystemUtils.deleteFile(layout.netpolFile()); + } + } + + private Map buildTemplateValues(GitRepo repo) { + Map values = new HashMap<>(); + values.put("tenantName", config.tenantName()); + + Map argocd = new HashMap<>(); + String url = config.url(); + + try { + String host = (url != null && !url.isEmpty()) ? URI.create(url).toURL().getHost() : ""; + argocd.put("host", host); + } catch (IllegalArgumentException | MalformedURLException e) { + throw new UncheckedIOException(new IOException("Malformed URL provided: " + url, e)); + } + values.put("argocd", argocd); + + Map scm = new HashMap<>(); + scm.put("baseUrl", repo.getGitProvider().getUrl()); + scm.put("host", repo.getGitProvider().getHost()); + scm.put("protocol", repo.getGitProvider().getProtocol()); + scm.put("repoUrl", repo.getGitProvider().repoPrefix()); + scm.put("centralScmUrl", gitHandler.getCentral() != null ? gitHandler.getCentral().repoPrefix() : ""); + values.put("scm", scm); + values.put("config", config.templateConfig()); + + try { + TemplateModel statics = new DefaultObjectWrapperBuilder(freemarker.template.Configuration.VERSION_2_3_32).build().getStaticModels(); + values.put("statics", statics); + } catch (Exception e) { + throw new RuntimeException("Failed to expose freemarker statics model", e); + } + + return values; + } + + private static FileFilter allowAllFilter() { + return file -> true; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfig.java new file mode 100644 index 000000000..3b5ef0804 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfig.java @@ -0,0 +1,43 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Collection; +import java.util.Map; + +@Builder +public record ArgoCDToolConfig( + boolean active, + String namespace, + String username, + String password, + CredentialsReference credentials, + boolean operator, + Collection activeNamespaces, + String smtpUser, + String smtpPassword, + CredentialsReference smtpCredentials, + Map values, + boolean multiTenant, + boolean netpols, + String tenantName, + String url, + Collection tenantNamespaces, + String centralNamespace, + boolean clusterAdmin, + ScmProviderType scmProviderType, + Map templateConfig, + Map rbacTemplateConfig +) { + + public ArgoCDToolConfig { + activeNamespaces = ImmutableConfigData.copyList(activeNamespaces); + tenantNamespaces = ImmutableConfigData.copyList(tenantNamespaces); + values = ImmutableConfigData.copyMap(values); + templateConfig = ImmutableConfigData.copyMap(templateConfig); + rbacTemplateConfig = ImmutableConfigData.copyMap(rbacTemplateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfigMapper.java new file mode 100644 index 000000000..f419ac453 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfigMapper.java @@ -0,0 +1,122 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Collection; +import java.util.List; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class ArgoCDToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public ArgoCDToolConfig map(DeploymentContext context) { + Config.ArgoCDSchema argocd = config.getFeatures().getArgocd(); + Collection activeNamespaces = config.getApplication().getNamespaces().getActiveNamespaces(); + Collection tenantNamespaces = config.getApplication().getNamespaces().getTenantNamespaces(); + return ArgoCDToolConfig.builder() + .active(argocd.getActive()) + .namespace(config.getApplication().getNamePrefix() + argocd.getNamespace()) + .username(config.getApplication().getUsername()) + .password(config.getApplication().getPassword()) + .credentials(CredentialsReference.from(config.getApplication().getCredentials())) + .operator(argocd.getOperator()) + .activeNamespaces(activeNamespaces) + .smtpUser(config.getFeatures().getMail().getSmtpUser()) + .smtpPassword(config.getFeatures().getMail().getSmtpPassword()) + .smtpCredentials(CredentialsReference.from(config.getFeatures().getMail().getCredentials())) + .values(argocd.getValues()) + .multiTenant(context.isMultiTenant()) + .netpols(config.getApplication().getNetpols()) + .tenantName(config.getApplication().getTenantName()) + .url(argocd.getUrl()) + .tenantNamespaces(tenantNamespaces) + .centralNamespace(config.getMultiTenant().getCentralArgocdNamespace()) + .clusterAdmin(config.getApplication().getClusterAdmin()) + .scmProviderType(config.getScm().getScmProviderType()) + .templateConfig(templateConfig(config, context)) + .rbacTemplateConfig(rbacTemplateConfig(config, context)) + .build(); + } + + private static Map rbacTemplateConfig(Config config, DeploymentContext context) { + return new TemplateConfig() + .put("application.openshift", context.isOpenshift()) + .put("features.monitoring.active", config.getFeatures().getMonitoring().getActive()) + .put("features.secrets.active", config.getFeatures().getSecrets().getActive()) + .values(); + } + + private static Map templateConfig(Config config, DeploymentContext context) { + String scmManagerNamespace = config.getScm() == null || config.getScm().getScmManager() == null + ? "scm-manager" + : config.getScm().getScmManager().getNamespace(); + return new TemplateConfig() + .put("application.clusterAdmin", config.getApplication().getClusterAdmin()) + .put("application.insecure", config.getApplication().getInsecure()) + .put("application.mirrorRepos", context.isAirgapped()) + .put("application.namePrefix", config.getApplication().getNamePrefix()) + .put("application.netpols", config.getApplication().getNetpols()) + .put("application.openshift", context.isOpenshift()) + .put("application.skipCrds", config.getApplication().getSkipCrds()) + .put( + "content.helmReleases", + config.getContent() == null + ? List.of() + : ToolConfigMapperSupport.helmReleaseRepositories(config.getContent().getHelmReleases()) + ) + .put("features.argocd.emailFrom", config.getFeatures().getArgocd().getEmailFrom()) + .put("features.argocd.emailToAdmin", config.getFeatures().getArgocd().getEmailToAdmin()) + .put("features.argocd.env", config.getFeatures().getArgocd().getEnv()) + .put("features.argocd.namespace", config.getFeatures().getArgocd().getNamespace()) + .put("features.argocd.oidc", ToolConfigMapperSupport.oidc(config.getFeatures().getArgocd().getOidc())) + .put("features.argocd.operator", config.getFeatures().getArgocd().getOperator()) + .put( + "features.argocd.resourceInclusionsCluster", + config.getFeatures().getArgocd().getResourceInclusionsCluster() + ) + .put("features.argocd.url", config.getFeatures().getArgocd().getUrl()) + .put("features.certManager.active", config.getFeatures().getCertManager().getActive()) + .put("features.certManager.issuer", config.getFeatures().getCertManager().getIssuer()) + .put("features.mail.active", config.getFeatures().getMail().getActive()) + .put("features.mail.smtpAddress", config.getFeatures().getMail().getSmtpAddress()) + .put("features.mail.smtpPasswordConfigured", smtpPasswordConfigured(config)) + .put("features.mail.smtpPort", config.getFeatures().getMail().getSmtpPort()) + .put("features.mail.smtpUserConfigured", smtpUserConfigured(config)) + .put("features.monitoring.active", config.getFeatures().getMonitoring().getActive()) + .put("features.monitoring.namespace", config.getFeatures().getMonitoring().getNamespace()) + .put("features.secrets.active", config.getFeatures().getSecrets().getActive()) + .put("multiTenant.centralArgocdNamespace", config.getMultiTenant().getCentralArgocdNamespace()) + .put("scm.scmManager.namespace", scmManagerNamespace) + .put("scm.scmProviderType", config.getScm().getScmProviderType()) + .values(); + } + + private static boolean smtpUserConfigured(Config config) { + return hasText(config.getFeatures().getMail().getSmtpUser()) || hasMailSecretReference(config); + } + + private static boolean smtpPasswordConfigured(Config config) { + return hasText(config.getFeatures().getMail().getSmtpPassword()) || hasMailSecretReference(config); + } + + private static boolean hasMailSecretReference(Config config) { + var credentials = config.getFeatures().getMail().getCredentials(); + return credentials != null + && (hasText(credentials.getSecretName()) || hasText(credentials.getSecretNamespace())); + } + + private static boolean hasText(String value) { + return value != null && !value.isEmpty(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DedicatedMultiTenantMode.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DedicatedMultiTenantMode.java new file mode 100644 index 000000000..ed7f5c5c4 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DedicatedMultiTenantMode.java @@ -0,0 +1,185 @@ +package com.cloudogu.gitops.tools.core.argocd.mode; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.infrastructure.kubernetes.rbac.RbacDefinition; +import com.cloudogu.gitops.infrastructure.kubernetes.rbac.Role; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDRepoLayout; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDRepoSetup; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDToolConfig; +import com.cloudogu.gitops.utils.Tuple; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +@RequiredArgsConstructor +@Slf4j +public class DedicatedMultiTenantMode implements DeploymentMode { + + private static final String SECRET_RESOURCE = "secret"; + private static final String ARGOCD_DEFAULT_CLUSTER_CONFIG = "argocd-default-cluster-config"; + + private final ArgoCDToolConfig config; + private final K8sClient k8sClient; + private final GitHandler gitHandler; + private final RepositoryWorkspace repositoryWorkspace; + private final ArgoCDRepoSetup repoSetup; + private final ArgoCDRepoLayout clusterResourcesRepo; + private final String namespace; + + @Override + public void createSCMCredentialsSecret() { + log.debug( + "Creating tenant repo credential secret that is used by tenant ArgoCD to access repos in {}", + config.scmProviderType() + ); + + createRepoCredentialsSecret( + "argocd-repo-creds-scm", namespace, gitHandler.getTenant() + .getUrl(), gitHandler.getTenant() + .getCredentials() + .getUsername(), gitHandler.getTenant() + .getCredentials() + .getPassword() + ); + + log.debug( + "Creating central repo credential secret that is used by central ArgoCD to access repos in {}", + config.scmProviderType() + ); + + createRepoCredentialsSecret( + "argocd-repo-creds-central-scm", + config.centralNamespace(), + gitHandler.getCentral() + .getUrl(), + gitHandler.getCentral() + .getCredentials() + .getUsername(), + gitHandler.getCentral() + .getCredentials() + .getPassword() + ); + } + + @Override + public void generateRBAC() { + log.debug("Generate RBAC permissions for tenant ArgoCD and central ArgoCD."); + + generateTenantArgoCDRBAC(); + generateCentralArgoCDRBAC(); + } + + @Override + public void updateManagedNamespaces() { + log.debug("Updating managed namespaces in tenant ArgoCD configuration secret."); + + k8sClient.patch( + SECRET_RESOURCE, ARGOCD_DEFAULT_CLUSTER_CONFIG, namespace, Map.of( + "stringData", Map.of( + "namespaces", String.join( + ",", config.tenantNamespaces() + ) + ) + ) + ); + + updateCentralManagedNamespaces(); + } + + @Override + public void applyBootstrapResources() { + k8sClient.applyYaml(Path.of(clusterResourcesRepo.projectsDir(), "tenant.yaml").toString()); + k8sClient.applyYaml(Path.of(clusterResourcesRepo.applicationsDir(), "bootstrap.yaml").toString()); + + ArgoCDRepoLayout tenantRepoLayout = repoSetup.tenantRepoLayout(); + k8sClient.applyYaml(Path.of(tenantRepoLayout.projectsDir(), "argocd.yaml").toString()); + k8sClient.applyYaml(Path.of(tenantRepoLayout.applicationsDir(), "bootstrap.yaml").toString()); + } + + private void generateTenantArgoCDRBAC() { + for (String ns : config.tenantNamespaces()) { + new RbacDefinition(Role.Variant.ARGOCD).withName("argocd") + .withNamespace(ns) + .withServiceAccountsFrom(namespace, ARGOCD_SERVICE_ACCOUNTS) + .withTemplateConfig(config.rbacTemplateConfig()) + .withRepo(repositoryWorkspace.getClusterResourcesRepository()) + .withSubfolder(ArgoCDRepoLayout.operatorRbacTenantSubfolder()) + .generate(); + } + } + + private void generateCentralArgoCDRBAC() { + for (String ns : config.activeNamespaces()) { + log.debug("Generate RBAC permissions for centralized ArgoCD to access tenant ArgoCDs"); + + new RbacDefinition(Role.Variant.ARGOCD).withName("argocd-central") + .withNamespace(ns) + .withServiceAccountsFrom( + config.centralNamespace(), ARGOCD_SERVICE_ACCOUNTS + ) + .withTemplateConfig(config.rbacTemplateConfig()) + .withRepo(repositoryWorkspace.getClusterResourcesRepository()) + .withSubfolder(ArgoCDRepoLayout.operatorRbacSubfolder()) + .generate(); + } + } + + private void updateCentralManagedNamespaces() { + String base64Namespaces = (String) k8sClient.getArgoCDNamespacesSecret( + ARGOCD_DEFAULT_CLUSTER_CONFIG, config.centralNamespace() + ); + + String decoded = ""; + if (base64Namespaces != null) { + byte[] decodedBytes = Base64.getDecoder().decode(base64Namespaces); + decoded = new String(decodedBytes, StandardCharsets.UTF_8); + } + + List decodedList = decoded.isEmpty() ? new ArrayList<>() : Arrays.asList(decoded.split(",")); + java.util.Collection activeList = config.activeNamespaces(); + if (activeList == null) { + activeList = new ArrayList<>(); + } + + List mergedList = new ArrayList<>(decodedList); + mergedList.addAll(activeList); + String merged = mergedList.stream().distinct().collect(Collectors.joining(",")); + + log.debug("Updating Central Argocd 'argocd-default-cluster-config' secret"); + + k8sClient.patch( + SECRET_RESOURCE, + ARGOCD_DEFAULT_CLUSTER_CONFIG, + config.centralNamespace(), + Map.of("stringData", Map.of("namespaces", merged)) + ); + } + + private void createRepoCredentialsSecret( + String secretName, + String ns, + String url, + String username, + String password) { + k8sClient.createSecret( + "generic", + secretName, + ns, + new Tuple<>("url", url), + new Tuple<>("username", username), + new Tuple<>("password", password) + ); + + k8sClient.label(SECRET_RESOURCE, secretName, ns, new Tuple<>("argocd.argoproj.io/secret-type", "repo-creds")); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DeploymentMode.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DeploymentMode.java new file mode 100644 index 000000000..257b355e5 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DeploymentMode.java @@ -0,0 +1,20 @@ +package com.cloudogu.gitops.tools.core.argocd.mode; + +import java.util.List; + +public interface DeploymentMode { + + List ARGOCD_SERVICE_ACCOUNTS = List.of( + "argocd-argocd-server", + "argocd-argocd-application-controller", + "argocd-applicationset-controller" + ); + + void createSCMCredentialsSecret(); + + void generateRBAC(); + + void updateManagedNamespaces(); + + void applyBootstrapResources(); +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DeploymentModeFactory.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DeploymentModeFactory.java new file mode 100644 index 000000000..ec2fb63f9 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/DeploymentModeFactory.java @@ -0,0 +1,44 @@ +package com.cloudogu.gitops.tools.core.argocd.mode; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDRepoLayout; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDRepoSetup; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDToolConfig; +import jakarta.inject.Singleton; + +@Singleton +public class DeploymentModeFactory { + + public DeploymentMode create( + ArgoCDToolConfig config, + K8sClient k8sClient, + GitHandler gitHandler, + RepositoryWorkspace repositoryWorkspace, + ArgoCDRepoSetup repoSetup, + ArgoCDRepoLayout clusterResourcesRepo, + String namespace) { + + if (config.multiTenant()) { + return new DedicatedMultiTenantMode( + config, + k8sClient, + gitHandler, + repositoryWorkspace, + repoSetup, + clusterResourcesRepo, + namespace + ); + } + + return new SingleTenantMode( + config, + k8sClient, + gitHandler, + repositoryWorkspace, + clusterResourcesRepo, + namespace + ); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/SingleTenantMode.java b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/SingleTenantMode.java new file mode 100644 index 000000000..43e34b606 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/argocd/mode/SingleTenantMode.java @@ -0,0 +1,107 @@ +package com.cloudogu.gitops.tools.core.argocd.mode; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.infrastructure.kubernetes.rbac.RbacDefinition; +import com.cloudogu.gitops.infrastructure.kubernetes.rbac.Role; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDRepoLayout; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDToolConfig; +import com.cloudogu.gitops.utils.Tuple; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.nio.file.Path; +import java.util.Map; + +@RequiredArgsConstructor +@Slf4j +public class SingleTenantMode implements DeploymentMode { + + private final ArgoCDToolConfig config; + private final K8sClient k8sClient; + private final GitHandler gitHandler; + private final RepositoryWorkspace repositoryWorkspace; + private final ArgoCDRepoLayout clusterResourcesRepo; + private final String namespace; + + @Override + public void createSCMCredentialsSecret() { + log.debug( + "Creating repo credential secret that is used by ArgoCD to access repos in {}", config.scmProviderType() + ); + + createRepoCredentialsSecret( + "argocd-repo-creds-scm", namespace, gitHandler.getTenant() + .getUrl(), gitHandler.getTenant() + .getCredentials() + .getUsername(), gitHandler.getTenant() + .getCredentials() + .getPassword() + ); + } + + @Override + public void generateRBAC() { + log.debug("Generate RBAC permissions for ArgoCD in all managed namespaces"); + + for (String ns : config.activeNamespaces()) { + new RbacDefinition(Role.Variant.ARGOCD).withName("argocd") + .withNamespace(ns) + .withServiceAccountsFrom(namespace, ARGOCD_SERVICE_ACCOUNTS) + .withTemplateConfig(config.rbacTemplateConfig()) + .withRepo(repositoryWorkspace.getClusterResourcesRepository()) + .withSubfolder(ArgoCDRepoLayout.operatorRbacSubfolder()) + .generate(); + } + + if (config.clusterAdmin()) { + new RbacDefinition(Role.Variant.CLUSTER_ADMIN).withName("argocd-cluster-admin") + .withNamespace(namespace) + .withServiceAccountsFrom(namespace, ARGOCD_SERVICE_ACCOUNTS) + .withTemplateConfig(config.rbacTemplateConfig()) + .withRepo(repositoryWorkspace.getClusterResourcesRepository()) + .withSubfolder(ArgoCDRepoLayout.operatorRbacSubfolder()) + .generate(); + } + } + + @Override + public void updateManagedNamespaces() { + log.debug("Updating managed namespaces in ArgoCD configuration secret."); + + k8sClient.patch( + "secret", "argocd-default-cluster-config", namespace, Map.of( + "stringData", Map.of( + "namespaces", String.join( + ",", config.activeNamespaces() + ) + ) + ) + ); + } + + @Override + public void applyBootstrapResources() { + k8sClient.applyYaml(Path.of(clusterResourcesRepo.projectsDir(), "argocd.yaml").toString()); + k8sClient.applyYaml(Path.of(clusterResourcesRepo.applicationsDir(), "bootstrap.yaml").toString()); + } + + private void createRepoCredentialsSecret( + String secretName, + String ns, + String url, + String username, + String password) { + k8sClient.createSecret( + "generic", + secretName, + ns, + new Tuple<>("url", url), + new Tuple<>("username", username), + new Tuple<>("password", password) + ); + + k8sClient.label("secret", secretName, ns, new Tuple<>("argocd.argoproj.io/secret-type", "repo-creds")); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManager.java b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManager.java new file mode 100644 index 000000000..671bba33a --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManager.java @@ -0,0 +1,127 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManagerProvider; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractMappedTool; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import io.micronaut.core.annotation.Order; +import jakarta.inject.Singleton; +import lombok.Getter; +import lombok.Setter; +import lombok.extern.slf4j.Slf4j; + +@Singleton +@Order(10) +@Slf4j +public class ScmManager extends AbstractMappedTool { + + @Getter + @Setter + private String namespace; + private final ImagePullSecretCreator imagePullSecretCreator; + private final ScmManagerConfigUpdater configUpdater; + private final K8sClient k8sClient; + private ScmManagerSetup setup; + + public ScmManager( + GitHandler gitHandler, + Deployer deployer, + FileSystemUtils fileSystemUtils, + AirGappedUtils airGappedUtils, + ImagePullSecretCreator imagePullSecretCreator, + ScmManagerToolConfigMapper configMapper, + ScmManagerConfigUpdater configUpdater, + K8sClient k8sClient) { + super(configMapper); + this.gitHandler = gitHandler; + this.deployer = deployer; + this.fileSystemUtils = fileSystemUtils; + this.airGappedUtils = airGappedUtils; + this.imagePullSecretCreator = imagePullSecretCreator; + this.configUpdater = configUpdater; + this.k8sClient = k8sClient; + } + + @Override + protected boolean isEnabled(ScmManagerToolConfig config) { + return config.active(); + } + + @Override + protected void preDeploy() { + log.info("Preparing internal SCM-Manager deployment."); + + prepareNamespace(); + imagePullSecretCreator.createIfRequired(toolConfig().imagePullSecret(), namespace); + + ScmManagerProvider scmManager = getTenantScmManager(); + + this.setup = new ScmManagerSetup( + scmManager, deployer, context, repositoryWorkspace, fileSystemUtils, toolConfig(), k8sClient + ); + } + + @Override + protected void deploy() { + log.info("Deploying internal SCM-Manager."); + + setup.setupHelm(); + setup.waitForScmmAvailable(); + } + + @Override + protected void postDeploy() { + log.info("Configuring internal SCM-Manager after deployment."); + + setup.configure(); + + /* + * Special bootstrap preparation: + * Creates/initializes the remote repositories and prepares the local workspace + * from the remote main branch before generated GitOps artifacts are written. + */ + setup.prepareBootstrapRepositoriesAfterScmManagerDeployment(); + + /* + * The SCM-Manager ArgoCD Application is created through ArgoCdApplicationStrategy. + * The strategy writes into the shared RepositoryWorkspace and does not push itself. + */ + setup.createArgocdApplication(); + } + + @Override + protected void publishChanges() { + /* + * Push the complete bootstrap state, including generated SCM-Manager GitOps artifacts. + */ + setup.pushBootstrapRepositoriesAfterScmManagerDeployment(); + + log.info("Internal SCM-Manager setup finished."); + } + + private void prepareNamespace() { + this.namespace = activeNamespace(toolConfig()); + configUpdater.updateNamespace(namespace); + } + + @Override + protected String activeNamespace(ScmManagerToolConfig config) { + return config.namespace(); + } + + private ScmManagerProvider getTenantScmManager() { + GitProvider tenantScm = gitHandler.getTenant(); + + if (!(tenantScm instanceof ScmManagerProvider)) { + throw new IllegalStateException("Tenant SCM provider is not an SCM-Manager. Actual provider: " + (tenantScm != null ? tenantScm.getClass() + .getSimpleName() : "null")); + } + + return (ScmManagerProvider) tenantScm; + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerConfigUpdater.java b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerConfigUpdater.java new file mode 100644 index 000000000..27a8c146d --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerConfigUpdater.java @@ -0,0 +1,16 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.config.Config; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +@Singleton +@RequiredArgsConstructor +public class ScmManagerConfigUpdater { + + private final Config config; + + public void updateNamespace(String namespace) { + config.getScm().getScmManager().setNamespace(namespace); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerSetup.java b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerSetup.java new file mode 100644 index 000000000..63ad9c4b3 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerSetup.java @@ -0,0 +1,368 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManagerProvider; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerUser; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.MapUtils; +import com.cloudogu.gitops.utils.TemplatingEngine; +import com.cloudogu.gitops.utils.Tuple; +import freemarker.template.Configuration; +import freemarker.template.DefaultObjectWrapperBuilder; +import freemarker.template.TemplateModel; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +@RequiredArgsConstructor +@Slf4j +public class ScmManagerSetup { + + private static final String HELM_VALUES_PATH = "argocd/cluster-resources/apps/scm-manager/templates/values.ftl.yaml"; + private static final long MILLIS_PER_SECOND = 1000L; + private static final int SCMM_AVAILABILITY_TIMEOUT_SECONDS = 180; + private static final int SCMM_AVAILABILITY_POLL_INTERVAL_MILLIS = 5000; + private static final int SCMM_RESTART_POLL_INTERVAL_MILLIS = 2000; + private static final int SCMM_RESTART_START_DELAY_MILLIS = 100; + private static final int DEFAULT_PROXY_PORT = 8080; + private static final int DEFAULT_LOGIN_ATTEMPT_LIMIT_TIMEOUT_SECONDS = 300; + static final String CREDENTIALS_SECRET_NAME = "scm-manager-credentials"; + + private final ScmManagerProvider scmManager; + private final Deployer deployer; + private final DeploymentContext context; + private final RepositoryWorkspace repositoryWorkspace; + private final FileSystemUtils fileSystemUtils; + private final ScmManagerToolConfig config; + private final K8sClient k8sClient; + + private Path tempValuesPath; + + public void setupHelm() { + createCredentialsSecret(); + Path valuesPath = prepareHelmValues(); + HelmChartConfig helmConfig = config.helm(); + String releaseName = scmmReleaseName(); + + log.info( + "Deploying SCM-Manager via Helm with releaseName='{}', namespace='{}', namePrefix='{}', dedicatedInstance={}", + releaseName, + config.namespace(), + config.namePrefix(), + config.multiTenant() + ); + + deployer.getHelmStrategy() + .deployFeature( + helmConfig.repoURL(), + "scm-manager", + helmConfig.chart(), + helmConfig.version(), + config.namespace(), + releaseName, + valuesPath, + DeploymentStrategy.RepoType.HELM + ); + } + + public void createArgocdApplication() { + Path valuesPath = tempValuesPath != null ? tempValuesPath : prepareHelmValues(); + HelmChartConfig helmConfig = config.helm(); + String releaseName = scmmReleaseName(); + + log.info( + "Creating SCM-Manager ArgoCD application with releaseName='{}', namespace='{}', namePrefix='{}', dedicatedInstance={}", + releaseName, + config.namespace(), + config.namePrefix(), + config.multiTenant() + ); + + deployer.deployFeature( + helmConfig.repoURL(), + "scm-manager", + helmConfig.chart(), + helmConfig.version(), + config.namespace(), + releaseName, + valuesPath, + DeploymentStrategy.RepoType.HELM, + false, + context, + repositoryWorkspace + ); + } + + public void prepareBootstrapRepositoriesAfterScmManagerDeployment() { + try { + repositoryWorkspace.ensureRemoteRepositoriesExist(); + repositoryWorkspace.initLocalRepositoriesIfNeeded(); + repositoryWorkspace.alignWithRemoteMainIfPresent(); + repositoryWorkspace.createLocalDirectories(); + } catch (Exception e) { + throw new RuntimeException("Failed to prepare bootstrap repositories", e); + } + } + + public void pushBootstrapRepositoriesAfterScmManagerDeployment() { + try { + repositoryWorkspace.commitAndPushClusterResourcesChanges( + "Bootstrap cluster-resources repository after SCM-Manager deployment"); + + if (repositoryWorkspace.hasTenantBootstrapRepository()) { + repositoryWorkspace.commitAndPushTenantBootstrapChanges( + "Bootstrap tenant repository after SCM-Manager deployment"); + } + } catch (Exception e) { + throw new RuntimeException("Failed to push bootstrap repositories", e); + } + } + + private Path prepareHelmValues() { + String releaseName = scmmReleaseName(); + + log.debug( + "Preparing SCM-Manager Helm values with releaseName='{}', namespace='{}'", + releaseName, + config.namespace() + ); + + Map templateVars = new HashMap<>(); + templateVars.put("config", config.templateConfig()); + templateVars.put("host", config.ingress()); + templateVars.put("credentialsSecretName", CREDENTIALS_SECRET_NAME); + templateVars.put("helm", config.helm()); + templateVars.put("releaseName", releaseName); + + try { + TemplateModel statics = new DefaultObjectWrapperBuilder(Configuration.VERSION_2_3_32).build() + .getStaticModels(); + templateVars.put("statics", statics); + } catch (Exception e) { + throw new RuntimeException("Failed to expose freemarker statics model", e); + } + + Map templatedMap = TemplatingEngine.templateToMap(HELM_VALUES_PATH, templateVars); + Map values = config.helm().values(); + + Map mergedMap = MapUtils.deepMerge(values, templatedMap); + tempValuesPath = fileSystemUtils.writeTempFile(mergedMap); + + return tempValuesPath; + } + + private void createCredentialsSecret() { + var runtimeCredentials = scmManager.getCredentials(); + k8sClient.createNamespace(config.namespace()); + k8sClient.createSecret( + "generic", + CREDENTIALS_SECRET_NAME, + config.namespace(), + new Tuple<>("SCM_WEBAPP_INITIALUSER", runtimeCredentials.getUsername()), + new Tuple<>("SCM_WEBAPP_INITIALPASSWORD", runtimeCredentials.getPassword()) + ); + } + + private String scmmReleaseName() { + return config.releaseName(); + } + + public void waitForScmmAvailable() { + waitForScmmAvailable(SCMM_AVAILABILITY_TIMEOUT_SECONDS, SCMM_AVAILABILITY_POLL_INTERVAL_MILLIS, 0); + } + + public void waitForScmmAvailable(int timeoutSeconds, int intervalMillis, int startDelay) { + long startTime = System.currentTimeMillis(); + long timeoutMillis = timeoutSeconds * MILLIS_PER_SECOND; + + if (startDelay > 0) { + try { + Thread.sleep(startDelay); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Interrupted while waiting for SCM-Manager", e); + } + } + + while (System.currentTimeMillis() - startTime < timeoutMillis) { + try { + retrofit2.Call call = scmManager.getApiClient().generalApi().checkScmmAvailable(); + retrofit2.Response response = call.execute(); + + if (response.isSuccessful()) { + log.debug("SCM-Manager is available."); + return; + } + } catch (Exception e) { + log.debug("Waiting for SCM-Manager... Error: {}", e.getMessage()); + } + + try { + Thread.sleep(intervalMillis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Interrupted while waiting for SCM-Manager", e); + } + } + + throw new IllegalStateException("Timeout: SCM-Manager did not respond with 200 OK within " + timeoutSeconds + " seconds"); + } + + public void configure() { + installScmmPlugins(); + setSetupConfigs(); + + if (config.jenkinsActive()) { + configureJenkinsPlugin(); + } + + addDefaultUsers(); + + log.info("ScmManager Setup finished!"); + } + + private void installScmmPlugins() { + if (config.skipPlugins()) { + log.debug("Skipping SCM plugin installation"); + return; + } + + List pluginNames = new ArrayList<>(List.of( + "scm-mail-plugin", + "scm-review-plugin", + "scm-code-editor-plugin", + "scm-editor-plugin", + "scm-landingpage-plugin", + "scm-el-plugin", + "scm-readme-plugin", + "scm-webhook-plugin", + "scm-ci-plugin", + "scm-metrics-prometheus-plugin" + )); + + if (config.jenkinsActive()) { + pluginNames.add("scm-jenkins-plugin"); + } + + boolean restartForThisPlugin = false; + + for (int i = 0; i < pluginNames.size(); i++) { + String pluginName = pluginNames.get(i); + log.debug("Installing Plugin {} ...", pluginName); + + restartForThisPlugin = !config.skipRestart() && i == pluginNames.size() - 1; + + ScmManagerApiClient.handleApiResponse(scmManager.getApiClient() + .pluginApi() + .install(pluginName, restartForThisPlugin)); + } + + log.debug("SCM-Manager plugin installation finished successfully!"); + + if (restartForThisPlugin) { + waitForScmmAvailable( + SCMM_AVAILABILITY_TIMEOUT_SECONDS, + SCMM_RESTART_POLL_INTERVAL_MILLIS, + SCMM_RESTART_START_DELAY_MILLIS + ); + } + } + + private void setSetupConfigs() { + Map setupConfigs = new HashMap<>(); + setupConfigs.put("enableProxy", false); + setupConfigs.put("proxyPort", DEFAULT_PROXY_PORT); + setupConfigs.put("proxyServer", "proxy.mydomain.com"); + setupConfigs.put("proxyUser", null); + setupConfigs.put("proxyPassword", null); + setupConfigs.put("realmDescription", "SONIA :: SCM Manager"); + setupConfigs.put("disableGroupingGrid", false); + setupConfigs.put("dateFormat", "YYYY-MM-DD HH:mm:ss"); + setupConfigs.put("anonymousAccessEnabled", false); + setupConfigs.put("anonymousMode", "OFF"); + setupConfigs.put("baseUrl", this.scmManager.getUrl()); + setupConfigs.put("forceBaseUrl", false); + setupConfigs.put("loginAttemptLimit", -1); + setupConfigs.put("proxyExcludes", new ArrayList<>()); + setupConfigs.put("skipFailedAuthenticators", false); + setupConfigs.put( + "pluginUrl", + "https://plugin-center-api.scm-manager.org/api/v1/plugins/{version}?os={os}&arch={arch}" + ); + setupConfigs.put("loginAttemptLimitTimeout", DEFAULT_LOGIN_ATTEMPT_LIMIT_TIMEOUT_SECONDS); + setupConfigs.put("enabledXsrfProtection", true); + setupConfigs.put("namespaceStrategy", "CustomNamespaceStrategy"); + setupConfigs.put("loginInfoUrl", "https://login-info.scm-manager.org/api/v1/login-info"); + setupConfigs.put("releaseFeedUrl", "https://scm-manager.org/download/rss.xml"); + setupConfigs.put("mailDomainName", "scm-manager.local"); + setupConfigs.put("adminGroups", new ArrayList<>()); + setupConfigs.put("adminUsers", new ArrayList<>()); + + ScmManagerApiClient.handleApiResponse(scmManager.getApiClient().generalApi().setConfig(setupConfigs)); + + log.debug("Successfully added SCMM Setup Configs"); + } + + private void configureJenkinsPlugin() { + Map jenkinsPluginConfig = new HashMap<>(); + jenkinsPluginConfig.put("disableRepositoryConfiguration", false); + jenkinsPluginConfig.put("disableMercurialTrigger", false); + jenkinsPluginConfig.put("disableGitTrigger", false); + jenkinsPluginConfig.put("disableEventTrigger", false); + jenkinsPluginConfig.put("url", config.jenkinsUrl()); + + ScmManagerApiClient.handleApiResponse(this.scmManager.getApiClient() + .pluginApi() + .configureJenkinsPlugin(jenkinsPluginConfig)); + + log.debug("Successfully configured JenkinsPlugin in SCM-Manager."); + } + + private void addDefaultUsers() { + String metricsUsername = config.namePrefix() + "metrics"; + String runtimePassword = scmManager.getCredentials().getPassword(); + + addUser( + config.gitOpsUsername(), runtimePassword, "changeme@test.local" + ); + addUser(metricsUsername, runtimePassword, "changeme@test.local"); + grantUserPermissions(metricsUsername, List.of("metrics:read")); + } + + private void addUser(String username, String password, String email) { + ScmManagerUser userRequest = new ScmManagerUser(); + userRequest.setName(username); + userRequest.setDisplayName(username); + userRequest.setMail(email); + userRequest.setExternal(false); + userRequest.setPassword(password); + userRequest.setActive(true); + userRequest.setLinks(new HashMap<>()); + + ScmManagerApiClient.handleApiResponse(scmManager.getApiClient().usersApi().addUser(userRequest)); + + log.debug("Successfully created SCM-Manager User {}.", username); + } + + private void grantUserPermissions(String username, List permissions) { + Map> permissionBody = new HashMap<>(); + permissionBody.put("permissions", permissions); + + ScmManagerApiClient.handleApiResponse(scmManager.getApiClient() + .usersApi() + .setPermissionForUser(username, permissionBody)); + + log.debug("Granted permissions {} to user {}.", permissions, username); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfig.java b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfig.java new file mode 100644 index 000000000..dea5e0583 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfig.java @@ -0,0 +1,31 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import com.cloudogu.gitops.tools.common.ImmutableConfigData; +import lombok.Builder; + +import java.util.Map; + +@Builder +public record ScmManagerToolConfig( + boolean active, + boolean multiTenant, + String namePrefix, + String namespace, + String releaseName, + String ingress, + String gitOpsUsername, + boolean skipPlugins, + boolean skipRestart, + boolean jenkinsActive, + String jenkinsUrl, + HelmChartConfig helm, + ImagePullSecretConfig imagePullSecret, + Map templateConfig +) { + + public ScmManagerToolConfig { + templateConfig = ImmutableConfigData.copyMap(templateConfig); + } +} diff --git a/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfigMapper.java b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfigMapper.java new file mode 100644 index 000000000..7587fda63 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfigMapper.java @@ -0,0 +1,64 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.tools.common.TemplateConfig; +import com.cloudogu.gitops.tools.common.ToolConfigMapper; +import com.cloudogu.gitops.tools.common.ToolConfigMapperSupport; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; + +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +public class ScmManagerToolConfigMapper implements ToolConfigMapper { + + private final Config config; + + @Override + public ScmManagerToolConfig map(DeploymentContext context) { + ScmTenantSchema.ScmManagerTenantConfig scmManager = config.getScm() == null + || config.getScm().getScmManager() == null + ? new ScmTenantSchema.ScmManagerTenantConfig() + : config.getScm().getScmManager(); + String namePrefix = config.getApplication().getNamePrefix() == null ? "" : config.getApplication().getNamePrefix(); + String baseNamespace = scmManager.getNamespace() == null ? "scm-manager" : scmManager.getNamespace(); + String namespace = !namePrefix.isEmpty() && baseNamespace.startsWith(namePrefix) + ? baseNamespace + : namePrefix + baseNamespace; + String releaseName = namePrefix.strip().isEmpty() ? "scmm" : namePrefix.strip() + "scmm"; + + return ScmManagerToolConfig.builder() + .active(context.isInternalScmManager()) + .multiTenant(context.isMultiTenant()) + .namePrefix(namePrefix) + .namespace(namespace) + .releaseName(releaseName) + .ingress(scmManager.getIngress()) + .gitOpsUsername(scmManager.getGitOpsUsername()) + .skipPlugins(scmManager.getSkipPlugins()) + .skipRestart(scmManager.getSkipRestart()) + .jenkinsActive(config.getJenkins().getActive()) + .jenkinsUrl(config.getJenkins().getUrlForScm()) + .helm(ToolConfigMapperSupport.helmChart( + scmManager.getHelm(), + config.getApplication().getLocalHelmChartFolder() + )) + .imagePullSecret(ToolConfigMapperSupport.imagePullSecret(config.getRegistry())) + .templateConfig(templateConfig(config, scmManager)) + .build(); + } + + private static Map templateConfig( + Config config, + ScmTenantSchema.ScmManagerTenantConfig scmManager) { + return new TemplateConfig() + .put("features.certManager.active", config.getFeatures().getCertManager().getActive()) + .put("features.certManager.issuer", config.getFeatures().getCertManager().getIssuer()) + .put("registry.createImagePullSecrets", config.getRegistry().getCreateImagePullSecrets()) + .put("scm.scmManager.scmmImage", scmManager.getScmmImage()) + .values(); + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/AirGappedUtils.java b/src/main/java/com/cloudogu/gitops/utils/AirGappedUtils.java new file mode 100644 index 000000000..6852e9804 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/AirGappedUtils.java @@ -0,0 +1,146 @@ +package com.cloudogu.gitops.utils; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class AirGappedUtils { + + private static final String VERSION_KEY = "version"; + + private final GitRepoFactory repoProvider; + private final FileSystemUtils fileSystemUtils; + private final HelmClient helmClient; + private final GitHandler gitHandler; + + /** + * In air-gapped mode, the chart's dependencies can't be resolved. As helm does not provide an + * option for changing them interactively, we push the charts into a separate repo. We alter these + * repos to resolve dependencies locally from SCM. + * + * @return the repo namespace and name + */ + public String mirrorHelmRepoToGit(HelmChartConfig helmConfig) { + String repoName = helmConfig.chart(); + String namespace = GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES; + String repoNamespaceAndName = namespace + "/" + repoName; + String localHelmChartFolder = helmConfig.localHelmChartFolder() + "/" + repoName; + + validateChart(repoNamespaceAndName, localHelmChartFolder, repoName); + + GitRepo repo = repoProvider.create(repoNamespaceAndName, gitHandler.getTenant()); + + try { + repo.createRepositoryAndSetPermission( + "Mirror of Helm chart " + repoName + " from " + helmConfig.repoURL(), + false + ); + + repo.cloneRepo(); + + repo.copyDirectoryContents(localHelmChartFolder); + + Map chartYaml = localizeChartYaml(repo); + + // Chart.lock contains pinned dependencies and digest. + // We either have to update or remove them. Take the easier approach. + Files.deleteIfExists(Path.of(repo.getAbsoluteLocalRepoTmpDir(), "Chart.lock")); + + repo.commitAndPush( + "Chart " + chartYaml.get("name") + ", version: " + chartYaml.get(VERSION_KEY) + "\n\n" + "Source: " + helmConfig.repoURL() + "\n" + "Dependencies localized to run in air-gapped environments", + String.valueOf(chartYaml.get(VERSION_KEY)) + ); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException("Failed to mirror helm repo to Git for " + repoName, e); + } + return repoNamespaceAndName; + } + + private void validateChart(String repoNamespaceAndName, String localHelmChartFolder, String repoName) { + log.debug( + "Validating helm chart before pushing it to SCM, by running helm template.\n" + "Potential repo: {}, chart folder: {}", + repoNamespaceAndName, + localHelmChartFolder + ); + try { + helmClient.template(repoName, localHelmChartFolder); + } catch (RuntimeException e) { + throw new RuntimeException("Helm chart in folder " + localHelmChartFolder + " seems invalid.", e); + } + } + + private Map localizeChartYaml(GitRepo gitRepo) { + log.debug( + "Preparing repo {} for air-gapped use: Changing Chart.yaml to resolve depencies locally", + gitRepo.getRepoTarget() + ); + + Path chartYamlPath = Path.of(gitRepo.getAbsoluteLocalRepoTmpDir(), "Chart.yaml"); + + Map chartYaml = fileSystemUtils.readYaml(chartYamlPath); + Map chartLock = parseChartLockIfExists(gitRepo); + + List> dependencies = MapUtils.asListOfStringObjectMaps(chartYaml.get("dependencies")); + if (dependencies == null) { + dependencies = Collections.emptyList(); + } + for (Map chartYamlDep : dependencies) { + resolveDependencyVersion(chartLock, chartYamlDep, gitRepo); + + // Remove link to external repo, to force using local one + chartYamlDep.put("repository", ""); + } + fileSystemUtils.writeYaml(chartYaml, chartYamlPath.toFile()); + return chartYaml; + } + + private Map parseChartLockIfExists(GitRepo scmmRepo) { + Path chartLock = Path.of(scmmRepo.getAbsoluteLocalRepoTmpDir(), "Chart.lock"); + if (!Files.exists(chartLock)) { + return Collections.emptyMap(); + } + return fileSystemUtils.readYaml(chartLock); + } + + /** + * Resolve proper dependency version from Chart.lock, e.g. 5.18.* -> 5.18.1 + */ + private void resolveDependencyVersion( + Map chartLock, + Map chartYamlDep, + GitRepo gitRepo) { + List> lockDependencies = MapUtils.asListOfStringObjectMaps(chartLock.get("dependencies")); + Map chartLockDep = findByName(lockDependencies, String.valueOf(chartYamlDep.get("name"))); + if (chartLockDep != null && !chartLockDep.isEmpty()) { + chartYamlDep.put(VERSION_KEY, chartLockDep.get(VERSION_KEY)); + } else if (String.valueOf(chartYamlDep.get(VERSION_KEY)).contains("*")) { + throw new IllegalStateException("Unable to determine proper version for dependency " + chartYamlDep.get( + "name") + " (version: " + chartYamlDep.get(VERSION_KEY) + ") from repo " + gitRepo.getRepoTarget()); + } else { + // version is already pinned (no wildcard); keep it as-is + } + } + + public Map findByName(List> list, String name) { + if (list == null || list.isEmpty()) { + return Collections.emptyMap(); + } + return list.stream().filter(map -> name.equals(map.get("name"))).findFirst().orElse(Collections.emptyMap()); + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/AllowListFreemarkerObjectWrapper.java b/src/main/java/com/cloudogu/gitops/utils/AllowListFreemarkerObjectWrapper.java new file mode 100644 index 000000000..978949c1b --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/AllowListFreemarkerObjectWrapper.java @@ -0,0 +1,41 @@ +package com.cloudogu.gitops.utils; + +import freemarker.template.DefaultObjectWrapper; +import freemarker.template.TemplateHashModel; +import freemarker.template.TemplateModel; +import freemarker.template.TemplateModelException; +import freemarker.template.Version; + +import java.util.HashSet; +import java.util.Set; + +public class AllowListFreemarkerObjectWrapper extends DefaultObjectWrapper { + + private final Set allowlist; + + public AllowListFreemarkerObjectWrapper(Version freemarkerVersion, Set allowlist) { + super(freemarkerVersion); + this.allowlist = new HashSet<>(allowlist); + } + + @Override + public TemplateHashModel getStaticModels() { + final TemplateHashModel originalStaticModels = super.getStaticModels(); + final Set allowlistCopy = this.allowlist; + + return new TemplateHashModel() { + @Override + public TemplateModel get(String key) throws TemplateModelException { + if (allowlistCopy.contains(key)) { + return originalStaticModels.get(key); + } + return null; + } + + @Override + public boolean isEmpty() throws TemplateModelException { + return allowlistCopy.isEmpty(); + } + }; + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/ClusterResourcesCopyFilter.java b/src/main/java/com/cloudogu/gitops/utils/ClusterResourcesCopyFilter.java new file mode 100644 index 000000000..a275e6035 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/ClusterResourcesCopyFilter.java @@ -0,0 +1,104 @@ +package com.cloudogu.gitops.utils; + +import lombok.extern.slf4j.Slf4j; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.Collection; +import java.util.Set; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +@Slf4j +public class ClusterResourcesCopyFilter { + + private static final Pattern LEADING_SLASHES = Pattern.compile("^/+"); + private static final Pattern TRAILING_SLASHES = Pattern.compile("/+$"); + + private ClusterResourcesCopyFilter() { + } + + public static FileFilter forSubDir(String copyFromDirectory, String subDirToCopy) { + return forSubDirs(copyFromDirectory, java.util.List.of(subDirToCopy)); + } + + public static FileFilter forSubDirs(String copyFromDirectory, Collection subDirsToCopy) { + if (subDirsToCopy == null || subDirsToCopy.isEmpty()) { + return allowAllFilter(); + } + + File srcRoot = canonicalFile(copyFromDirectory); + Set prefixes = normalizedPrefixes(subDirsToCopy); + Set templateIncludePrefixes = Set.of("apps/argocd/argocd/templates/"); + + return candidateFile -> matches(candidateFile, srcRoot, prefixes, templateIncludePrefixes); + } + + private static File canonicalFile(String path) { + try { + return new File(path).getCanonicalFile(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to get canonical file for " + path, e); + } + } + + private static Set normalizedPrefixes(Collection subDirsToCopy) { + return subDirsToCopy.stream().map(ClusterResourcesCopyFilter::normalizePrefix).collect(Collectors.toSet()); + } + + private static String normalizePrefix(String subDir) { + String norm = subDir.replace('\\', '/'); + norm = TRAILING_SLASHES.matcher(LEADING_SLASHES.matcher(norm).replaceAll("")).replaceAll(""); + return norm + "/"; + } + + private static boolean matches( + File candidateFile, + File srcRoot, + Set prefixes, + Set templateIncludePrefixes) { + String rel = relativePath(candidateFile, srcRoot); + if (rel == null) { + return false; + } + if (rel.isEmpty() || ".".equals(rel)) { + return true; + } + + boolean isDir = candidateFile.isDirectory(); + String relDir = rel.endsWith("/") ? rel : (rel + "/"); + + if (templateIncludePrefixes.stream().anyMatch((isDir ? relDir : rel)::startsWith)) { + return true; + } + + if (rel.startsWith("apps/") && relDir.contains("/templates/")) { + return false; + } + + if (isDir) { + return prefixes.stream() + .anyMatch(prefix -> relDir.equals(prefix) || relDir.startsWith(prefix) || prefix.startsWith( + relDir)); + } + + return prefixes.stream().anyMatch(rel::startsWith); + } + + private static String relativePath(File candidateFile, File srcRoot) { + try { + File canon = candidateFile.getCanonicalFile(); + String rel = srcRoot.toURI().relativize(canon.toURI()).toString(); + return rel.replace('\\', '/'); + } catch (IOException e) { + log.debug("Failed to compute relative path for {} against {}", candidateFile, srcRoot, e); + return null; + } + } + + private static FileFilter allowAllFilter() { + return file -> true; + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/CommandExecutor.java b/src/main/java/com/cloudogu/gitops/utils/CommandExecutor.java new file mode 100644 index 000000000..edc412cab --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/CommandExecutor.java @@ -0,0 +1,264 @@ +package com.cloudogu.gitops.utils; + +import jakarta.inject.Singleton; +import lombok.Value; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.io.output.TeeOutputStream; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +@Singleton +@Slf4j +public class CommandExecutor { + + /* This timeout is mainly here to not freeze forever the apply process in the worst case scenario. + + Calls to init-scmm.sh and init-jenkins.sh take several minutes at best and might be slower with poor connections + to the internet. + Once they are migrated to groovy we can reduce this timeout.*/ + public static final int PROCESS_TIMEOUT_MINUTES = 15; + + private static final String FAILED_TO_EXECUTE_PREFIX = "Failed to execute command: "; + private static final String EXECUTING_FAILED_PREFIX = "Executing command failed: "; + + public Output execute(String[] command) { + return execute(command, true); + } + + public Output execute(String[] command, boolean failOnError) { + try { + Process proc = doExecute(command); + return getOutput(proc, String.join(" ", command), failOnError); + } catch (IOException e) { + throw new UncheckedIOException(FAILED_TO_EXECUTE_PREFIX + String.join(" ", command), e); + } + } + + /** + * Please prefer using {@link #execute(java.lang.String[], boolean)}, because it avoids quoting + * issues when passing arguments containing whitespaces. + * + * @deprecated use {@link #execute(java.lang.String[], boolean)} instead + */ + @Deprecated(since = "1.0") + public Output execute(String command) { + return execute(command, true); + } + + /** + * @deprecated use {@link #execute(java.lang.String[], boolean)} instead + */ + @Deprecated(since = "1.0") + public Output execute(String command, boolean failOnError) { + try { + Process proc = doExecute(command); + return getOutput(proc, command, failOnError); + } catch (IOException e) { + throw new UncheckedIOException(FAILED_TO_EXECUTE_PREFIX + command, e); + } + } + + public Output execute(String command, Map additionalEnv) { + return execute(command, additionalEnv, true); + } + + /** + * @param additionalEnv a Map of env variables to add + */ + public Output execute(String command, Map additionalEnv, boolean failOnError) { + try { + Map env = new HashMap<>(System.getenv()); + if (additionalEnv != null) { + additionalEnv.forEach((key, value) -> env.put( + String.valueOf(key), + value != null ? String.valueOf(value) : null + )); + } + List envp = env.entrySet() + .stream() + .map(entry -> entry.getKey() + "=" + (entry.getValue() != null ? entry.getValue() : "")) + .toList(); + + Process proc = doExecute(command, envp); + return getOutput(proc, command, failOnError); + } catch (IOException e) { + throw new UncheckedIOException(FAILED_TO_EXECUTE_PREFIX + command, e); + } + } + + public Output execute(String[] command1, String[] command2) { + return execute(command1, command2, true); + } + + public Output execute(String[] command1, String[] command2, boolean failOnError) { + String pipedCommand = String.join(" ", command1) + " | " + String.join(" ", command2); + try { + ProcessBuilder pb1 = new ProcessBuilder(command1); + ProcessBuilder pb2 = new ProcessBuilder(command2); + List processes = ProcessBuilder.startPipeline(List.of(pb1, pb2)); + Process process1 = processes.get(0); + Process process2 = processes.get(1); + + Output finalOutput = getOutput(process2, pipedCommand, false); + // Proc1 should have finished when proc2 has. + // Still, there is the occasional "IllegalThreadStateException: process hasn't exited"... + // concurrency 🤷 + // Avoid the exceptions, by explicitly waiting for the process to end + waitForOrKill(process1, String.join(" ", command1)); + + if (process1.exitValue() > 0) { + log.error("Pipefail! First process of command failed {}.", pipedCommand); + logProcessStderr(process1); + } + if (process2.exitValue() > 0) { + log.error("Executing command failed: {}", pipedCommand); + log.error("Stderr: {}", finalOutput.getStdErr()); + log.error("StdOut: {}", finalOutput.getStdOut()); + } + + boolean success = process1.exitValue() == 0 && process2.exitValue() == 0; + if (!success && failOnError) { + throw new IllegalStateException(EXECUTING_FAILED_PREFIX + pipedCommand); + } + + return finalOutput; + } catch (IOException e) { + throw new UncheckedIOException("Failed to execute piped command: " + pipedCommand, e); + } + } + + private void logProcessStderr(Process process) { + try (InputStream is = process.getErrorStream()) { + ByteArrayOutputStream bos = new ByteArrayOutputStream(); + is.transferTo(bos); + log.error("Stderr: {}", bos.toString(StandardCharsets.UTF_8).trim()); + } catch (IOException e) { + log.debug("Failed to read stderr of process", e); + } + } + + protected Process doExecute(String command, List envp) throws IOException { + log.trace("Executing command: '{}'", command); + String[] envpArray = envp != null ? envp.toArray(new String[0]) : null; + return Runtime.getRuntime().exec(command, envpArray); + } + + protected Process doExecute(String command) throws IOException { + return doExecute(command, null); + } + + protected Process doExecute(String[] command) throws IOException { + log.trace("Executing command: '{}'", (Object) command); + return Runtime.getRuntime().exec(command); + } + + protected Output getOutput(Process proc, String command, boolean failOnError) { + ByteArrayOutputStream stdOut = new ByteArrayOutputStream(); + ByteArrayOutputStream stdErr = new ByteArrayOutputStream(); + OutputStream outDest = stdOut; + OutputStream errDest = stdErr; + + TeeOutputStream teeOut = null; + TeeOutputStream teeErr = null; + + if (log.isTraceEnabled()) { + // While waiting for the process to finish, also print stdout and stderr streams through to + // the main process + teeOut = new TeeOutputStream(stdOut, System.out); + teeErr = new TeeOutputStream(stdErr, System.err); + outDest = teeOut; + errDest = teeErr; + } + + final OutputStream finalOutDest = outDest; + final OutputStream finalErrDest = errDest; + + Thread outThread = new Thread(() -> { + try (InputStream is = proc.getInputStream()) { + is.transferTo(finalOutDest); + } catch (IOException e) { + log.debug("Failed to read stdout of process {}", command, e); + } + }); + Thread errThread = new Thread(() -> { + try (InputStream es = proc.getErrorStream()) { + es.transferTo(finalErrDest); + } catch (IOException e) { + log.debug("Failed to read stderr of process {}", command, e); + } + }); + + outThread.start(); + errThread.start(); + + waitForOrKill(proc, command); + + try { + outThread.join(); + errThread.join(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + + // Make sure all bytes have been written, before returning output + if (teeOut != null) { + try { + teeOut.flush(); + } catch (IOException e) { + log.debug("Failed to flush stdout tee stream for command {}", command, e); + } + } + if (teeErr != null) { + try { + teeErr.flush(); + } catch (IOException e) { + log.debug("Failed to flush stderr tee stream for command {}", command, e); + } + } + + Output output = new Output( + stdErr.toString(StandardCharsets.UTF_8) + .trim(), stdOut.toString(StandardCharsets.UTF_8).trim(), proc.exitValue() + ); + + if (failOnError && proc.exitValue() > 0) { + log.error("Executing command failed: {}", command); + log.error("Stderr: {}", output.getStdErr()); + log.error("StdOut: {}", output.getStdOut()); + throw new IllegalStateException(EXECUTING_FAILED_PREFIX + command); + } + + return output; + } + + protected void waitForOrKill(Process proc, String command) { + try { + boolean processFinished = proc.waitFor(PROCESS_TIMEOUT_MINUTES, TimeUnit.MINUTES); + if (!processFinished) { + log.error("Timeout waiting for command {}. Killing process.", command); + proc.destroyForcibly(); + proc.waitFor(); + } + } catch (InterruptedException e) { + log.error("Interrupted while waiting for command {}. Killing process.", command, e); + proc.destroyForcibly(); + Thread.currentThread().interrupt(); + } + } + + @Value + public static class Output { + String stdErr; + String stdOut; + int exitCode; + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/DockerImageParser.java b/src/main/java/com/cloudogu/gitops/utils/DockerImageParser.java new file mode 100644 index 000000000..d949b4dfe --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/DockerImageParser.java @@ -0,0 +1,77 @@ +package com.cloudogu.gitops.utils; + +import lombok.Getter; +import lombok.RequiredArgsConstructor; + +public class DockerImageParser { + + private static final int MIN_SEGMENTS_WITH_REGISTRY = 2; + private static final int REPOSITORY_SEGMENT_COUNT = 2; + + @Getter + @RequiredArgsConstructor + public static class Image { + private final String registry; + private final String repository; + private final String tag; + + public String getRegistryAndRepositoryAsString() { + if (registry == null || registry.isEmpty()) { + return repository; + } + return registry + "/" + repository; + } + + @Override + public String toString() { + return getRegistryAndRepositoryAsString() + ":" + tag; + } + } + + public static Image parse(String image) { + int lastSlash = image.lastIndexOf('/'); + int lastColon = image.lastIndexOf(':'); + if (lastColon == -1 || lastColon < lastSlash) { + throw new IllegalArgumentException("Cannot set image '" + image + "' due to missing tag. Must be the format '$repository:$tag'"); + } + + ImageAndTag tuple = splitTag(image); + String imageWithoutTag = tuple.imageWithoutTag(); + String tag = tuple.tag(); + + String[] parts = imageWithoutTag.split("/"); + String repository; + String registry; + + if (parts.length >= MIN_SEGMENTS_WITH_REGISTRY) { + repository = parts[parts.length - REPOSITORY_SEGMENT_COUNT] + "/" + parts[parts.length - 1]; + StringBuilder registryBuilder = new StringBuilder(); + for (int i = 0; i < parts.length - REPOSITORY_SEGMENT_COUNT; i++) { + if (i > 0) { + registryBuilder.append("/"); + } + registryBuilder.append(parts[i]); + } + registry = registryBuilder.toString(); + } else { + repository = imageWithoutTag; + registry = ""; + } + + return new Image(registry, repository, tag); + } + + private static ImageAndTag splitTag(String image) { + int lastColon = image.lastIndexOf(':'); + String imageWithoutTag = image.substring(0, lastColon); + String tag = image.substring(lastColon + 1); + return new ImageAndTag(imageWithoutTag, tag); + } + + private record ImageAndTag( + String imageWithoutTag, + + String tag + ) { + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/FileSystemUtils.java b/src/main/java/com/cloudogu/gitops/utils/FileSystemUtils.java new file mode 100644 index 000000000..74496d86a --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/FileSystemUtils.java @@ -0,0 +1,380 @@ +package com.cloudogu.gitops.utils; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator; +import jakarta.inject.Singleton; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.io.FileUtils; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.nio.file.FileVisitResult; +import java.nio.file.Files; +import java.nio.file.NoSuchFileException; +import java.nio.file.Path; +import java.nio.file.SimpleFileVisitor; +import java.nio.file.StandardCopyOption; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.Collections; +import java.util.Map; +import java.util.Set; +import java.util.stream.Stream; + +@Singleton +@Slf4j +public class FileSystemUtils { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final String TEMP_FILE_PREFIX = "gitops-playground-"; + + private static final ObjectMapper yamlMapper = new ObjectMapper(new YAMLFactory().disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER)); + + public static void deleteFile(String path) { + try { + Files.deleteIfExists(Path.of(path)); + } catch (IOException exception) { + log.warn("Failed to delete file {}", path, exception); + } + } + + public static void deleteDir(String path) { + try { + FileUtils.deleteDirectory(new File(path)); + } catch (IOException exception) { + log.warn("Failed to delete directory {}", path, exception); + } + } + + public String getRootDir() { + return System.getProperty("user.dir"); + } + + /** + * Compatibility overload for callers that still use {@link File}. + * + * @param directory root directory; {@code null} is ignored + */ + public static void makeWritable(File directory) { + if (directory != null) { + makeWritable(directory.toPath()); + } + } + + /** + * Makes the given root path and all contained files and directories writable. + * + *

Git and JGit may create and remove temporary lock files while a repository is being + * traversed. Paths that disappear during traversal are therefore skipped. Other I/O failures + * abort the operation. + * + * @param root root path; {@code null} or missing paths are ignored + * @throws UncheckedIOException if the directory tree cannot be processed + */ + public static void makeWritable(Path root) { + if (root == null || Files.notExists(root)) { + return; + } + + try { + Files.walkFileTree(root, new WritableFileVisitor()); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to make directory tree writable: " + root, exception); + } + } + + private static final class WritableFileVisitor extends SimpleFileVisitor { + + @Override + public FileVisitResult preVisitDirectory(Path directory, BasicFileAttributes attributes) throws IOException { + makePathWritable(directory); + return FileVisitResult.CONTINUE; + } + + @Override + public FileVisitResult visitFile(Path file, BasicFileAttributes attributes) throws IOException { + makePathWritable(file); + return FileVisitResult.CONTINUE; + } + + @Override + public FileVisitResult visitFileFailed(Path file, IOException exception) throws IOException { + if (exception instanceof NoSuchFileException) { + log.debug("Skipping path that disappeared during traversal: {}", file); + + return FileVisitResult.CONTINUE; + } + + throw exception; + } + + private static void makePathWritable(Path path) throws IOException { + try { + if (path.toFile().setWritable(true)) { + return; + } + + /* + * The path may have disappeared between discovery and the + * permission change. Temporary Git lock files commonly exhibit + * this behavior. + */ + if (Files.notExists(path)) { + return; + } + + throw new IOException("Failed to make path writable: " + path); + } catch (SecurityException exception) { + throw new IOException("Insufficient permissions to make path writable: " + path, exception); + } + } + } + + public void copyDirectory(String source, String destination, FileFilter fileFilter) { + log.debug("Copying directory {} to {}", source, destination); + + try { + FileUtils.copyDirectory(new File(source), new File(destination), fileFilter); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to copy directory from " + source + " to " + destination, exception); + } + } + + public void createDirectory(String directory) { + log.trace("Creating directory: {}", directory); + + try { + Files.createDirectories(Path.of(directory)); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to create directory: " + directory, exception); + } + } + + public Path copyToTempDir(String filePath) { + Path sourcePath = Path.of(filePath); + + try { + Path destinationDirectory = Files.createTempDirectory(TEMP_FILE_PREFIX); + + Path destinationPath = destinationDirectory.resolve(sourcePath.getFileName()); + + return Files.copy(sourcePath, destinationPath, StandardCopyOption.REPLACE_EXISTING); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to copy " + filePath + " to temporary directory", exception); + } + } + + public Path createTempFile() { + try { + Path file = Files.createTempFile(TEMP_FILE_PREFIX, ""); + + file.toFile().deleteOnExit(); + + return file; + } catch (IOException exception) { + throw new UncheckedIOException("Failed to create temporary file", exception); + } + } + + public Map readYaml(Path path) { + if (Files.exists(path)) { + try { + return yamlMapper.readValue(path.toFile(), YAML_MAP_TYPE); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to parse YAML file: " + path, exception); + } + } + + String resourceName = normalizeClasspathResource(path); + + log.debug("Path {} not found on filesystem, trying classpath: {}", path, resourceName); + + try (InputStream inputStream = FileSystemUtils.class.getResourceAsStream(resourceName)) { + + if (inputStream == null) { + log.warn("Could not find YAML at {} or on classpath {}", path, resourceName); + + return Collections.emptyMap(); + } + + return yamlMapper.readValue(inputStream, YAML_MAP_TYPE); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to read YAML resource from classpath: " + resourceName, exception); + } + } + + private static String normalizeClasspathResource(Path path) { + String resourceName = path.toString() + .replace('\\', '/') + .replace("/src/main/resources", "") + .replace("src/main/resources", ""); + + if (!resourceName.startsWith("/")) { + resourceName = "/" + resourceName; + } + + return resourceName; + } + + public Path writeTempFile(Map mapValues) { + Path temporaryHelmValues = createTempFile(); + + writeYaml(mapValues, temporaryHelmValues.toFile()); + + return temporaryHelmValues; + } + + public void writeYaml(Map yaml, File file) { + try { + yamlMapper.writeValue(file, yaml); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to write YAML to file: " + file, exception); + } + } + + public void deleteFilesExcept(File parentPath, String... fileOrFolderNamesToKeep) { + File[] files = parentPath.listFiles(); + + if (files == null) { + return; + } + + Set namesToKeep = Set.of(fileOrFolderNamesToKeep); + + for (File file : files) { + if (namesToKeep.contains(file.getName())) { + continue; + } + + try { + if (file.isDirectory()) { + FileUtils.deleteDirectory(file); + } else { + Files.deleteIfExists(file.toPath()); + } + } catch (IOException exception) { + throw new UncheckedIOException("Failed to delete path: " + file, exception); + } + } + } + + /** + * Moves all direct children of {@code sourceDir} into {@code targetDir}. + * + *

Existing files are overwritten. Directories are merged recursively. + */ + public void moveDirectoryMergeOverwrite(Path sourceDir, Path targetDir) { + try { + if (Files.notExists(targetDir)) { + if (tryMoveDirectoryDirect(sourceDir, targetDir)) { + return; + } + } else if (!Files.isDirectory(targetDir)) { + Files.delete(targetDir); + Files.createDirectories(targetDir); + } else { + // targetDir already exists as a directory; merge into it below + } + + mergeDirectoryChildren(sourceDir, targetDir); + + Files.deleteIfExists(sourceDir); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to move directory " + sourceDir + " to " + targetDir, exception); + } + } + + private boolean tryMoveDirectoryDirect(Path sourceDir, Path targetDir) throws IOException { + Path parent = targetDir.getParent(); + + if (parent != null) { + Files.createDirectories(parent); + } + + try { + Files.move(sourceDir, targetDir); + return true; + } catch (IOException moveException) { + log.debug( + "Could not move directory directly from {} to {}; falling back to recursive merge", + sourceDir, + targetDir, + moveException + ); + + Files.createDirectories(targetDir); + return false; + } + } + + private void mergeDirectoryChildren(Path sourceDir, Path targetDir) throws IOException { + try (Stream children = Files.list(sourceDir)) { + for (Path child : children.toList()) { + Path destination = targetDir.resolve(child.getFileName()); + + if (Files.isDirectory(child)) { + moveDirectoryMergeOverwrite(child, destination); + } else { + moveFileOverwrite(child, destination); + } + } + } + } + + private void moveFileOverwrite(Path sourceFile, Path targetFile) { + try { + Path parent = targetFile.getParent(); + + if (parent != null) { + Files.createDirectories(parent); + } + + moveOrCopyFile(sourceFile, targetFile); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to move file " + sourceFile + " to " + targetFile, exception); + } + } + + private void moveOrCopyFile(Path sourceFile, Path targetFile) throws IOException { + try { + Files.move(sourceFile, targetFile, StandardCopyOption.REPLACE_EXISTING); + } catch (IOException moveException) { + log.debug( + "Could not move file directly from {} to {}; falling back to copy and delete", + sourceFile, + targetFile, + moveException + ); + + Files.copy(sourceFile, targetFile, StandardCopyOption.REPLACE_EXISTING); + + Files.delete(sourceFile); + } + } + + /** + * Filter for copying directory content without Git metadata. + */ + public static class IgnoreDotGitFolderFilter implements FileFilter { + + @Override + public boolean accept(File file) { + return !containsGitDirectory(file.toPath()); + } + + private static boolean containsGitDirectory(Path path) { + for (Path part : path) { + if (".git".equals(part.toString())) { + return true; + } + } + + return false; + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/MapUtils.java b/src/main/java/com/cloudogu/gitops/utils/MapUtils.java new file mode 100644 index 000000000..995c23425 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/MapUtils.java @@ -0,0 +1,68 @@ +package com.cloudogu.gitops.utils; + +import java.util.List; +import java.util.Map; + +public class MapUtils { + + private MapUtils() { + } + + public static Map deepMerge(Map src, Map target) { + if (src == null) { + return target; + } + src.forEach((String key, Object value) -> { + Object oldVal = target.containsKey(key) ? target.get(key) : null; + if (oldVal instanceof Map && value instanceof Map) { + target.put(key, deepMerge(asStringObjectMap(value), asStringObjectMap(oldVal))); + } else { + target.put(key, value); + } + }); + return target; + } + + public static Map deepMergeDefaults(Map src, Map target) { + if (src == null) { + return target; + } + src.forEach((String key, Object value) -> mergeDefaultEntry(key, value, target)); + return target; + } + + private static void mergeDefaultEntry(String key, Object value, Map target) { + if (value == null && target.containsKey(key)) { + return; + } + + Object oldVal = target.containsKey(key) ? target.get(key) : null; + if (oldVal instanceof Map && value instanceof Map) { + target.put(key, deepMergeDefaults(asStringObjectMap(value), asStringObjectMap(oldVal))); + } else { + target.put(key, value); + } + } + + /** + * Casts untyped YAML/JSON data or a nested map value to {@code Map}. + * + *

By convention, every map produced by our YAML/JSON parsing has {@code String} keys, but + * generic type erasure means the JVM can only verify at runtime that {@code value} is a raw + * {@code Map}, not that it is parameterized with {@code String} keys. Callers are expected to + * have already checked {@code value instanceof Map} (or know it from the surrounding YAML/JSON + * schema) before calling this. + */ + @SuppressWarnings("unchecked") + public static Map asStringObjectMap(Object value) { + return (Map) value; + } + + /** + * Same rationale as {@link #asStringObjectMap(Object)}, but for a list of such maps. + */ + @SuppressWarnings("unchecked") + public static List> asListOfStringObjectMaps(Object value) { + return (List>) value; + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/NetworkingUtils.java b/src/main/java/com/cloudogu/gitops/utils/NetworkingUtils.java new file mode 100644 index 000000000..f0beecbe5 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/NetworkingUtils.java @@ -0,0 +1,93 @@ +package com.cloudogu.gitops.utils; + +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import jakarta.inject.Singleton; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.io.UncheckedIOException; +import java.net.InetAddress; +import java.net.NetworkInterface; +import java.net.SocketException; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; + +@Singleton +@RequiredArgsConstructor +@Slf4j +public class NetworkingUtils { + + private final K8sClient k8sClient; + private final CommandExecutor commandExecutor; + + public NetworkingUtils() { + this(new K8sClient(), new CommandExecutor()); + } + + public NetworkingUtils(K8sClient k8sClient) { + this(k8sClient, new CommandExecutor()); + } + + public String createUrl(String hostname, String port) { + return createUrl(hostname, port, ""); + } + + public String createUrl(String hostname, String port, String postfix) { + String url = "http://" + hostname + ":" + port + postfix; + log.debug("Creating url: {}", url); + return url; + } + + public String findClusterBindAddress() { + log.debug("Figuring out the address of the k8s cluster"); + + String potentialClusterBindAddress = k8sClient.waitForInternalNodeIp(); + if (potentialClusterBindAddress != null) { + potentialClusterBindAddress = potentialClusterBindAddress.replace("'", ""); + } + + String localAddress = getLocalAddress(); + + log.debug("Local address: {}", localAddress); + log.debug("Cluster address: {}", potentialClusterBindAddress); + + if (potentialClusterBindAddress == null || potentialClusterBindAddress.isEmpty()) { + throw new IllegalStateException("Could not connect to kubernetes cluster: no cluster bind address"); + } + + if (localAddress.equals(potentialClusterBindAddress)) { + log.debug("Local address and cluster bind address are equal, so returning localhost"); + return "localhost"; + } else { + log.debug("Installing on external cluster, so returning cluster ip address"); + return potentialClusterBindAddress; + } + } + + public String getLocalAddress() { + try { + List sortedInterfaces = Collections.list(NetworkInterface.getNetworkInterfaces()); + sortedInterfaces.sort(Comparator.comparingInt(NetworkInterface::getIndex)); + + for (NetworkInterface anInterface : sortedInterfaces) { + String address = firstSiteLocalAddress(anInterface); + if (address != null) { + return address; + } + } + return ""; + } catch (SocketException e) { + throw new UncheckedIOException("Could not determine local ip address", e); + } + } + + private static String firstSiteLocalAddress(NetworkInterface networkInterface) { + for (InetAddress address : Collections.list(networkInterface.getInetAddresses())) { + if (!address.isLoopbackAddress() && address.isSiteLocalAddress()) { + return address.getHostAddress(); + } + } + return null; + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/TemplatingEngine.java b/src/main/java/com/cloudogu/gitops/utils/TemplatingEngine.java new file mode 100644 index 000000000..a856c5189 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/TemplatingEngine.java @@ -0,0 +1,141 @@ +package com.cloudogu.gitops.utils; + +import freemarker.template.Configuration; +import freemarker.template.Template; +import freemarker.template.Version; + +import java.io.BufferedWriter; +import java.io.File; +import java.io.IOException; +import java.io.StringReader; +import java.io.StringWriter; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; +import java.util.stream.Stream; + +public class TemplatingEngine { + private static final Pattern FTL_FILE_PATTERN = Pattern.compile("\\.ftl"); + + private final Configuration engine; + + public TemplatingEngine() { + this(null); + } + + public TemplatingEngine(Configuration engine) { + if (engine == null) { + engine = new Configuration(new Version("2.3.32")); + } + this.engine = engine; + try { + this.engine.setSharedVariable("nullToEmpty", ""); + } catch (Exception e) { + throw new RuntimeException("Failed to set shared variable in freemarker configuration", e); + } + } + + /** + * Executes template with parameters and replaces the .ftl in the file name. + */ + public File replaceTemplate( + File templateFile, + Map parameters) throws IOException, freemarker.template.TemplateException { + File targetFile = new File(templateFile.toString().replace(".ftl", "")); + String rendered = template(templateFile, parameters); + + // Only write file if template has non-empty output. + // This avoids creating empty files when the entire template is skipped via <#if>. + if (rendered != null && !rendered.trim().isEmpty()) { + Files.writeString(targetFile.toPath(), rendered); + } else { + Files.deleteIfExists(targetFile.toPath()); + } + + Files.deleteIfExists(templateFile.toPath()); + return targetFile; + } + + /** + * Recursively templates all .ftl files in path. + * + *

That is, apply {@link #replaceTemplate(java.io.File, java.util.Map)} to all files matching + * filepathMatches. + */ + public void replaceTemplates( + File path, + Map parameters) throws IOException, freemarker.template.TemplateException { + replaceTemplates(path, parameters, FTL_FILE_PATTERN); + } + + public void replaceTemplates( + File path, + Map parameters, + Pattern filepathMatches) throws IOException, freemarker.template.TemplateException { + try (Stream stream = Files.walk(path.toPath())) { + List files = stream.filter(candidatePath -> filepathMatches.matcher(candidatePath.toString()).find()) + .toList(); + for (Path file : files) { + replaceTemplate(file.toFile(), parameters); + } + } + } + + public static Map templateToMap(String filePath, Map parameters) { + String hydratedString; + try { + hydratedString = new TemplatingEngine().template(new File(filePath), parameters); + } catch (Exception e) { + throw new RuntimeException("Failed to hydrate template to map: " + filePath, e); + } + + if (hydratedString == null || hydratedString.trim().isEmpty()) { + return Collections.emptyMap(); + } + return YamlUtils.parseYamlMap(hydratedString); + } + + /** + * Executes template and writes to targetFile, keeping the template file. + */ + public File template( + File templateFile, + File targetFile, + Map parameters) throws IOException, freemarker.template.TemplateException { + Template template = prepareTemplate(templateFile); + try (BufferedWriter writer = Files.newBufferedWriter(targetFile.toPath())) { + template.process(parameters, writer); + } + return targetFile; + } + + public String template( + File templateFile, + Map parameters) throws IOException, freemarker.template.TemplateException { + Template template = prepareTemplate(templateFile); + StringWriter writer = new StringWriter(); + template.process(parameters, writer); + return writer.toString(); + } + + public String template( + String template, + Map parameters) throws IOException, freemarker.template.TemplateException { + StringWriter writer = new StringWriter(); + Template templateObj = new Template("template", new StringReader(template), engine); + templateObj.process(parameters, writer); + return writer.toString(); + } + + protected Template prepareTemplate(File templateFile) throws IOException { + if (!templateFile.getName().contains(".ftl")) { + throw new IllegalArgumentException("File must contain .ftl to be a template"); + } + + engine.setDirectoryForTemplateLoading(templateFile.getParentFile()); + return engine.getTemplate(templateFile.getName()); + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/Tuple.java b/src/main/java/com/cloudogu/gitops/utils/Tuple.java new file mode 100644 index 000000000..1273783e3 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/Tuple.java @@ -0,0 +1,24 @@ +package com.cloudogu.gitops.utils; + +public record Tuple( + F first, + + S second +) { + + public F getFirst() { + return first; + } + + public S getSecond() { + return second; + } + + public F getV1() { + return first; + } + + public S getV2() { + return second; + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/YamlUtils.java b/src/main/java/com/cloudogu/gitops/utils/YamlUtils.java new file mode 100644 index 000000000..8fbab5cb6 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/YamlUtils.java @@ -0,0 +1,28 @@ +package com.cloudogu.gitops.utils; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.Map; + +public final class YamlUtils { + + private static final ObjectMapper YAML_MAPPER = new ObjectMapper(new YAMLFactory()); + + private YamlUtils() { + } + + public static Map parseYamlMap(String yaml) { + try { + Object parsedYaml = YAML_MAPPER.readValue(yaml, Object.class); + if (!(parsedYaml instanceof Map)) { + throw new IllegalArgumentException("Could not parse YAML as map: " + parsedYaml); + } + return MapUtils.asStringObjectMap(parsedYaml); + } catch (IOException exception) { + throw new UncheckedIOException("Failed to parse YAML", exception); + } + } +} diff --git a/src/main/java/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProvider.java b/src/main/java/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProvider.java new file mode 100644 index 000000000..8ce8ce252 --- /dev/null +++ b/src/main/java/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProvider.java @@ -0,0 +1,72 @@ +package com.cloudogu.gitops.utils.jgit.helpers; + +import org.eclipse.jgit.errors.UnsupportedCredentialItem; +import org.eclipse.jgit.transport.CredentialItem; +import org.eclipse.jgit.transport.CredentialsProvider; +import org.eclipse.jgit.transport.URIish; + +import java.util.Arrays; +import java.util.regex.Pattern; + +/** + * JGit, a project used within eclipse, is developed with an interactive UI in mind. The + * documentation for the CredentialsProvider says > CredentialItems are usually presented in bulk, + * allowing implementors to combine them into a single UI widget and streamline the authentication + * process for an end-user. This highlights the focus on the UI for an end-user. + * + *

As a result, checking for SSL verification is a little clunky as we need to check for messages + * intended for end-users. + * + *

Other options would have included overwriting the HttpConnection or saving the git + * configuration on disk. + * + * @link https://archive.eclipse.org/jgit/site/4.10.0.201712302008-r/apidocs/org/eclipse/jgit/transport/CredentialsProvider.html + */ +public class InsecureCredentialProvider extends CredentialsProvider { + private static final Pattern INSECURE_CONNECTION_PATTERN = Pattern.compile( + "^A secure connection to .* could not be established"); + private static final Pattern SKIP_SSL_PATTERN = Pattern.compile( + "^Skip SSL verification for git operations for repository"); + + @Override + public boolean isInteractive() { + return false; + } + + @Override + public boolean supports(CredentialItem... items) { + if (items == null) { + return false; + } + return Arrays.stream(items) + .filter(item -> item instanceof CredentialItem.InformationalMessage) + .map(item -> (CredentialItem.InformationalMessage) item) + .anyMatch(message -> INSECURE_CONNECTION_PATTERN.matcher(message.getPromptText()).find()); + } + + // JGit's CredentialsProvider contract: true means "these items were handled", regardless of + // which prompt was matched, so both return paths are intentionally the same value. + @Override + @SuppressWarnings("java:S3516") + public boolean get(URIish uri, CredentialItem... items) throws UnsupportedCredentialItem { + if (items == null) { + return true; + } + for (CredentialItem item : items) { + if (item instanceof CredentialItem.YesNoType yesNo) { + String prompt = yesNo.getPromptText(); + if ("Skip SSL verification for this single git operation".equals(prompt) || SKIP_SSL_PATTERN.matcher( + prompt) + .find()) { + yesNo.setValue(true); + } else if ("Always skip SSL verification for this server from now on".equals(prompt)) { + // otherwise we would persistently overwrite our $HOME/.gitconfig + yesNo.setValue(false); + } else { + // unrecognized prompt; leave the default value untouched + } + } + } + return true; + } +} diff --git a/src/main/resources/application-content-examples.yaml b/src/main/resources/application-content-examples.yaml index 0837c9b72..9367f959c 100644 --- a/src/main/resources/application-content-examples.yaml +++ b/src/main/resources/application-content-examples.yaml @@ -47,12 +47,12 @@ content: - ${config.application.namePrefix}example-apps-staging variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "alpine/kubectl:latest" helm: "ghcr.io/cloudogu/helm:latest" kubeval: "ghcr.io/cloudogu/helm:latest" helmKubeval: "ghcr.io/cloudogu/helm:latest" yamllint: "cytopia/yamllint:1.25-0.7" - petclinic: "eclipse-temurin:17-jre-alpine" + petclinic: "eclipse-temurin:17-jre" maven: "" diff --git a/src/main/resources/application-full-prefix.yaml b/src/main/resources/application-full-prefix.yaml index 8e87144fe..7addb4d52 100644 --- a/src/main/resources/application-full-prefix.yaml +++ b/src/main/resources/application-full-prefix.yaml @@ -55,12 +55,12 @@ content: - ${config.application.namePrefix}example-apps-staging variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "alpine/kubectl:latest" helm: "ghcr.io/cloudogu/helm:latest" kubeval: "ghcr.io/cloudogu/helm:latest" helmKubeval: "ghcr.io/cloudogu/helm:latest" yamllint: "cytopia/yamllint:1.25-0.7" - petclinic: "eclipse-temurin:17-jre-alpine" + petclinic: "eclipse-temurin:17-jre" maven: "" diff --git a/src/main/resources/application-full-secrets.yaml b/src/main/resources/application-full-secrets.yaml new file mode 100644 index 000000000..032d6b8fd --- /dev/null +++ b/src/main/resources/application-full-secrets.yaml @@ -0,0 +1,81 @@ +# $schema: https://raw.githubusercontent.com/cloudogu/gitops-playground/main/docs/configuration.schema.json +# Keep this profile in sync with application-full.yaml. It only adds settings required to exercise Secret-based credentials. +application: + "yes": true + baseUrl: http://localhost + credentials: + secretName: argocd-credentials + secretNamespace: gop-job +scm: + scmManager: + credentials: + secretName: scm-tenant-credentials + secretNamespace: gop-job +features: + certManager: + active: true + argocd: + active: true + operator: false + ingress: + active: true + monitoring: + active: true + secrets: + vault: + mode: "dev" +jenkins: + active: true + credentials: + secretName: jenkins-credentials + secretNamespace: gop-job +registry: + active: true + createImagePullSecrets: true + credentials: + secretName: registry-credentials + secretNamespace: gop-job +content: + repos: + - url: https://github.com/cloudogu/gitops-build-lib + target: 3rd-party-dependencies/gitops-build-lib + overwriteMode: RESET + - url: https://github.com/cloudogu/ces-build-lib + target: 3rd-party-dependencies/ces-build-lib + overwriteMode: RESET + - url: https://github.com/cloudogu/spring-boot-helm-chart + target: 3rd-party-dependencies/spring-boot-helm-chart + overwriteMode: RESET + - url: https://github.com/cloudogu/spring-petclinic + target: argocd/petclinic-plain + ref: feature/gitops_ready + targetRef: main + overwriteMode: UPGRADE + createJenkinsJob: true + - url: https://github.com/cloudogu/spring-petclinic + target: argocd/petclinic-helm + ref: feature/gitops_ready + targetRef: main + overwriteMode: UPGRADE + createJenkinsJob: true + - url: https://github.com/cloudogu/gitops-examples + path: example-apps-via-content-loader/ + ref: main + templating: true + type: FOLDER_BASED + overwriteMode: UPGRADE + + namespaces: + - ${config.application.namePrefix}example-apps-production + - ${config.application.namePrefix}example-apps-staging + variables: + petclinic: + baseDomain: "petclinic" + images: + kubectl: "alpine/kubectl:latest" + helm: "ghcr.io/cloudogu/helm:latest" + kubeval: "ghcr.io/cloudogu/helm:latest" + helmKubeval: "ghcr.io/cloudogu/helm:latest" + yamllint: "cytopia/yamllint:1.25-0.7" + petclinic: "eclipse-temurin:17-jre" + maven: "" diff --git a/src/main/resources/application-full.yaml b/src/main/resources/application-full.yaml index 42bdfa30a..215e57876 100644 --- a/src/main/resources/application-full.yaml +++ b/src/main/resources/application-full.yaml @@ -54,12 +54,12 @@ content: - ${config.application.namePrefix}example-apps-staging variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "alpine/kubectl:latest" helm: "ghcr.io/cloudogu/helm:latest" kubeval: "ghcr.io/cloudogu/helm:latest" helmKubeval: "ghcr.io/cloudogu/helm:latest" yamllint: "cytopia/yamllint:1.25-0.7" - petclinic: "eclipse-temurin:17-jre-alpine" + petclinic: "eclipse-temurin:17-jre" maven: "" diff --git a/src/main/resources/application-keycloak.yaml b/src/main/resources/application-keycloak.yaml new file mode 100644 index 000000000..39f07ddaf --- /dev/null +++ b/src/main/resources/application-keycloak.yaml @@ -0,0 +1,96 @@ +# $schema: https://raw.githubusercontent.com/cloudogu/gitops-playground/main/docs/configuration.schema.json +application: + "yes": true + baseUrl: http://localhost + password: "admin" +features: + certManager: + active: true + argocd: + active: true + operator: false + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: argocd + clientSecret: Cq3U2Dvx6WR5ep2ZLH8rZXNKFNcIX5Zx + adminGroupName: gop-admins + ingress: + active: true + monitoring: + active: true + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: grafana + clientSecret: 46uALR2HeqLPtJxzkkn6tK6FVC9vMmRc + adminGroupName: gop-admins + secrets: + vault: + mode: "dev" + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: vault + clientSecret: XySg7UyAzVkcaU4Visqfe9EChDvARYA1 + adminGroupName: gop-admins +jenkins: + active: true + password: "admin" + metricsUsername: "admin" + metricsPassword: "admin" + oidc: + providerName: Keycloak + issuerUrl: http://keycloak.local.gd/realms/gop + clientId: jenkins + clientSecret: "mtuIbUdggI2ZSy7jSsQW0WkwK0CwMbvO" + adminGroupName: gop-admins +registry: + active: true + password: "admin" +scm: + scmManager: + password: "admin" +content: + repos: + - url: https://github.com/cloudogu/gitops-build-lib + target: 3rd-party-dependencies/gitops-build-lib + overwriteMode: RESET + - url: https://github.com/cloudogu/ces-build-lib + target: 3rd-party-dependencies/ces-build-lib + overwriteMode: RESET + - url: https://github.com/cloudogu/spring-boot-helm-chart + target: 3rd-party-dependencies/spring-boot-helm-chart + overwriteMode: RESET + - url: https://github.com/cloudogu/spring-petclinic + target: argocd/petclinic-plain + ref: feature/gitops_ready + targetRef: main + overwriteMode: UPGRADE + createJenkinsJob: true + - url: https://github.com/cloudogu/spring-petclinic + target: argocd/petclinic-helm + ref: feature/gitops_ready + targetRef: main + overwriteMode: UPGRADE + createJenkinsJob: true + - url: https://github.com/cloudogu/gitops-examples + path: example-apps-via-content-loader/ + ref: main + templating: true + type: FOLDER_BASED + overwriteMode: UPGRADE + namespaces: + - ${config.application.namePrefix}example-apps-production + - ${config.application.namePrefix}example-apps-staging + variables: + petclinic: + baseDomain: "petclinic" + images: + kubectl: "alpine/kubectl:latest" + helm: "ghcr.io/cloudogu/helm:latest" + kubeval: "ghcr.io/cloudogu/helm:latest" + helmKubeval: "ghcr.io/cloudogu/helm:latest" + yamllint: "cytopia/yamllint:1.25-0.7" + petclinic: "eclipse-temurin:17-jre" + maven: "" diff --git a/src/main/resources/application-operator-content-examples.yaml b/src/main/resources/application-operator-content-examples.yaml index 42dc4b91c..7e3e5aef1 100644 --- a/src/main/resources/application-operator-content-examples.yaml +++ b/src/main/resources/application-operator-content-examples.yaml @@ -47,12 +47,12 @@ content: - ${config.application.namePrefix}example-apps-staging variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "alpine/kubectl:latest" helm: "ghcr.io/cloudogu/helm:latest" kubeval: "ghcr.io/cloudogu/helm:latest" helmKubeval: "ghcr.io/cloudogu/helm:latest" yamllint: "cytopia/yamllint:1.25-0.7" - petclinic: "eclipse-temurin:17-jre-alpine" + petclinic: "eclipse-temurin:17-jre" maven: "" diff --git a/src/main/resources/application-operator-full.yaml b/src/main/resources/application-operator-full.yaml index 52010f75f..1c867e899 100644 --- a/src/main/resources/application-operator-full.yaml +++ b/src/main/resources/application-operator-full.yaml @@ -56,12 +56,12 @@ content: - ${config.application.namePrefix}example-apps-staging variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "alpine/kubectl:latest" helm: "ghcr.io/cloudogu/helm:latest" kubeval: "ghcr.io/cloudogu/helm:latest" helmKubeval: "ghcr.io/cloudogu/helm:latest" yamllint: "cytopia/yamllint:1.25-0.7" - petclinic: "eclipse-temurin:17-jre-alpine" + petclinic: "eclipse-temurin:17-jre" maven: "" diff --git a/src/main/resources/application-single-namespace-example.yaml b/src/main/resources/application-single-namespace-example.yaml index 9ef7ba016..adf9741f7 100644 --- a/src/main/resources/application-single-namespace-example.yaml +++ b/src/main/resources/application-single-namespace-example.yaml @@ -48,7 +48,7 @@ content: variables: petclinic: - baseDomain: "petclinic.localhost" + baseDomain: "petclinic" images: kubectl: "alpine/kubectl:latest" helm: "ghcr.io/cloudogu/helm:latest" diff --git a/src/main/resources/logback.xml b/src/main/resources/logback.xml index a4bfb45f9..6c9f5ed85 100644 --- a/src/main/resources/logback.xml +++ b/src/main/resources/logback.xml @@ -1,16 +1,16 @@ - true - %cyan(%d{HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}) - %msg%n + %cyan(%d{HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}) - %msg%n + - + diff --git a/src/main/version/version-name.txt b/src/main/version/version-name.txt new file mode 100644 index 000000000..117d4ef2a --- /dev/null +++ b/src/main/version/version-name.txt @@ -0,0 +1 @@ +${versionName} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/application/ApplicationTest.groovy b/src/test/groovy/com/cloudogu/gitops/application/ApplicationTest.groovy deleted file mode 100644 index 3efdc6da2..000000000 --- a/src/test/groovy/com/cloudogu/gitops/application/ApplicationTest.groovy +++ /dev/null @@ -1,120 +0,0 @@ -package com.cloudogu.gitops.application - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.ScmTenantSchema -import io.micronaut.context.ApplicationContext -import org.junit.jupiter.api.Test - -import static org.assertj.core.api.Assertions.assertThat - -class ApplicationTest { - - private Config config = new Config() - - @Test - void 'feature\'s ordering is correct'() { - def application = ApplicationContext.run() - .registerSingleton(config) - .getBean(Application) - def features = application.features.collect { it.class.simpleName } - - assertThat(features).isEqualTo(['Registry', 'GitHandler', 'Jenkins', 'ArgoCD', 'Ingress', 'CertManager', 'Monitoring', 'ExternalSecretsOperator', 'Vault', 'ContentLoader']) - } - - @Test - void 'get active namespaces correctly'() { - config.registry.active = true - config.jenkins.active = true - config.features.monitoring.active = true - config.features.argocd.active = true - config.features.ingress.active = true - config.application.namePrefix = 'test1-' - config.content.namespaces = ['${config.application.namePrefix}example-apps-staging', - '${config.application.namePrefix}example-apps-production'] - List namespaceList = new ArrayList<>(Arrays.asList("test1-argocd", - "test1-example-apps-staging", - "test1-example-apps-production", - "test1-" + config.features.ingress.ingressNamespace, - "test1-monitoring", - "test1-registry", - "test1-jenkins")) - def application = ApplicationContext.run() - .registerSingleton(config) - .getBean(Application) - application.setNamespaceListToConfig(config) - assertThat(config.application.namespaces.getActiveNamespaces()).containsExactlyInAnyOrderElementsOf(namespaceList) - } - - @Test - void 'get active namespaces correctly in Openshift'() { - config.registry.active = true - config.jenkins.active = true - config.features.monitoring.active = true - config.features.argocd.active = true - config.features.ingress.active = true - config.application.namePrefix = 'test1-' - config.application.openshift = true - config.content.namespaces = ['${config.application.namePrefix}example-apps-staging', - '${config.application.namePrefix}example-apps-production'] - List namespaceList = new ArrayList<>(Arrays.asList("test1-argocd", - "test1-example-apps-staging", - "test1-example-apps-production", - "test1-" + config.features.ingress.ingressNamespace, - "test1-monitoring", - "test1-registry", - "test1-jenkins")) - def application = ApplicationContext.run() - .registerSingleton(config) - .getBean(Application) - application.setNamespaceListToConfig(config) - assertThat(config.application.namespaces.getActiveNamespaces()).containsExactlyInAnyOrderElementsOf(namespaceList) - } - - @Test - void 'handles content namespaces without template'() { - config.content.namespaces = ['example-apps-staging', - 'example-apps-production'] - def application = ApplicationContext.run() - .registerSingleton(config) - .getBean(Application) - application.setNamespaceListToConfig(config) - assertThat(config.application.namespaces.getActiveNamespaces()).containsAll(["example-apps-staging", - "example-apps-production",]) - } - - @Test - void 'handles empty content namespaces'() { - def application = ApplicationContext.run() - .registerSingleton(config) - .getBean(Application) - application.setNamespaceListToConfig(config) - // No exception == happy - } - - @Test - void 'get active namespaces correctly in Openshift if jenkins and scm are external'() { - config.registry.active = true - config.jenkins.active = true - config.jenkins.internal = false - config.scm.scmManager = new ScmTenantSchema.ScmManagerTenantConfig() - config.scm.scmManager.internal = false - config.features.monitoring.active = true - config.features.argocd.active = true - config.features.ingress.active = true - config.application.namePrefix = 'test1-' - config.application.openshift = true - config.content.namespaces = ['${config.application.namePrefix}example-apps-staging', - '${config.application.namePrefix}example-apps-production'] - List namespaceList = new ArrayList<>(Arrays.asList("test1-argocd", - "test1-example-apps-staging", - "test1-example-apps-production", - "test1-" + config.features.ingress.ingressNamespace, - "test1-monitoring", - "test1-registry",)) - def application = ApplicationContext.run() - .registerSingleton(config) - .getBean(Application) - application.setNamespaceListToConfig(config) - assertThat(config.application.namespaces.getActiveNamespaces()).containsExactlyInAnyOrderElementsOf(namespaceList) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/application/content/ContentLoaderTest.groovy b/src/test/groovy/com/cloudogu/gitops/application/content/ContentLoaderTest.groovy deleted file mode 100644 index bb4b68bde..000000000 --- a/src/test/groovy/com/cloudogu/gitops/application/content/ContentLoaderTest.groovy +++ /dev/null @@ -1,1061 +0,0 @@ -package com.cloudogu.gitops.application.content - -import static com.cloudogu.gitops.application.content.ContentLoader.RepoCoordinate -import static com.cloudogu.gitops.config.Config.ContentRepoType -import static com.cloudogu.gitops.config.Config.ContentSchema.ContentRepositorySchema -import static com.cloudogu.gitops.config.Config.OverwriteMode -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.ArgumentMatchers.eq -import static org.mockito.Mockito.* - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.config.scm.ScmTenantSchema -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.testhelper.git.TestScmManagerApiClient -import com.cloudogu.gitops.tools.core.Jenkins -import com.cloudogu.gitops.utils.FileSystemUtils - -import java.nio.file.Files -import java.nio.file.Path -import groovy.util.logging.Slf4j -import groovy.yaml.YamlSlurper - -import io.fabric8.kubernetes.api.model.Secret -import io.fabric8.kubernetes.api.model.SecretBuilder -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import org.apache.commons.io.FileUtils -import org.eclipse.jgit.api.CloneCommand -import org.eclipse.jgit.api.Git -import org.eclipse.jgit.lib.Ref -import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider -import org.eclipse.jgit.util.SystemReader -import org.junit.jupiter.api.AfterAll -import org.junit.jupiter.api.Disabled -import org.junit.jupiter.api.DisplayName -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.io.TempDir -import org.mockito.ArgumentCaptor - -@Slf4j -@EnableKubernetesMockClient(crud = true) -class ContentLoaderTest { - - static List foldersToDelete = new ArrayList() - - Config config = new Config(application: new Config.ApplicationSchema(namePrefix: 'foo-'), - scm: new ScmTenantSchema(scmManager: new ScmTenantSchema.ScmManagerTenantConfig(url: '')), - registry: new Config.RegistrySchema(url: 'reg-url', - path: 'reg-path', - username: 'reg-user', - password: 'reg-pw', - createImagePullSecrets: false)) - - KubernetesClient client - K8sClient k8sClient = new K8sClient() - TestGitRepoFactory scmmRepoProvider = new TestGitRepoFactory(config, new FileSystemUtils()) - TestScmManagerApiClient scmmApiClient = new TestScmManagerApiClient(config) - Jenkins jenkins = mock(Jenkins.class) - ScmManagerMock scmManagerMock = new ScmManagerMock() - GitHandler gitHandler = new GitHandlerForTests(config, scmManagerMock) - DeploymentStrategy deploymentStrategy = mock(DeploymentStrategy) - FileSystemUtils fileSystemUtils = new FileSystemUtils() - - @TempDir - File tmpDir - - List expectedTargetRepos = [new RepoCoordinate(namespace: "common", repoName: "repo"), - new RepoCoordinate(namespace: "ns1a", repoName: "repo1a1"), - new RepoCoordinate(namespace: "ns1a", repoName: "repo1a2"), - new RepoCoordinate(namespace: "ns1b", repoName: "repo1b1"), - new RepoCoordinate(namespace: "ns1b", repoName: "repo1b2"), - new RepoCoordinate(namespace: "ns2a", repoName: "repo2a1"), - new RepoCoordinate(namespace: "ns2a", repoName: "repo2a2"), - new RepoCoordinate(namespace: "ns2b", repoName: "repo2b1"), - new RepoCoordinate(namespace: "ns2b", repoName: "repo2b2"), - new RepoCoordinate(namespace: "copy", repoName: "repo1"), - new RepoCoordinate(namespace: "copy", repoName: "repo2"),] - - List contentRepos = [// copy-typed repo writing to their own target - new ContentRepositorySchema(url: createContentRepo('copyRepo1'), type: ContentRepoType.COPY, target: 'copy/repo1'), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'copy/repo2', path: 'subPath'), - - // Same folder as in copyRepos -> Should be combined - new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo'), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', path: 'subPath'), - - // Contains ftl - new ContentRepositorySchema(url: createContentRepo('folderBasedRepo1'), type: ContentRepoType.FOLDER_BASED, templating: true), - // Contains a templated file that should be ignored - new ContentRepositorySchema(url: createContentRepo('folderBasedRepo2'), type: ContentRepoType.FOLDER_BASED, path: 'subPath'), - - ] - - @AfterAll - static void cleanFolders() { - foldersToDelete.each { it.deleteDir() } - - } - - @Disabled("TODO: Does not run on Jenkins: Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name or service not known") - @Test - void 'deploys image pull secrets'() { - config.registry.createImagePullSecrets = true - config.content.namespaces = ['example-apps-staging', 'example-apps-production'] - - createContent(config).install() - - assertRegistrySecrets('reg-user', 'reg-pw') - } - - @Disabled("TODO: Does not run on Jenkins: Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name or service not known") - @Test - void 'deploys image pull secrets from read-only vars'() { - config.registry.createImagePullSecrets = true - config.content.namespaces = ['example-apps-staging', 'example-apps-production'] - config.registry.readOnlyUsername = 'other-user' - config.registry.readOnlyPassword = 'other-pw' - - createContent(config).install() - - assertRegistrySecrets('other-user', 'other-pw') - } - - @Disabled("TODO: Does not run on Jenkins: Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name or service not known") - @Test - void 'deploys additional image pull secrets for proxy registry'() { - config.registry.createImagePullSecrets = true - config.content.namespaces = ['example-apps-staging', 'example-apps-production'] - config.registry.twoRegistries = true - config.registry.proxyUrl = 'proxy-url' - config.registry.proxyUsername = 'proxy-user' - config.registry.proxyPassword = 'proxy-pw' - - createContent(config).install() - - assertRegistrySecrets('reg-user', 'reg-pw') - } - - @Test - void 'Combines content repos successfully'() { - - config.content.repos = contentRepos - - def repos = createContent(config).cloneContentRepos() - - expectedTargetRepos.each { expected -> assertThat(new File(findRoot(repos), "${expected.namespace}/${expected.repoName}/file")).exists().isFile() - } - - assertThat(new File(findRoot(repos), "common/repo/file").text).contains("folderBasedRepo2") // Last repo "wins" - - assertThat(new File(findRoot(repos), "common/repo/folderBasedRepo1")).exists().isFile() - assertThat(new File(findRoot(repos), "common/repo/folderBasedRepo2")).exists().isFile() - assertThat(new File(findRoot(repos), "common/repo/copyRepo1")).exists().isFile() - assertThat(new File(findRoot(repos), "common/repo/copyRepo2")).exists().isFile() - - // Assert Templating - assertThat(new File(findRoot(repos), "common/repo/some.yaml")).exists() - assertThat(new File(findRoot(repos), "common/repo/some.yaml").text).contains("namePrefix: foo-") - // Assert not templating for this folder-based repo - assertThat(new File(findRoot(repos), "common/repo/someOther.yaml.ftl")).exists() - assertThat(new File(findRoot(repos), "common/repo/someOther.yaml.ftl").text).contains('namePrefix: ${config.application.namePrefix}') - } - - @Test - void 'supports content variables'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('folderBasedRepo1'), type: ContentRepoType.FOLDER_BASED, templating: true)] - config.content.variables.someapp = [somevalue: 'this is a custom variable'] - - def repos = createContent(config).cloneContentRepos() - - // Assert Templating - assertThat(new File(findRoot(repos), "common/repo/some.yaml")).exists() - assertThat(new File(findRoot(repos), "common/repo/some.yaml").text).contains("namePrefix: foo-") - assertThat(new File(findRoot(repos), "common/repo/some.yaml").text).contains("myvar: this is a custom variable") - } - - @Test - void 'Authenticates content Repos'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo', credentials: new Credentials('user', 'pw'))] - - def content = createContent(config) - content.cloneContentRepos() - - ArgumentCaptor captor = ArgumentCaptor.forClass(UsernamePasswordCredentialsProvider) - verify(content.cloneSpy).setCredentialsProvider(captor.capture()) - - def value = captor.value - assertThat(value.properties.username).isEqualTo('user') - assertThat(value.properties.password).isEqualTo('pw'.toCharArray()) - } - - @Test - @DisplayName("Authenticates content Repos with secret") - void authenticatesContentReposWithSecret() { - this.k8sClient.client = client - Secret secret = new SecretBuilder() - .withNewMetadata() - .withName("secret-test-name") - .withNamespace("default") - .endMetadata() - .withType("Opaque") - .withData(Map.of("username", "YWRtaW4=", - "password", "czNjcjN0")) - .build() - - this.k8sClient.client.secrets() - .inNamespace("default") - .resource(secret) - .create() - - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), - ref: 'main', type: ContentRepoType.COPY, - target: 'common/repo', - credentials: new Credentials(null, null, 'secret-test-name', 'default'))] - - def content = createContent(config) - content.cloneContentRepos() - - ArgumentCaptor captor = ArgumentCaptor.forClass(UsernamePasswordCredentialsProvider) - verify(content.cloneSpy).setCredentialsProvider(captor.capture()) - def value = captor.value - assertThat(value.properties.username).isEqualTo('admin') - assertThat(value.properties.password).isEqualTo('s3cr3t'.toCharArray()) - } - - @Test - void 'Checks out commit refs, tags and non-default branches for content repos'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), ref: 'someTag', type: ContentRepoType.COPY, target: 'common/tag'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), ref: '8bc1d1165468359b16d9771d4a9a3df26afc03e8', type: ContentRepoType.COPY, target: 'common/ref'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), ref: 'someBranch', type: ContentRepoType.COPY, target: 'common/branch')] - - def repos = createContent(config).cloneContentRepos() - - assertThat(new File(findRoot(repos), "common/tag/README.md")).exists().isFile() - assertThat(new File(findRoot(repos), "common/tag/README.md").text).contains("someTag") - - assertThat(new File(findRoot(repos), "common/ref/README.md")).exists().isFile() - assertThat(new File(findRoot(repos), "common/ref/README.md").text).contains("main") - - assertThat(new File(findRoot(repos), "common/branch/README.md")).exists().isFile() - assertThat(new File(findRoot(repos), "common/branch/README.md").text).contains("someBranch") - } - - @Test - void 'Checks out default branch when no ref set'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('', 'git-repo-different-default-branch'), target: 'common/default', type: ContentRepoType.COPY),] - - def repos = createContent(config).cloneContentRepos() - - assertThat(new File(findRoot(repos), "common/default/README.md")).exists().isFile() - assertThat(new File(findRoot(repos), "common/default/README.md").text).contains("different") - } - - @Test - void 'Fails if commit ref does not exist'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), ref: 'someTag', type: ContentRepoType.COPY, target: 'common/tag'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), ref: 'does/not/exist', type: ContentRepoType.FOLDER_BASED, target: 'does not matter'),] - - def exception = shouldFail(RuntimeException) { - createContent(config).cloneContentRepos() - } - - assertThat(exception.message).startsWith("Reference 'does/not/exist' not found in content repository") - } - - @Test - void 'Respects order of folder-based repositories'() { - config.content.repos = [// Note the different order! - new ContentRepositorySchema(url: createContentRepo('folderBasedRepo1'), ref: 'main', type: ContentRepoType.FOLDER_BASED), - new ContentRepositorySchema(url: createContentRepo('folderBasedRepo2'), ref: 'main', type: ContentRepoType.FOLDER_BASED, path: 'subPath'), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo', path: 'subPath'), - new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo'),] - - def repos = createContent(config).cloneContentRepos() - - assertThat(new File(findRoot(repos), "common/repo/file").text).contains("copyRepo1") - // Last repo "wins" - } - - @Test - void 'Is able to COPY into MIRRORED repo'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('mirrorRepo1', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'common/repo'), - new ContentRepositorySchema(url: createContentRepo('folderBasedRepo1'), type: ContentRepoType.FOLDER_BASED, overwriteMode: OverwriteMode.UPGRADE), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', overwriteMode: OverwriteMode.UPGRADE, path: 'subPath')] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - - def expectedRepo = 'common/repo' - // clone target repo, to ensure, changes in remote repo. - try (def git = cloneRepo(expectedRepo, tmpDir)) { - assertThat(new File(tmpDir, "file").text).contains("copyRepo2") // Last repo "wins" - assertThat(new File(tmpDir, "mirrorRepo1")).exists().isFile() - assertThat(new File(tmpDir, "copyRepo2")).exists().isFile() - assertThat(new File(tmpDir, "folderBasedRepo1")).exists().isFile() - - // Assert mirrors branches and tags of non-folderBased repos - // Verify tag exists and points to correct content - git.fetch().setRefSpecs("refs/*:refs/*").call() // Fetch all tags and branches - - assertTag(git, 'someTag') - assertBranch(git, 'someBranch') - } - } - - @Test - void 'Handles mirror and copy together'() { - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('folderBasedRepo1'), type: ContentRepoType.FOLDER_BASED), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', overwriteMode: OverwriteMode.UPGRADE, path: 'subPath'), - new ContentRepositorySchema(url: createContentRepo('mirrorRepo1', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, overwriteMode: OverwriteMode.RESET, target: 'common/repo'),] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - - def expectedRepo = 'common/repo' - // clone target repo, to ensure, changes in remote repo. - try (def git = cloneRepo(expectedRepo, tmpDir)) { - assertThat(new File(tmpDir, "file").text).contains("mirrorRepo1") // Last repo "wins" - assertThat(new File(tmpDir, "folderBasedRepo1")).doesNotExist() - assertThat(new File(tmpDir, "copyRepo2")).doesNotExist() - - // Assert mirrors branches and tags of non-folderBased repos - // Verify tag exists and points to correct content - git.fetch().setRefSpecs("refs/*:refs/*").call() // Fetch all tags and branches - - assertTag(git, 'someTag') - assertBranch(git, 'someBranch') - } - } - - @Test - void 'Handles multiple mirrors of the same repo with different refs'() { - def repoToMirror = createContentRepo('mirrorRepo1', 'git-repository-with-branches-tags') - config.content.repos = [new ContentRepositorySchema(url: repoToMirror, type: ContentRepoType.MIRROR, ref: 'main', target: 'common/repo'), - new ContentRepositorySchema(url: repoToMirror, type: ContentRepoType.MIRROR, ref: 'someBranch', target: 'common/repo', overwriteMode: OverwriteMode.UPGRADE), - new ContentRepositorySchema(url: repoToMirror, type: ContentRepoType.MIRROR, ref: 'someTag', target: 'common/repo', overwriteMode: OverwriteMode.UPGRADE), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', overwriteMode: OverwriteMode.UPGRADE, path: 'subPath')] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - - def expectedRepo = 'common/repo' - // clone target repo, to ensure, changes in remote repo. - try (def git = cloneRepo(expectedRepo, tmpDir)) { - assertThat(new File(tmpDir, "file").text).contains("copyRepo2") // Last repo "wins" - assertThat(new File(tmpDir, "mirrorRepo1")).exists().isFile() - - git.fetch().setRefSpecs("refs/*:refs/*").call() // Fetch all tags and branches - - assertTag(git, 'someTag') - assertBranch(git, 'someBranch') - } - } - - @Test - void 'Handles targetRefs'() { - config.content.repos = [// From branch to branch or tag to tag - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'mirror/tag', ref: 'someTag', targetRef: 'my-tag'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'mirror/branch', ref: 'someBranch', targetRef: 'my-branch'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.COPY, target: 'copy/tag', ref: 'someTag', targetRef: 'my-tag'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.COPY, target: 'copy/branch', ref: 'someBranch', targetRef: 'my-branch'), - - // From tag to branch or the other way round - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'mirror/tag2branch', ref: 'someTag', targetRef: 'refs/heads/my-branch'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'mirror/branch2tag', ref: 'someBranch', targetRef: 'refs/tags/my-tag'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.COPY, target: 'copy/tag2branch', ref: 'someTag', targetRef: 'refs/heads/my-branch'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.COPY, target: 'copy/branch2tag', ref: 'someBranch', targetRef: 'refs/tags/my-tag'),] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - - // From branch to branch or tag to tag - assertTagAndReadme('mirror/tag', 'my-tag', "someTag") - assertBranchAndReadme('mirror/branch', 'my-branch', "someBranch") - - assertTagAndReadme('copy/tag', 'my-tag', "someTag") - assertBranchAndReadme('copy/branch', 'my-branch', "someBranch") - - // From tag to branch or the other way round - assertTagAndReadme('mirror/branch2tag', 'my-tag', "someBranch") - assertBranchAndReadme('mirror/tag2branch', 'my-branch', "someTag") - - assertTagAndReadme('copy/branch2tag', 'my-tag', "someBranch") - assertBranchAndReadme('copy/tag2branch', 'my-branch', "someTag") - } - - @Test - void 'Handles multiple mirrors of the same repo with different refs, where one is not pushed'() { - // This test case does not make too much sense but used to cause git problems when we merged all content repos into a single folder, like - // TransportException: Missing unknown 5bcf50f0537bf4d2719a82e9b0950fbac92b3ecc - def repoToMirror = createContentRepo('copyRepo1', 'git-repository-with-branches-tags') - config.content.repos = [new ContentRepositorySchema(url: repoToMirror, type: ContentRepoType.MIRROR, ref: 'main', target: 'common/repo'), - new ContentRepositorySchema(url: repoToMirror, type: ContentRepoType.MIRROR, ref: 'someBranch', target: 'common/repo') /* Deliberately not use overwriteMode here !*/, - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', overwriteMode: OverwriteMode.UPGRADE, path: 'subPath')] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - // No exception means success - } - - @Test - void 'Is able to MIRROR into repo that has same commits'() { - // This test case does not make too much sense but used to cause git problems when copying .git from source to target - // java.lang.IllegalArgumentException: File parameter 'destFile is not writable: '/tmp/../.git/objects/pack/pack-524e3f54c7b28a98a4995948dfc8e75f1642840f.pack' - // This only occurs when the same .pack files exists in .git because they are read-only - // So for our testcase we just mirror the same repo twice - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('mirrorRepo1', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'common/repo'), - new ContentRepositorySchema(url: createContentRepo('mirrorRepo1', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'common/repo', overwriteMode: OverwriteMode.RESET),] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - // No exception means success - } - - @Test - void 'Parses Repo coordinates'() { - - config.content.repos = contentRepos - - def content = createContent(config) - - def actualTargetRepos = content.cloneContentRepos() - def repos = actualTargetRepos - - assertThat(actualTargetRepos).hasSameSizeAs(expectedTargetRepos) - - expectedTargetRepos.each { expected -> - - def actual = actualTargetRepos.findAll { actual -> actual.namespace == expected.namespace && actual.repoName == expected.repoName - } - assertThat(actual).withFailMessage("Could not find repo with namespace=${expected.namespace} and repo=${expected.repoName} in ${actualTargetRepos}").hasSize(1) - - assertThat(actual[0].clonedContentRepo.absolutePath).isEqualTo(new File(findRoot(repos), "${expected.namespace}/${expected.repoName}").absolutePath) - } - } - - @Test - void 'Creates and pushes content repos, whole flow '() { - config.content.repos = contentRepos + [new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, target: 'common/mirror'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, ref: 'main', target: 'common/mirrorWithBranchRef'), - new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, ref: 'someTag', target: 'common/mirrorWithTagRef'),] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - - def expectedRepo = 'copy/repo1' - // clone target repo, to ensure, changes in remote repo. - try (def git = cloneRepo(expectedRepo, tmpDir)) { - - def commitMsg = git.log().call().iterator().next().getFullMessage() - assertThat(commitMsg).isEqualTo("Initialize content repo ${expectedRepo}".toString()) - - assertThat(new File(tmpDir, "file").text).contains("copyRepo1") - assertThat(new File(tmpDir, "copyRepo1")).exists().isFile() - } - - expectedRepo = 'common/mirror' - try (def git = cloneRepo(expectedRepo, createRandomSubDir())) { - // Assert mirrors branches and tags of non-folderBased repos - // Verify tag exists and points to correct content - git.fetch().setRefSpecs("refs/*:refs/*").call() // Fetch all tags and branches - - assertTag(git, 'someTag') - assertBranch(git, 'someBranch') - } - - expectedRepo = 'common/mirrorWithBranchRef' - try (def git = cloneRepo(expectedRepo, createRandomSubDir())) { - - git.fetch().setRefSpecs("refs/*:refs/*").call() - - assertNoTags(git) - assertOnlyBranch(git, 'main') - } - - expectedRepo = 'common/mirrorWithTagRef' - try (def git = cloneRepo(expectedRepo, createRandomSubDir())) { - - git.fetch().setRefSpecs("refs/*:refs/*").call() - - assertTag(git, 'someTag') - assertOnlyBranch(git, 'main') - } - - // Mirroring commit references is not supported - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, ref: '8bc1d1165468359b16d9771d4a9a3df26afc03e8', target: 'common/mirrorWithCommitRef')] - - def exception = shouldFail(RuntimeException) { - createContent(config).install() - } - assertThat(exception.message).startsWith('Mirroring commit references is not supported for content repos at the moment. content repository') - assertThat(exception.message).endsWith('ref: 8bc1d1165468359b16d9771d4a9a3df26afc03e8') - - - // Mirroring short commit references is not supported as well - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('', 'git-repository-with-branches-tags'), type: ContentRepoType.MIRROR, ref: '8bc1d11', target: 'common/mirrorWithShortCommitRef')] - - exception = shouldFail(RuntimeException) { - createContent(config).install() - } - assertThat(exception.message).startsWith('Mirroring commit references is not supported for content repos at the moment. content repository') - assertThat(exception.message).endsWith('ref: 8bc1d11') - - // Don't bother validating all other repos here. - // If it works for the most complex one, the other ones will work as well. - // The other tests are already asserting correct combining (including order) and parsing of the repos. - } - - static void assertOnlyBranch(Git git, String branch) { - def branches = assertBranch(git, branch) - def otherBranches = branches.findAll { !it.name.contains(branch) } - assertThat(otherBranches) - .withFailMessage("More than the expected branch main found. Available branches: ${otherBranches.collect { it.name }}") - .hasSize(0) - } - - static void assertNoTags(Git git) { - def tags = git.tagList().call() - assertThat(tags) - .withFailMessage("No tags in mirrored repo with ref expected. Available tags: ${tags.collect { it.name }}") - .hasSize(0) - } - - static List assertBranch(Git git, String someBranch) { - def branches = git.branchList().call() - assertThat(branches.findAll { it.name == "refs/heads/${someBranch}" }) - .withFailMessage("Branch '${someBranch}' not found in git repository. Available branches: ${branches.collect { it.name }}") - .hasSize(1) - return branches - } - - static void assertTag(Git git, String expectedTag) { - def tags = git.tagList().call() - assertThat(tags.findAll { it.name == "refs/tags/$expectedTag" }) - .withFailMessage("Tag '$expectedTag' not found in git repository. Available tags: ${tags.collect { it.name }}") - .hasSize(1) - } - - @Test - void 'Reset common repo to repo '() { - /** - * Prepare Testcase - * using all defined repos -> common/repo is used by copyRepo1 + 2 - * file content after that: copyRepo2 - * - * Then again "RESET" to copyRepo1. - * file content after that should be: copyRepo1*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo'), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', path: 'subPath') - - ] - def expectedRepo = 'common/repo' - def repo = scmmRepoProvider.getRepo(expectedRepo, scmManagerMock) - scmManagerMock.initOnceRepo(repo.repoTarget) - createContent(config).install() - - String url = repo.getGitRepositoryUrl() - // clone repo, to ensure, changes in remote repo. - try (def git = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(tmpDir).call()) { - - verify(repo).createRepositoryAndSetPermission(any(String.class), eq(false)) - - def commitMsg = git.log().call().iterator().next().getFullMessage() - assertThat(commitMsg).isEqualTo("Initialize content repo ${expectedRepo}".toString()) - - assertThat(new File(tmpDir, "file").text).contains("copyRepo2") - assertThat(new File(tmpDir, "copyRepo2")).exists().isFile() - } - - /** - * End of preparation - * - * Now Reset to an copied repo*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo', overwriteMode: OverwriteMode.RESET),] - - createContent(config).install() - scmManagerMock.clearInitOnce() - - def folderAfterReset = File.createTempDir('second-cloned-repo') - folderAfterReset.deleteOnExit() - // clone repo, to ensure, changes in remote repo. - try (def git2 = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(folderAfterReset).call()) { - - assertThat(git2).isNotNull() - // because copyRepo1 is only part of repo1 - assertThat(new File(folderAfterReset, "file").text).contains("copyRepo1") - // should not exists, if RESET to first repo - assertThat(new File(folderAfterReset, "copyRepo2").exists()).isFalse() - - } - - } - - @Test - void 'Update common repo test '() { - /** - * Prepare Testcase - * using all defined repos -> common/repo is used by copyRepo1 + 2 - * file content after that: copyRepo2 - * - * Then again "RESET" to copyRepo1. - * file content after that should be: copyRepo1*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo'),] - - scmmApiClient.mockRepoApiBehaviour() - - createContent(config).install() - - def expectedRepo = 'common/repo' - def repo = scmmRepoProvider.getRepo(expectedRepo, new ScmManagerMock()) - - def url = repo.getGitRepositoryUrl() - // clone repo, to ensure, changes in remote repo. - try (def git = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(tmpDir).call()) { - - verify(repo).createRepositoryAndSetPermission(any(String.class), eq(false)) - - def commitMsg = git.log().call().iterator().next().getFullMessage() - assertThat(commitMsg).isEqualTo("Initialize content repo ${expectedRepo}".toString()) - - assertThat(new File(tmpDir, "file").text).contains("copyRepo1") - assertThat(new File(tmpDir, "copyRepo1")).exists().isFile() - - } - /** - * End of preparation - * - * Now Upgrade to type copy*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', path: 'subPath', overwriteMode: OverwriteMode.UPGRADE)] - - createContent(config).install() - - def folderAfterReset = File.createTempDir('second-cloned-repo') - folderAfterReset.deleteOnExit() - // clone repo, to ensure, changes in remote repo. - try (def git2 = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(folderAfterReset).call()) { - - assertThat(git2).isNotNull() - // because copyRepo1 is only part of repo1 - assertThat(new File(folderAfterReset, "file").text).contains("copyRepo2") - // should not exists, if RESET to first repo - assertThat(new File(folderAfterReset, "copyRepo2").exists()).isTrue() - - } - } - - @Test - void 'init common repo, expect unchanged repo'() { - /** - * Prepare Testcase - * using all defined repos -> common/repo is used by copyRepo1 + 2 - * file content after that: copyRepo2 - * - * Then again "RESET" to copyRepo1. - * file content after that should be: copyRepo1*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo'), - new ContentRepositorySchema(url: createContentRepo('copyRepo2'), type: ContentRepoType.COPY, target: 'common/repo', path: 'subPath') - - ] - def expectedRepo = 'common/repo' - def repo = scmmRepoProvider.getRepo(expectedRepo, scmManagerMock) - scmManagerMock.initOnceRepo(repo.repoTarget) - createContent(config).install() - - def url = repo.getGitRepositoryUrl() - // clone repo, to ensure, changes in remote repo. - try (def git = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(tmpDir).call()) { - - verify(repo).createRepositoryAndSetPermission(any(String.class), eq(false)) - - def commitMsg = git.log().call().iterator().next().getFullMessage() - assertThat(commitMsg).isEqualTo("Initialize content repo ${expectedRepo}".toString()) - - assertThat(new File(tmpDir, "file").text).contains("copyRepo2") - assertThat(new File(tmpDir, "copyRepo2")).exists().isFile() - } - - /** - * End of preparation - * - * Now INit to a copied repo - * no changes expected, file still has copyRepo2 and so on*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, target: 'common/repo', overwriteMode: OverwriteMode.INIT),] - - createContent(config).install() - scmManagerMock.clearInitOnce() - - def folderAfterReset = File.createTempDir('second-cloned-repo') - folderAfterReset.deleteOnExit() - // clone repo, to ensure, changes in remote repo. - try (def git = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(folderAfterReset).call()) { - - assertThat(git).isNotNull() - // because copyRepo1 is only part of repo1 - assertThat(new File(folderAfterReset, "file").text).contains("copyRepo2") - // should not exists, if RESET to first repo - assertThat(new File(folderAfterReset, "copyRepo2").exists()).isTrue() - - } - - } - - @Test - void 'ensure Jenkinsjob will be created'() { - /** - * Prepare Testcase - * using all defined repos -> common/repo is used by copyRepo1 + 2 - * file content after that: copyRepo2 - * - * Then again "RESET" to copyRepo1. - * file content after that should be: copyRepo1*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, createJenkinsJob: true, target: 'common/repo'),] - scmmApiClient.mockRepoApiBehaviour() - when(jenkins.isEnabled()).thenReturn(true) - - createContent(config).install() - verify(jenkins).createJenkinsjob(any(), any()) - } - - @Test - void 'ensure Jenkinsjob creation will be ignored'() { - /** - * Prepare Testcase - * using all defined repos -> common/repo is used by copyRepo1 + 2 - * file content after that: copyRepo2 - * - * Then again "RESET" to copyRepo1. - * file content after that should be: copyRepo1*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, createJenkinsJob: false, target: 'common/repo'),] - scmmApiClient.mockRepoApiBehaviour() - when(jenkins.isEnabled()).thenReturn(false) - createContent(config).install() - verify(jenkins, never()).createJenkinsjob(any(), any()) - } - - @Test - void 'ensure Jenkinsjob will not be created, if jenkins is not enables'() { - /** - * Prepare Testcase - * using all defined repos -> common/repo is used by copyRepo1 + 2 - * file content after that: copyRepo2 - * - * Then again "RESET" to copyRepo1. - * file content after that should be: copyRepo1*/ - config.content.repos = [new ContentRepositorySchema(url: createContentRepo('copyRepo1'), ref: 'main', type: ContentRepoType.COPY, createJenkinsJob: false, target: 'common/repo'),] - scmmApiClient.mockRepoApiBehaviour() - when(jenkins.isEnabled()).thenReturn(false) - - createContent(config).install() - verify(jenkins, never()).createJenkinsjob(any(), any()) - } - - @Test - void 'deployHelmReleasesFromContent skips when helmReleases missing or empty'() { - def contentLoader = createContent(config) - contentLoader.install() - - assertThat(contentLoader.deployCalls).isEmpty() - } - - @Test - void 'deployHelmReleasesFromContent calls deployHelmChart with valuesPath and helm config'() { - // Arrange: create a real values file on disk - Path valuesFile = Files.createTempFile("harbor-values-", ".yaml") - Files.writeString(valuesFile, """ - expose: - type: ingress - """.stripIndent()) - - def cfg = Config.fromMap(content: [helmReleases: [[name : 'harbor', - repoURL : 'https://helm.goharbor.io', - chart : 'harbor', - version : '1.18.2', - namespace : 'my-prefix-harbor', - releaseName: 'harbor', - valuesPath : valuesFile.toString()]]]) - - def contentLoader = createContent(cfg) - contentLoader.install() - - assertThat(contentLoader.deployCalls).hasSize(1) - def call = contentLoader.deployCalls[0] - - assertThat(call.featureName).isEqualTo('harbor') - assertThat(call.releaseName).isEqualTo('harbor') - assertThat(call.namespace).isEqualTo('my-prefix-harbor') - - // IMPORTANT: With the new implementation you likely pass a merged temp file, - // not the original valuesPath. So assert it's a file that exists. - assertThat(call.valuesPath).isNotBlank() - assertThat(Path.of(call.valuesPath).toFile()).exists() - - assertThat(call.helmConfig.repoURL).isEqualTo('https://helm.goharbor.io') - assertThat(call.helmConfig.chart).isEqualTo('harbor') - assertThat(call.helmConfig.version).isEqualTo('1.18.2') - assertThat(call.config).isSameAs(cfg) - } - - @Test - void 'deployHelmReleasesFromContent reads values file and inline values override file values'(@TempDir Path tempDir) { - // values file: replicas=1 - Path valuesFile = tempDir.resolve("harbor-values.yaml") - Files.writeString(valuesFile, """ - replicas: 1 - service: - type: ClusterIP - """.stripIndent()) - - def cfg = Config.fromMap(content: [helmReleases: [[name : 'harbor', - repoURL : 'https://helm.goharbor.io', - chart : 'harbor', - version : '1.18.2', - namespace : 'my-prefix-harbor', - releaseName: 'harbor', - valuesPath : valuesFile.toString(), - values : [replicas: 2, // override file - service : [type: 'NodePort'] // override nested - ]]]]) - - def contentLoader = createContent(cfg) - contentLoader.install() - - assertThat(contentLoader.deployCalls).hasSize(1) - - def call = contentLoader.deployCalls[0] - - // IMPORTANT: valuesPath is a temp file created by writeTempFile(...) - Path mergedTemp = Path.of(call.valuesPath) - assertThat(mergedTemp).exists() - - def mergedYaml = new YamlSlurper().parse(mergedTemp.toFile()) as Map - - // inline overrides file - assertThat(mergedYaml['replicas']).isEqualTo(2) - assertThat(((Map) mergedYaml['service'])['type']).isEqualTo('NodePort') - } - - @Test - void 'deployHelmReleasesFromContent uses values file when inline values are empty'(@TempDir Path tempDir) { - Path valuesFile = tempDir.resolve("values.yaml") - Files.writeString(valuesFile, """ - replicas: 1 - """.stripIndent()) - - def cfg = Config.fromMap(content: [helmReleases: [[name : 'elasticsearch', - repoURL : 'https://helm.elastic.co', - chart : 'elasticsearch', - version : '8.5.1', - namespace : 'my-prefix-elasticsearch', - valuesPath: valuesFile.toString() - // no values - ]]]) - - def contentLoader = createContent(cfg) - contentLoader.install() - - assertThat(contentLoader.deployCalls).hasSize(1) - - def call = contentLoader.deployCalls[0] - Path mergedTemp = Path.of(call.valuesPath) - assertThat(mergedTemp).exists() - - def mergedYaml = new YamlSlurper().parse(mergedTemp.toFile()) as Map - assertThat(mergedYaml['replicas']).isEqualTo(1) - } - - @Test - void 'deployHelmReleasesFromContent uses inline values when no helmValuesPath is set'() { - def cfg = Config.fromMap(content: [helmReleases: [[name : 'elasticsearch', - repoURL : 'https://helm.elastic.co', - chart : 'elasticsearch', - version : '8.5.1', - namespace: 'my-prefix-elasticsearch', - values : [replicas: 2] - // helmValuesPath empty / missing - ]]]) - - def contentLoader = createContent(cfg) - contentLoader.install() - - assertThat(contentLoader.deployCalls).hasSize(1) - - def call = contentLoader.deployCalls[0] - Path mergedTemp = Path.of(call.valuesPath) - assertThat(mergedTemp).exists() - - def mergedYaml = new YamlSlurper().parse(mergedTemp.toFile()) as Map - assertThat(mergedYaml['replicas']).isEqualTo(2) - } - - @Test - void 'deployHelmReleasesFromContent defaults chart version to wildcard when missing'() { - def cfg = Config.fromMap(content: [helmReleases: [[name : 'harbor', - repoURL : 'https://helm.goharbor.io', - chart : 'harbor', - version : ' ', // blank - namespace : 'my-prefix-harbor', - releaseName: 'harbor', - values : [foo: 'bar']]]]) - - def contentLoader = createContent(cfg) - contentLoader.install() - - assertThat(contentLoader.deployCalls).hasSize(1) - def call = contentLoader.deployCalls[0] - - assertThat(call.helmConfig.version).isEqualTo('*') - } - - static String createContentRepo(String initPath = '', String baseBareRepo = 'git-repository') { - // The bare repo works as the "remote" - def bareRepoDir = File.createTempDir('gitops-playground-test-content-repo') - bareRepoDir.deleteOnExit() - foldersToDelete << bareRepoDir - // init with bare repo - FileUtils.copyDirectory(new File(System.getProperty("user.dir") + "/src/test/groovy/com/cloudogu/gitops/utils/data/${baseBareRepo}/"), bareRepoDir) - def bareRepoUri = 'file://' + bareRepoDir.absolutePath - log.debug("Repo $initPath: bare repo $bareRepoUri") - - if (initPath) { - // Add initPath to bare repo - def tempRepo = File.createTempDir('gitops-playground-temp-repo') - tempRepo.deleteOnExit() - foldersToDelete << tempRepo - log.debug("Repo $initPath: cloned bare repo to $tempRepo") - try (def git = Git.cloneRepository() - .setURI(bareRepoUri) - .setBranch('main') - .setDirectory(tempRepo) - .call()) { - - FileUtils.copyDirectory(new File(System.getProperty("user.dir") + '/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/' + initPath), tempRepo) - - git.add().addFilepattern(".").call() - - // Avoid complications with local developer's git config, e.g. when git config --global commit.gpgSign true - SystemReader.getInstance().userConfig.clear() - git.commit().setMessage("Initialize with $initPath").call() - git.push().call() - tempRepo.delete() - } - } - - return bareRepoUri - } - - private Map parseYaml(String path) { - return new YamlSlurper().parse(new File(path)) as Map - } - - private void assertRegistrySecrets(String regUser, String regPw) {} - - private ContentLoaderForTest createContent(Config config) { - new ContentLoaderForTest(config, k8sClient, scmmRepoProvider, jenkins, gitHandler, fileSystemUtils, deploymentStrategy) - } - - private static parseActualYaml(File pathToYamlFile) { - def ys = new YamlSlurper() - return ys.parse(pathToYamlFile) - } - - private static String findRoot(List repos) { - def result = new File(repos.get(0).getClonedContentRepo().getParent()).getParent() - return result; - - } - - Git cloneRepo(String expectedRepo, File repoFolder) { - def repo = scmmRepoProvider.getRepo(expectedRepo, new ScmManagerMock()) - def url = repo.getGitRepositoryUrl() - - def git = Git.cloneRepository().setURI(url).setBranch('main').setDirectory(repoFolder).call() - git.getRepository().getConfig().setBoolean("gc", null, "autoDetach", false) - return git - } - - private File createRandomSubDir(String prefix = '') { - def randomDir = tmpDir.toPath().resolve("${prefix ? "${prefix}-" : ''}${System.currentTimeMillis()}").toFile() - randomDir.mkdirs() - return randomDir - } - - void assertTagAndReadme(String repo, String expectedTag, String expectedReadmeContent) { - def repoFolder = createRandomSubDir() - try (def git = cloneRepo(repo, repoFolder)) { - git.fetch().setRefSpecs("refs/*:refs/*").call() - assertTag(git, expectedTag) - - git.checkout().setName(expectedTag).call() - assertThat(new File(repoFolder, "README.md")).exists().isFile() - assertThat(new File(repoFolder, "README.md").text).contains(expectedReadmeContent) - } - } - - void assertBranchAndReadme(String repo, String expectedBranch, String expectedReadmeContent) { - def repoFolder = createRandomSubDir() - try (def git = cloneRepo(repo, repoFolder)) { - git.fetch().setRefSpecs("refs/*:refs/*").call() - assertBranch(git, expectedBranch) - - git.checkout().setName(expectedBranch).call() - assertThat(new File(repoFolder, "README.md")).exists().isFile() - assertThat(new File(repoFolder, "README.md").text).contains(expectedReadmeContent) - } - } - - class ContentLoaderForTest extends ContentLoader { - List deployCalls = [] - CloneCommand cloneSpy - - ContentLoaderForTest(Config config, K8sClient k8sClient, GitRepoFactory repoProvider, Jenkins jenkins, GitHandler gitHandler, FileSystemUtils fileSystemUtils, - DeploymentStrategy deploymentStrategy) { - super(config, k8sClient, repoProvider, jenkins, gitHandler, fileSystemUtils, deploymentStrategy) - } - - @Override - protected void deployHelmChart(String featureName, - String releaseName, - String namespace, - Config.HelmConfigWithValues helmConfig, - String helmValuesTemplatePath, - Config config) { - deployCalls << new DeployCall(featureName: featureName, - releaseName: releaseName, - namespace: namespace, - helmConfig: helmConfig, - valuesPath: helmValuesTemplatePath, - config: config) - } - - @Override - protected CloneCommand gitClone() { - cloneSpy = spy(super.gitClone().setNoCheckout(true)) - } - } - - static class DeployCall { - String featureName - String releaseName - String namespace - Config.HelmConfigWithValues helmConfig - String valuesPath - Config config - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/application/orchestration/GitHandlerTest.groovy b/src/test/groovy/com/cloudogu/gitops/application/orchestration/GitHandlerTest.groovy deleted file mode 100644 index 36a9503b1..000000000 --- a/src/test/groovy/com/cloudogu/gitops/application/orchestration/GitHandlerTest.groovy +++ /dev/null @@ -1,183 +0,0 @@ -package com.cloudogu.gitops.application.orchestration - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.util.ScmProviderType -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.GitlabMock -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.NetworkingUtils -import org.junit.jupiter.api.Test - -import static org.junit.jupiter.api.Assertions.* -import static org.mockito.Mockito.mock - -class GitHandlerTest { - - private static Config config(Map overrides = [:]) { - Map base = [application: [namePrefix: ''], - scm : [scmProviderType: ScmProviderType.SCM_MANAGER, // default - scmManager : [internal: true], - gitlab : [url: '']], - multiTenant: [scmManager : [url: ''], - gitlab : [url: ''], - useDedicatedInstance: false]] - Map merged = deepMerge(base, overrides) - return new Config().fromMap(merged) - } - - /** simple deep merge for nested maps */ - @SuppressWarnings('unchecked') - private static Map deepMerge(Map left, Map right) { - Map out = [:] + left - right.each { k, v -> - if (v instanceof Map && left[k] instanceof Map) { - out[k] = deepMerge((Map) left[k], (Map) v) - } else { - out[k] = v - } - } - return out - } - - private static GitHandler handler(Config cfg) { - return new GitHandler(cfg, - mock(HelmStrategy), - mock(FileSystemUtils), - mock(K8sClient), - mock(NetworkingUtils)) - } - - // ---------- validate() ------------------------------------------------------------ - - @Test - void 'validate(): ScmManager external url sets internal=false and urlForJenkins equals url'() { - def cfg = config([application: [namePrefix: 'fv40-'], - scm : [scmManager: [url: 'https://scmm.example.com/scm', internal: true]]]) - def gh = handler(cfg) - - gh.validate() - - assertFalse(cfg.scm.scmManager.internal) - assertEquals('https://scmm.example.com/scm', cfg.scm.scmManager.urlForJenkins) - } - - @Test - void 'validate(): GitLab chosen, provider switched, scmm nulled, missing PAT or parentGroupId throws'() { - def cfg = config([scm: [gitlab: [url: 'https://gitlab.example.com']]]) - def gh = handler(cfg) - - def ex = assertThrows(RuntimeException) { gh.validate() } - assertTrue(ex.message.toLowerCase().contains('gitlab')) - assertEquals(ScmProviderType.GITLAB, cfg.scm.scmProviderType) - assertNull(cfg.scm.scmManager) - } - - // ---------- getResourcesScm() ----------------------------------------------------- - - @Test - void 'getResourcesScm(): central wins over tenant'() { - def cfg = config() - def gitHandler = handler(cfg) - - gitHandler.tenant = mock(GitProvider, 'tenant') - gitHandler.central = mock(GitProvider, 'central') - - assertSame(gitHandler.central, gitHandler.getResourcesScm()) - } - - @Test - void 'getResourcesScm(): tenant returned when central absent, throws when none'() { - def cfg = config() - def gitHandler = handler(cfg) - - gitHandler.tenant = mock(GitProvider) - assertSame(gitHandler.tenant, gitHandler.getResourcesScm()) - - gitHandler.tenant = null - def ex = assertThrows(IllegalStateException) { gitHandler.getResourcesScm() } - assertTrue(ex.message.contains('No SCM provider')) - } - - // ---------- enable(): SCM_MANAGER tenant only ------------------------------------ - @Test - void 'ScmManager tenant-only: tenant gets 1 repository'() { - def cfg = new Config().fromMap([scm : [scmManager: [internal: true], gitlab: [url: '']], - multiTenant: [useDedicatedInstance: false]]) - - def tenant = new ScmManagerMock() - def gitHandler = new GitHandlerForTests(cfg, tenant) - - gitHandler.enable() - - assertEquals('scm-manager', cfg.scm.scmManager.namespace) - - assertTrue(tenant.createdRepos.contains('argocd/cluster-resources')) - assertEquals(1, tenant.createdRepos.size()) - - // No central provider in tenant-only scenario - assertNull(gitHandler.getCentral()) - } - - @Test - void 'ScmManager dedicated: central gets 1 repo, tenant gets 1 repo'() { - def cfg = config([application: [namePrefix: 'fv40-'], - scm : [scmProviderType: ScmProviderType.SCM_MANAGER, - scmManager : [internal: true], - gitlab : [url: '']], - multiTenant: [useDedicatedInstance: true, - scmManager : [url: ''], - gitlab : [url: '']]]) - - def tenant = new ScmManagerMock(namePrefix: 'fv40-') - def central = new ScmManagerMock(namePrefix: 'fv40-') - def gitHandler = new GitHandlerForTests(cfg, tenant, central) - - gitHandler.enable() - - // Central: argocd/cluster-resources - assertTrue(central.createdRepos.contains('fv40-argocd/cluster-resources')) - assertEquals(1, central.createdRepos.size()) - - // Tenant: argocd/cluster-resources - assertTrue(tenant.createdRepos.contains('fv40-argocd/cluster-resources')) - assertEquals(1, tenant.createdRepos.size()) - } - - @Test - void 'Gitlab dedicated: same layout as ScmManager dedicated'() { - def cfg = config([application: [namePrefix: 'fv40-'], - scm : [scmProviderType: ScmProviderType.GITLAB, - gitlab : [url: 'https://gitlab.example.com', password: 'pat', parentGroupId: 123], - scmManager : [internal: true]], - multiTenant: [useDedicatedInstance: true, - gitlab : [url: 'https://gitlab.example.com', password: 'pat2', parentGroupId: 456], - scmManager : [url: '']]]) - - // Assumes your GitlabMock has a similar contract to ScmManagerMock (collects createdRepos) - def tenant = new GitlabMock(base: new URI(cfg.scm.gitlab.url), namePrefix: 'fv40-') - def central = new GitlabMock(base: new URI(cfg.multiTenant.gitlab.url), namePrefix: 'fv40-') - def gitHandler = new GitHandlerForTests(cfg, tenant, central) - - gitHandler.enable() - - // Central: argocd/cluster-resources - assertTrue(central.createdRepos.contains('fv40-argocd/cluster-resources')) - assertEquals(1, central.createdRepos.size()) - - // Tenant: argocd/cluster-resources - assertTrue(tenant.createdRepos.contains('fv40-argocd/cluster-resources')) - assertEquals(1, tenant.createdRepos.size()) - } - - @Test - void 'withOrgPrefix helper behaves as expected'() { - assertEquals('argocd/argocd', GitHandler.withOrgPrefix('', 'argocd/argocd')) - assertEquals('argocd/argocd', GitHandler.withOrgPrefix(null, 'argocd/argocd')) - assertEquals('fv40-argocd/argocd', GitHandler.withOrgPrefix('fv40-', 'argocd/argocd')) - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/cli/ApplicationConfiguratorTest.groovy b/src/test/groovy/com/cloudogu/gitops/cli/ApplicationConfiguratorTest.groovy deleted file mode 100644 index d6a70f3f3..000000000 --- a/src/test/groovy/com/cloudogu/gitops/cli/ApplicationConfiguratorTest.groovy +++ /dev/null @@ -1,643 +0,0 @@ -package com.cloudogu.gitops.cli - -import com.cloudogu.gitops.application.content.ContentLoader -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.ScmTenantSchema -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.TestLogger -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.tools.common.CommonToolConfig -import com.cloudogu.gitops.tools.core.Jenkins -import com.cloudogu.gitops.tools.core.argocd.ArgoCD -import com.cloudogu.gitops.utils.FileSystemUtils -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.mockito.Mock -import org.mockito.Mockito - -import static com.github.stefanbirkner.systemlambda.SystemLambda.withEnvironmentVariable -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat - -class ApplicationConfiguratorTest { - - static final String EXPECTED_REGISTRY_URL = 'http://my-reg' - static final int EXPECTED_REGISTRY_INTERNAL_PORT = 33333 - static final Config.VaultMode EXPECTED_VAULT_MODE = Config.VaultMode.dev - public static final String EXPECTED_JENKINS_URL = 'http://my-jenkins' - public static final String EXPECTED_SCMM_URL = 'http://my-scmm' - - private ApplicationConfigurator applicationConfigurator - private FileSystemUtils fileSystemUtils - private TestLogger testLogger - private CommonToolConfig commonFeatureConfig - private ContentLoader featureContent - private ArgoCD featureArgoCd - - @Mock - ScmManagerMock scmManagerMock = new ScmManagerMock() - - Config testConfig = Config.fromMap([application: [localHelmChartFolder: 'someValue', - namePrefix : ''], - registry : [url : EXPECTED_REGISTRY_URL, - proxyUrl : "proxy-$EXPECTED_REGISTRY_URL", - proxyUsername: "proxy-user", - proxyPassword: "proxy-pw", - internalPort : EXPECTED_REGISTRY_INTERNAL_PORT,], - jenkins : [url: EXPECTED_JENKINS_URL], - scm : [scmManager: [url: EXPECTED_SCMM_URL],], - multiTenant: [scmManager: [url: '']], - features : [secrets: [vault: [mode: EXPECTED_VAULT_MODE]],]]) - - // // We have to set this value using env vars, which makes tests complicated, so ignore it - // Config almostEmptyConfig = Config.fromMap([ - // application: [ - // localHelmChartFolder: 'someValue', - // ], - // ]) - - @BeforeEach - void setup() { - fileSystemUtils = new FileSystemUtils() - applicationConfigurator = new ApplicationConfigurator(fileSystemUtils) - testLogger = new TestLogger(applicationConfigurator.getClass()) - commonFeatureConfig = new CommonToolConfig() - - K8sClient k8sClient = Mockito.mock(K8sClient) - HelmClient helmClient = Mockito.mock(HelmClient) - GitRepoFactory gitRepoFactory = Mockito.mock(GitRepoFactory) - - DeploymentStrategy deploymentStrategy = Mockito.mock(DeploymentStrategy) - - GitHandler gitHandler = new GitHandlerForTests(testConfig, scmManagerMock) - featureContent = Mockito.spy(new ContentLoader(testConfig, k8sClient, gitRepoFactory, Mockito.mock(Jenkins), gitHandler, fileSystemUtils, deploymentStrategy)) - featureArgoCd = Mockito.spy(new ArgoCD(testConfig, k8sClient, helmClient, fileSystemUtils, gitRepoFactory, gitHandler)) - } - - @Test - void "correct config with no programm arguments"() { - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.jenkins.url).isEqualTo(EXPECTED_JENKINS_URL) - assertThat(actualConfig.jenkins.internal).isEqualTo(false) - assertThat(actualConfig.features.secrets.vault.mode).isEqualTo(EXPECTED_VAULT_MODE) - - // Dynamic value (depends on vault mode) - assertThat(actualConfig.features.secrets.active).isEqualTo(true) - } - - @Test - void "sets config application runningInsideK8s"() { - withEnvironmentVariable("KUBERNETES_SERVICE_HOST", "127.0.0.1").execute { - Config actualConfig = applicationConfigurator.initConfig(testConfig) - assertThat(actualConfig.application.runningInsideK8s).isEqualTo(true) - } - } - - @Test - void 'Sets jenkins active if external url is set'() { - testConfig.jenkins.url = 'external' - def actualConfig = applicationConfigurator.initConfig(testConfig) - assertThat(actualConfig.jenkins.active).isEqualTo(true) - } - - @Test - void 'Leaves Jenkins urlForScmm empty, if not active'() { - testConfig.jenkins.url = '' - testConfig.jenkins.active = false - - def actualConfig = applicationConfigurator.initConfig(testConfig) - assertThat(actualConfig.jenkins.urlForScm).isEmpty() - } - - @Test - void 'Fails if monitoring local is not set'() { - testConfig.application.mirrorRepos = true - testConfig.application.localHelmChartFolder = '' - - def exception = shouldFail(RuntimeException) { - commonFeatureConfig.validateConfig(testConfig) - } - assertThat(exception.message).isEqualTo('Missing config for localHelmChartFolder.\n' + - 'Either run inside the official container image or setting env var LOCAL_HELM_CHART_FOLDER=\'charts\' ' + - 'after running \'scripts/downloadHelmCharts.sh\' from the repo') - } - - @Test - void 'Fails if createImagePullSecrets is used without secrets'() { - testConfig.registry.createImagePullSecrets = true - - def exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - assertThat(exception.message).isEqualTo('createImagePullSecrets needs to be used with either registry username and password or the readOnly variants') - } - - @Test - void 'Fails if content repo is set without mandatory params'() { - - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: ''),] - def exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.repos requires a url parameter.') - - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.COPY, target: "missing_slash"),] - exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.target needs / to separate namespace/group from repo name. Repo: abc') - } - - @Test - void 'Fails if COPY repo misses target parameter'() { - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.COPY),] - def exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.repos.type COPY requires content.repos.target to be set. Repo: abc') - } - - @Test - void 'Fails if FOLDER_BASED repo has target parameter'() { - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.FOLDER_BASED, target: 'namespace/repo'),] - def exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.repos.type FOLDER_BASED does not support target parameter. Repo: abc') - - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.FOLDER_BASED, targetRef: 'someRef'),] - exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.repos.type FOLDER_BASED does not support targetRef parameter. Repo: abc') - } - - @Test - void 'Fails if MIRROR repo has invalid configuration'() { - // Test missing target parameter - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.MIRROR),] - def exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.repos.type MIRROR requires content.repos.target to be set. Repo: abc') - - // Test setting path - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.MIRROR, - target: 'namespace/repo', path: 'non-default-path'),] - exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo("content.repos.type MIRROR does not support path. Current path: non-default-path. Repo: abc") - - // Test templating enabled - testConfig.content.repos = [new Config.ContentSchema.ContentRepositorySchema(url: 'abc', type: Config.ContentRepoType.MIRROR, - target: 'namespace/repo', templating: true),] - exception = shouldFail(RuntimeException) { - featureContent.preConfigInit(testConfig) - } - assertThat(exception.message).isEqualTo('content.repos.type MIRROR does not support templating. Repo: abc') - } - - @Test - void 'Ignores empty localHemlChartFolder, if mirrorRepos is not set'() { - testConfig.application.mirrorRepos = false - testConfig.application.localHelmChartFolder = '' - - applicationConfigurator.initConfig(testConfig) - // no exceptions means success - } - - @Test - void "base url: evaluates for all tools"() { - testConfig.application.baseUrl = 'http://localhost' - - testConfig.features.argocd.active = true - testConfig.features.monitoring.active = true - testConfig.features.secrets.active = true - - Config actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.url).isEqualTo("http://argocd.localhost") - assertThat(actualConfig.features.monitoring.grafanaUrl).isEqualTo("http://grafana.localhost") - assertThat(actualConfig.features.secrets.vault.url).isEqualTo("http://vault.localhost") - assertThat(actualConfig.scm.scmManager.ingress).isEqualTo("scmm.localhost") - assertThat(actualConfig.jenkins.ingress).isEqualTo("jenkins.localhost") - } - - @Test - void "base url with url-hyphens: evaluates for all tools"() { - testConfig.application.baseUrl = 'http://localhost' - testConfig.application.urlSeparatorHyphen = true - - testConfig.features.argocd.active = true - testConfig.features.monitoring.active = true - testConfig.features.secrets.active = true - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.url).isEqualTo("http://argocd-localhost") - assertThat(actualConfig.features.monitoring.grafanaUrl).isEqualTo("http://grafana-localhost") - assertThat(actualConfig.features.secrets.vault.url).isEqualTo("http://vault-localhost") - assertThat(actualConfig.scm.scmManager.ingress).isEqualTo("scmm-localhost") - assertThat(actualConfig.jenkins.ingress).isEqualTo("jenkins-localhost") - } - - @Test - void "base url: also works when port is included "() { - testConfig.application.baseUrl = 'http://localhost:8080' - testConfig.features.argocd.active = true - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.url).isEqualTo("http://argocd.localhost:8080") - } - - @Test - void "base url: also works when port is included and use url-hyphens is set"() { - testConfig.application.baseUrl = 'http://localhost:6502' - testConfig.features.argocd.active = true - testConfig.application.urlSeparatorHyphen = true - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.url).isEqualTo("http://argocd-localhost:6502") - } - - @Test - void "base url: does not evaluate for inactive tools"() { - testConfig.features.argocd.active = false - testConfig.features.mail.active = false - testConfig.features.monitoring.active = false - testConfig.features.secrets.active = false - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.url).isEqualTo('') - assertThat(actualConfig.features.monitoring.grafanaUrl).isEqualTo('') - assertThat(actualConfig.features.secrets.vault.url).isEqualTo('') - } - - @Test - void "base url: individual url params take precedence"() { - testConfig.application.baseUrl = 'http://localhost' - - testConfig.features.argocd.active = true - testConfig.features.mail.active = true - testConfig.features.monitoring.active = true - testConfig.features.secrets.active = true - - testConfig.features.argocd.url = 'argocd' - testConfig.features.monitoring.grafanaUrl = 'grafana' - testConfig.features.secrets.vault.url = 'vault' - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.url).isEqualTo("argocd") - assertThat(actualConfig.features.monitoring.grafanaUrl).isEqualTo("grafana") - assertThat(actualConfig.features.secrets.vault.url).isEqualTo("vault") - } - - @Test - void "Sets namePrefix"() { - testConfig.application.namePrefix = 'my-prefix' - - def actualConfig = applicationConfigurator.initConfig(testConfig) - assertThat(actualConfig.application.namePrefix.toString()).isEqualTo('my-prefix-') - assertThat(actualConfig.application.namePrefixForEnvVars.toString()).isEqualTo('MY_PREFIX_') - } - - @Test - void "Sets namePrefix when ending in hyphen"() { - testConfig.application.namePrefix = 'my-prefix-' - - def actualConfig = applicationConfigurator.initConfig(testConfig) - assertThat(actualConfig.application.namePrefix.toString()).isEqualTo('my-prefix-') - assertThat(actualConfig.application.namePrefixForEnvVars.toString()).isEqualTo('MY_PREFIX_') - } - - @Test - void "Registry: Sets to external when only registry URL set"() { - testConfig.registry.proxyUrl = null - - def actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.registry.internal).isEqualTo(false) - assertThat(actualConfig.registry.active).isEqualTo(true) - } - - @Test - void "Registry: Fails when proxy but no username and password set"() { - def expectedException = 'Proxy URL needs to be used with proxy-username and proxy-password' - - testConfig.registry.proxyUsername = null - def exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - assertThat(exception.message).isEqualTo(expectedException) - - testConfig.registry.proxyUsername = 'something' - testConfig.registry.proxyPassword = null - exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - assertThat(exception.message).isEqualTo(expectedException) - - testConfig.registry.proxyUsername = null - exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - assertThat(exception.message).isEqualTo(expectedException) - } - - @Test - void "validateEnvConfig allows valid env entries"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = 'https://100.125.0.1:443' - testConfig.features.argocd.env = [[name: "ENV_VAR_1", value: "value1"], - [name: "ENV_VAR_2", value: "value2"]] as List> - - // No exception should be thrown - applicationConfigurator.initConfig(testConfig) - } - - @Test - void "validateEnvConfig throws exception for missing 'name' in env entry"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = 'https://100.125.0.1:443' - testConfig.features.argocd.env = [[name: "ENV_VAR_1", value: "value1"], - [value: "value2"] // Missing 'name' - ] as List> - - def exception = shouldFail(IllegalArgumentException) { - applicationConfigurator.initConfig(testConfig) - featureArgoCd.postConfigInit(testConfig) - } - - assertThat(exception.message).contains("Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: [value:value2]") - } - - @Test - void "validateEnvConfig throws exception for missing 'value' in env entry"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = 'https://100.125.0.1:443' - testConfig.features.argocd.env = [[name: "ENV_VAR_1", value: "value1"], - [name: "ENV_VAR_2"] // Missing 'value' - ] as List> - - def exception = shouldFail(IllegalArgumentException) { - applicationConfigurator.initConfig(testConfig) - featureArgoCd.postConfigInit(testConfig) - } - - assertThat(exception.message).contains("Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: [name:ENV_VAR_2]") - } - - @Test - void "validateEnvConfig throws exception for non-map env entry"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = 'https://100.125.0.1:443' - testConfig.features.argocd.env = [[name: "ENV_VAR_1", value: "value1"], - "invalid_entry" // Invalid entry - ] as List> - - def exception = shouldFail(IllegalArgumentException) { - applicationConfigurator.initConfig(testConfig) - featureArgoCd.postConfigInit(testConfig) - } - - assertThat(exception.message).contains("Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: invalid_entry") - } - - @Test - void "validateEnvConfig allows empty env list"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = 'https://100.125.0.1:443' - testConfig.features.argocd.env - - // No exception should be thrown - applicationConfigurator.initConfig(testConfig) - } - - @Test - void "validateEnvConfig skips validation when operator is false"() { - testConfig.features.argocd.operator = false - testConfig.features.argocd.env = [[name: "ENV_VAR_1", value: "value1"], - [value: "value2"] // Invalid entry, but should be ignored - ] as List> - - // No exception should be thrown - applicationConfigurator.initConfig(testConfig) - } - - @Test - void "should skip resourceInclusionsCluster setup when ArgoCD operator is not enabled"() { - testConfig.features.argocd.operator = false - - // Calling the method should not make any changes to the config - applicationConfigurator.initConfig(testConfig) - - assertThat(testLogger.getLogs().search("ArgoCD operator is not enabled. Skipping features.argocd.resourceInclusionsCluster setup.")) - .isNotEmpty() - } - - @Test - void "should validate and accept user-provided valid resourceInclusionsCluster URL"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = "https://valid-url.com" - - // Calling the method should accept the valid URL and not throw any exception - applicationConfigurator.initConfig(testConfig) - - assertThat(testConfig.features.argocd.resourceInclusionsCluster).isEqualTo("https://valid-url.com") - assertThat(testLogger.getLogs().search("Validating user-provided features.argocd.resourceInclusionsCluster URL: https://valid-url.com")) - .isNotEmpty() - assertThat(testLogger.getLogs().search("Found valid URL in features.argocd.resourceInclusionsCluster: https://valid-url.com")) - .isNotEmpty() - } - - @Test - void "should throw exception for user-provided invalid resourceInclusionsCluster URL"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = "invalid-url" - - def exception = shouldFail(IllegalArgumentException) { - applicationConfigurator.initConfig(testConfig) - } - - assertThat(exception.message).contains("Invalid URL for 'features.argocd.resourceInclusionsCluster': invalid-url.") - } - - @Test - void "should set resourceInclusionsCluster using Kubernetes ENV variables when not provided by user"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = null - - // Set Kubernetes ENV variables - withEnvironmentVariable("KUBERNETES_SERVICE_HOST", "127.0.0.1") - .and("KUBERNETES_SERVICE_PORT", "6443") - .execute { - Config actualConfig = applicationConfigurator.initConfig(testConfig) - - assertThat(actualConfig.features.argocd.resourceInclusionsCluster).isEqualTo("https://127.0.0.1:6443") - - assertThat(testLogger.getLogs().search("Successfully set features.argocd.resourceInclusionsCluster via Kubernetes ENV to: https://127.0.0.1:6443")) - .isNotEmpty() - } - } - - @Test - void "MultiTenant Mode Central SCM Url"() { - testConfig.multiTenant.scmManager.url = "scmm.localhost/scm" - testConfig.application.namePrefix = "foo" - applicationConfigurator.initConfig(testConfig) - assertThat(testConfig.multiTenant.scmManager.url).toString() == "scmm.localhost/scm/" - } - - @Test - void "should throw exception when Kubernetes ENV variables are not set and resourceInclusionsCluster is null"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = null - - def exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - - assertThat(exception.message).contains("Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true. Ensure Kubernetes environment variables 'KUBERNETES_SERVICE_HOST' and 'KUBERNETES_SERVICE_PORT' are set properly.") - } - - @Test - void "should throw exception when Kubernetes ENV variables are not set and resourceInclusionsCluster is empty"() { - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = '' - - def exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - - assertThat(exception.message).contains("Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true. Ensure Kubernetes environment variables 'KUBERNETES_SERVICE_HOST' and 'KUBERNETES_SERVICE_PORT' are set properly.") - } - - @Test - void "should throw exception for invalid Kubernetes constructed URL"() { - // Set ArgoCD operator to true - testConfig.features.argocd.operator = true - testConfig.features.argocd.resourceInclusionsCluster = null - - // Set invalid Kubernetes ENV variables - withEnvironmentVariable("KUBERNETES_SERVICE_HOST", "invalid_host") - .and("KUBERNETES_SERVICE_PORT", "not_a_port") - .execute { - def exception = shouldFail(RuntimeException) { - applicationConfigurator.initConfig(testConfig) - } - - assertThat(exception.message).contains("Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true.") - } - - assertThat(testLogger.getLogs().search("Constructed internal Kubernetes API Server URL: https://invalid_host:not_a_port")).isNotEmpty() - } - - @Test - void "sets all tool namespaces to application namespace when configured"() { - Config config = minimalConfig() - config.application.namespace = 'platform' - config.application.namePrefix = 'tenant-a' - - config.application.gopNamespace = 'custom-gop' - config.registry.namespace = 'custom-registry' - config.jenkins.namespace = 'custom-jenkins' - config.scm.scmManager.namespace = 'custom-scm' - config.features.argocd.namespace = 'custom-argocd' - config.features.monitoring.namespace = 'custom-monitoring' - config.features.secrets.namespace = 'custom-secrets' - config.features.ingress.ingressNamespace = 'custom-ingress' - config.features.certManager.namespace = 'custom-cert-manager' - config.content.namespaces = ['old-namespace', 'another-namespace'] - - Config actualConfig = applicationConfigurator.initConfig(config) - - assertThat(actualConfig.application.gopNamespace).isEqualTo('platform') - assertThat(actualConfig.registry.namespace).isEqualTo('platform') - assertThat(actualConfig.jenkins.namespace).isEqualTo('platform') - assertThat(actualConfig.scm.scmManager.namespace).isEqualTo('platform') - assertThat(actualConfig.features.argocd.namespace).isEqualTo('platform') - assertThat(actualConfig.features.monitoring.namespace).isEqualTo('platform') - assertThat(actualConfig.features.secrets.namespace).isEqualTo('platform') - assertThat(actualConfig.features.ingress.ingressNamespace).isEqualTo('platform') - assertThat(actualConfig.features.certManager.namespace).isEqualTo('platform') - assertThat(actualConfig.content.namespaces).containsExactly('tenant-a-platform') - } - - @Test - void "keeps individual tool namespaces when application namespace is not configured"() { - Config config = minimalConfig() - config.application.namespace = '' - config.application.namePrefix = 'tenant-a' - - config.application.gopNamespace = 'custom-gop' - config.registry.namespace = 'custom-registry' - config.jenkins.namespace = 'custom-jenkins' - config.scm.scmManager.namespace = 'custom-scm' - config.features.argocd.namespace = 'custom-argocd' - config.features.monitoring.namespace = 'custom-monitoring' - config.features.secrets.namespace = 'custom-secrets' - config.features.ingress.ingressNamespace = 'custom-ingress' - config.features.certManager.namespace = 'custom-cert-manager' - config.content.namespaces = ['old-namespace', 'another-namespace'] - - Config actualConfig = applicationConfigurator.initConfig(config) - - assertThat(actualConfig.application.gopNamespace).isEqualTo('custom-gop') - assertThat(actualConfig.registry.namespace).isEqualTo('custom-registry') - assertThat(actualConfig.jenkins.namespace).isEqualTo('custom-jenkins') - assertThat(actualConfig.scm.scmManager.namespace).isEqualTo('custom-scm') - assertThat(actualConfig.features.argocd.namespace).isEqualTo('custom-argocd') - assertThat(actualConfig.features.monitoring.namespace).isEqualTo('custom-monitoring') - assertThat(actualConfig.features.secrets.namespace).isEqualTo('custom-secrets') - assertThat(actualConfig.features.ingress.ingressNamespace).isEqualTo('custom-ingress') - assertThat(actualConfig.features.certManager.namespace).isEqualTo('custom-cert-manager') - assertThat(actualConfig.content.namespaces).containsExactly('old-namespace', 'another-namespace') - } - - List getAllFieldNames(Class clazz, String parentField = '', List fieldNames = []) { - clazz.declaredFields.each { field -> - def currentField = parentField + field.name - if (field.type instanceof Class && !field.type.isArray() && field.type.name.startsWith(Config.class.getPackageName())) { - println "nested class $field.type, $currentField + '.', $fieldNames" - getAllFieldNames(field.type, currentField + '.', fieldNames) - } else { - if (!field.name.startsWith('_') && !field.name.startsWith('$') && field.name != 'metaClass') { - fieldNames.add(currentField) - } - } - } - return fieldNames - } - - List getAllKeys(Map map, String parentKey = '', List keysList = []) { - map.each { key, value -> - def currentKey = parentKey + key - if (value instanceof Map && !value.isEmpty()) { - getAllKeys(value, currentKey + '.', keysList) - } else { - keysList.add(currentKey) - } - } - return keysList - } - - private static Config minimalConfig() { - def config = new Config() - config.application = new Config.ApplicationSchema(localHelmChartFolder: 'someValue', - namePrefix: '') - config.scm = new ScmTenantSchema(scmManager: new ScmTenantSchema.ScmManagerTenantConfig(url: '')) - return config - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliMainTest.groovy b/src/test/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliMainTest.groovy deleted file mode 100644 index 264c81d0f..000000000 --- a/src/test/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliMainTest.groovy +++ /dev/null @@ -1,65 +0,0 @@ -package com.cloudogu.gitops.cli - -import static org.assertj.core.api.Assertions.assertThat - -import com.github.stefanbirkner.systemlambda.SystemLambda -import org.junit.jupiter.api.Test -import picocli.CommandLine.Command -import picocli.CommandLine.Option - -class GitopsPlaygroundCliMainTest { - - @Test - void 'application returns exit code 0 on success'() { - def gitopsPlaygroundCliMain = new GitopsPlaygroundCliMain() - int status = SystemLambda.catchSystemExit(() -> { - gitopsPlaygroundCliMain.exec(['--mock'] as String[], MockedCommand.class) - }) - - assertThat(status).isZero() - } - - @Test - void 'application returns exit code 1 on exception'() { - def gitopsPlaygroundCliMain = new GitopsPlaygroundCliMain() - int status = SystemLambda.catchSystemExit(() -> { - gitopsPlaygroundCliMain.exec(['--mock'] as String[], ThrowingCommand.class) - }) - - assertThat(status).isNotZero() - } - - @Test - void 'application returns exit code != 0 on invalid param'() { - int status = SystemLambda.catchSystemExit(() -> { - GitopsPlaygroundCliMain.main(['--parameter-that-doesnt-exist ', - '--debug' // avoids changing default log pattern - ] as String[]) - }) - - assertThat(status).isNotZero() - } - - static class ThrowingCommand extends MockedCommand { - @Override - ReturnCode run(String[] args) { - throw new RuntimeException("mock") - } - } - - @SuppressWarnings('unused') - // Used for annotations - static class MockedCommand extends GitopsPlaygroundCli { - - @Override - ReturnCode run(String[] args) { - return ReturnCode.SUCCESS - } - - @Command - void mockedCommand() {} - - @Option(names = ['--mock']) - private boolean mock - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliTest.groovy b/src/test/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliTest.groovy deleted file mode 100644 index e15909292..000000000 --- a/src/test/groovy/com/cloudogu/gitops/cli/GitopsPlaygroundCliTest.groovy +++ /dev/null @@ -1,344 +0,0 @@ -package com.cloudogu.gitops.cli - -import ch.qos.logback.classic.Logger -import ch.qos.logback.classic.LoggerContext -import ch.qos.logback.classic.encoder.PatternLayoutEncoder -import ch.qos.logback.core.ConsoleAppender -import com.cloudogu.gitops.application.Application -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.destroy.Destroyer -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.fasterxml.jackson.dataformat.yaml.YAMLMapper -import io.micronaut.context.ApplicationContext -import org.junit.jupiter.api.AfterEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.Timeout -import org.mockito.invocation.InvocationOnMock -import org.mockito.stubbing.Answer -import org.slf4j.LoggerFactory - -import java.util.concurrent.TimeUnit - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.* - -// Avoids blocking if input is read by error -@Timeout(value = 10, unit = TimeUnit.SECONDS) -class GitopsPlaygroundCliTest { - - static final String ORIGINAL_LOGGING_PATTERN = loggingEncoder.pattern - - K8sClient k8sClient = mock(K8sClient) - Application application = mock(Application) - ApplicationConfigurator applicationConfigurator = mock(ApplicationConfigurator) - Destroyer destroyer = mock(Destroyer) - GitopsPlaygroundCliForTest cli = new GitopsPlaygroundCliForTest() - static YAMLMapper yamlMapper = new YAMLMapper() - - @AfterEach - void setup() { - // Restore logging pattern, if modified - loggingEncoder.setPattern(ORIGINAL_LOGGING_PATTERN) - } - - @Test - void 'Starts regularly'() { - def status = cli.run('--yes') - - assertThat(status).isEqualTo(ReturnCode.SUCCESS) - verify(applicationConfigurator).initConfig(any(Config)) - verify(application).start() - } - - @Test - void 'Starts with config file'() { - String pathToConfigFile = "./src/test/resources/testMainConfig.yaml" - - assertThat(new File(pathToConfigFile).isFile()).withFailMessage("config file for test do not exists anymore.").isTrue() - - def status = cli.run('--config-file=' + pathToConfigFile) - assertThat(status).isEqualTo(ReturnCode.SUCCESS) - - // Verify the first interaction - verify(applicationConfigurator).initConfig(any(Config)) - - // Check application starts - verify(application).start() - } - - @Test - void 'Starts with config map'() { - when(k8sClient.getConfigMap('my-config', 'config.yaml')).thenReturn('{"application": {"yes": true}}') - - def status = cli.run("--config-map=my-config") - - assertThat(status).isEqualTo(ReturnCode.SUCCESS) - // ensure init is called with Config - verify(applicationConfigurator).initConfig(any(Config)) - verify(application).start() - } - - @Test - void 'Outputs config file'() { - def status = cli.run('--output-config-file') - - assertThat(status).isEqualTo(ReturnCode.SUCCESS) - verify(applicationConfigurator, never()).initConfig(any(Config)) - verify(application, never()).start() - } - - @Test - void 'Outputs version'() { - def cli = new GitopsPlaygroundCliForTest() - def status = cli.run('--version') - - assertThat(status).isEqualTo(ReturnCode.SUCCESS) - verify(applicationConfigurator, never()).initConfig(any(Config)) - verify(application, never()).start() - } - - @Test - void 'Outputs help'() { - def cli = new GitopsPlaygroundCliForTest() - def status = cli.run('--help') - - assertThat(status).isEqualTo(ReturnCode.SUCCESS) - verify(applicationConfigurator, never()).initConfig(any(Config)) - verify(application, never()).start() - } - - @Test - void 'Returns error, when applying is not confirmed'() { - writeViaSystemIn('something') - def status = cli.run() - - assertThat(status).isEqualTo(ReturnCode.NOT_CONFIRMED) - } - - @Test - void 'Runs when applying is confirmed'() { - writeViaSystemIn('y') - - cli.run() - - verify(application).start() - } - - @Test - void 'Runs without confirmation when yes parameter is set'() { - cli.run('--yes') - - verify(application).start() - } - - @Test - void 'Returns error, when destroying is not confirmed'() { - - writeViaSystemIn('something') - - def status = cli.run('--destroy') - - assertThat(status).isEqualTo(ReturnCode.NOT_CONFIRMED) - } - - @Test - void 'Destroys when confirmed'() { - - writeViaSystemIn('y') - - cli.run '--destroy' - - verify(destroyer).destroy() - verify(application, never()).start() - } - - @Test - void 'Destroys without confirmation when yes parameter is set'() { - cli.run('--destroy', '--yes') - - verify(destroyer).destroy() - } - - @Test - void 'sets simplified logging pattern'() { - cli.run('--yes') - - assertThat(getLoggingPattern()).doesNotContain('%logger', '%thread') - } - - @Test - void 'keeps simplified logging pattern when trace is enabled'() { - cli.run('--trace', '--yes') - - assertThat(getLoggingPattern()).contains('%logger', '%thread') - } - - @Test - void 'keeps simplified logging pattern when debug is enabled'() { - cli.run('--debug', '--yes') - - assertThat(getLoggingPattern()).contains('%logger', '%thread') - } - - @Test - void 'fails on invalid config file'() { - - def configFile = File.createTempFile("gop", '.yaml') - configFile.deleteOnExit() - configFile.text = 'something: not-matching-our-schema' - - def exception = shouldFail(RuntimeException) { - cli.run("--config-file=${configFile}", '--yes') - } - assertThat(exception.message).contains('Config file invalid') - } - - @Test - void 'fails on invalid config map'() { - when(k8sClient.getConfigMap('my-config', 'config.yaml')).thenReturn('something: not-matching-our-schema') - - def exception = shouldFail(RuntimeException) { - cli.run('--config-map=my-config', '--yes') - } - assertThat(exception.message).contains('Config file invalid') - } - - @Test - void 'Precedence: config file overwrite confiMap, cli overwrites config file'() { - - def cmConfig = [application: [username: 'cmUser', password: 'cmPw', namePrefix: 'cmPref']] - def fileConfig = [application: [username: 'fileUser', password: 'filePw']] - - def configFile = File.createTempFile("gop", '.yaml') - configFile.deleteOnExit() - - configFile.text = toYaml(fileConfig) - when(k8sClient.getConfigMap('my-config', 'config.yaml')).thenReturn(toYaml(cmConfig)) - - cli.run("--config-file=${configFile}", '--config-map=my-config', '--username=paramUser', '--yes') - - assertThat(cli.lastSchema.application.username).isEqualTo('paramUser') - assertThat(cli.lastSchema.application.password).isEqualTo('filePw') - assertThat(cli.lastSchema.application.namePrefix).isEqualTo('cmPref') - } - - @Test - void 'Helm null values overwrite'() { - - def fileConfig = [features: [monitoring: [helm: [repoURL: "https://prometheus-community.github.io/helm-chartsTEST"]]]] - - def configFile = File.createTempFile("gop", '.yaml') - configFile.deleteOnExit() - - configFile.text = toYaml(fileConfig) - - cli.run("--config-file=${configFile}", "--yes") - - assertThat(cli.lastSchema.features.monitoring.helm.chart).isEqualTo('kube-prometheus-stack') - assertThat(cli.lastSchema.features.monitoring.helm.repoURL).isEqualTo('https://prometheus-community.github.io/helm-chartsTEST') - assertThat(cli.lastSchema.features.monitoring.helm.version).isEqualTo('80.2.2') - } - - @Test - void 'ensure helm defaults are used, if not set'() { - // this test sets only a few values for helm configuration and expect, that defaults are used. - - def fileConfig = [jenkins : [helm: [version: '5.8.1']], - scm : [scmManager: [helm: [values: [initialDelaySeconds: 120]]]], - features: [monitoring : [helm: [version : '66.2.1', - grafanaImage: 'localhost:30000/proxy/grafana:latest']], - secrets : [externalSecrets: [helm: [chart: 'my-secrets']], - vault : [helm: [repoURL: 'localhost:3000/proxy/vault:latest']],], - certManager: [helm: [image: 'localhost:30000/proxy/cert-manager-controller:latest']]]] - - def configFile = File.createTempFile("gop", ".yaml") - configFile.deleteOnExit() - - configFile.text = toYaml(fileConfig) - - cli.run("--config-file=${configFile}", "--yes") - def myconfig = cli.lastSchema; - assertThat(myconfig.jenkins.helm.chart).isEqualTo('jenkins') - assertThat(myconfig.jenkins.helm.repoURL).isEqualTo('https://charts.jenkins.io') - assertThat(myconfig.jenkins.helm.version).isEqualTo('5.8.1') // overridden - - assertThat(myconfig.scm.scmManager.helm.chart).isEqualTo('scm-manager') - assertThat(myconfig.scm.scmManager.helm.repoURL).isEqualTo('https://packages.scm-manager.org/repository/helm-v2-releases/') - assertThat(myconfig.scm.scmManager.helm.version).isEqualTo('3.11.6') - assertThat(myconfig.scm.scmManager.helm.values.initialDelaySeconds).isEqualTo(120) // overridden - - assertThat(cli.lastSchema.features.monitoring.helm.chart).isEqualTo('kube-prometheus-stack') - assertThat(cli.lastSchema.features.monitoring.helm.repoURL).isEqualTo('https://prometheus-community.github.io/helm-charts') - assertThat(cli.lastSchema.features.monitoring.helm.version).isEqualTo('66.2.1') - assertThat(cli.lastSchema.features.monitoring.helm.grafanaSidecarImage).isEqualTo('') - assertThat(cli.lastSchema.features.monitoring.helm.prometheusImage).isEqualTo('') - assertThat(cli.lastSchema.features.monitoring.helm.prometheusConfigReloaderImage).isEqualTo('') - assertThat(cli.lastSchema.features.monitoring.helm.prometheusOperatorImage).isEqualTo('') - assertThat(cli.lastSchema.features.monitoring.helm.grafanaImage).isEqualTo('localhost:30000/proxy/grafana:latest') - - assertThat(cli.lastSchema.features.secrets.externalSecrets.helm.chart).isEqualTo('my-secrets') - assertThat(cli.lastSchema.features.secrets.externalSecrets.helm.repoURL).isEqualTo('https://charts.external-secrets.io') - assertThat(cli.lastSchema.features.secrets.externalSecrets.helm.version).isEqualTo('0.9.16') - - assertThat(cli.lastSchema.features.secrets.vault.helm.chart).isEqualTo('vault') - assertThat(cli.lastSchema.features.secrets.vault.helm.repoURL).isEqualTo('localhost:3000/proxy/vault:latest') - assertThat(cli.lastSchema.features.secrets.vault.helm.version).isEqualTo('0.25.0') - - assertThat(cli.lastSchema.features.certManager.helm.chart).isEqualTo('cert-manager') - assertThat(cli.lastSchema.features.certManager.helm.repoURL).isEqualTo('https://charts.jetstack.io') - assertThat(cli.lastSchema.features.certManager.helm.version).isEqualTo('1.19.4') - assertThat(cli.lastSchema.features.certManager.helm.startupAPICheckImage).isEqualTo('') - assertThat(cli.lastSchema.features.certManager.helm.webhookImage).isEqualTo('') - assertThat(cli.lastSchema.features.certManager.helm.cainjectorImage).isEqualTo('') - assertThat(cli.lastSchema.features.certManager.helm.acmeSolverImage).isEqualTo('') - assertThat(cli.lastSchema.features.certManager.helm.image).isEqualTo('localhost:30000/proxy/cert-manager-controller:latest') - } - - static String getLoggingPattern() { - loggingEncoder.pattern - } - - static PatternLayoutEncoder getLoggingEncoder() { - LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory() - def rootLogger = loggerContext.getLogger(Logger.ROOT_LOGGER_NAME) - def consoleAppender = rootLogger.getAppender('STDOUT') as ConsoleAppender - consoleAppender.getEncoder() as PatternLayoutEncoder - } - - void writeViaSystemIn(String value) { - ByteArrayInputStream inContent = new ByteArrayInputStream("${value}\n".getBytes()) - System.setIn(inContent) - } - - static String toYaml(Map map) { - yamlMapper.writeValueAsString(map) - } - - class GitopsPlaygroundCliForTest extends GitopsPlaygroundCli { - ApplicationContext applicationContext = mock(ApplicationContext) - Config lastSchema = null - - GitopsPlaygroundCliForTest() { - super(GitopsPlaygroundCliTest.this.k8sClient, GitopsPlaygroundCliTest.this.applicationConfigurator) - - when(applicationConfigurator.initConfig(any(Config))).thenAnswer(new Answer() { - @Override - Config answer(InvocationOnMock invocation) throws Throwable { - lastSchema = invocation.getArgument(0) - return lastSchema - } - }) - - } - - @Override - protected ApplicationContext createApplicationContext() { - when(applicationContext.getBean(Application)).thenReturn(application) - when(applicationContext.getBean(Destroyer)).thenReturn(destroyer) - - return applicationContext - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/config/ConfigToConfigFileConverterTest.groovy b/src/test/groovy/com/cloudogu/gitops/config/ConfigToConfigFileConverterTest.groovy deleted file mode 100644 index e69de29bb..000000000 diff --git a/src/test/groovy/com/cloudogu/gitops/config/schema/ConfigTest.groovy b/src/test/groovy/com/cloudogu/gitops/config/schema/ConfigTest.groovy deleted file mode 100644 index 30c84afe2..000000000 --- a/src/test/groovy/com/cloudogu/gitops/config/schema/ConfigTest.groovy +++ /dev/null @@ -1,59 +0,0 @@ -package com.cloudogu.gitops.config.schema - -import static com.cloudogu.gitops.config.Config.* -import static org.assertj.core.api.Assertions.assertThat - -import com.cloudogu.gitops.config.Config - -import org.junit.jupiter.api.Test - -class ConfigTest { - Config testConfig = new Config(registry: new RegistrySchema(twoRegistries: true, - internalPort: 123)) - - @Test - void 'converts to yaml including internals'() { - String config = testConfig.toYaml(true) - - assertThat(config).startsWith("""--- -registry: - internal: true -""") - } - - @Test - void 'converts config map to yaml'() { - - String config = testConfig.toYaml(false) - - assertThat(config).startsWith("""--- -registry: - active: false -""") - } - - @Test - void 'creates from schema overwriting only Map values, ignoring null values'() { - Config expectedValues = new Config(application: new ApplicationSchema(// Overwrites a default String - username: 'myUser', - // Overwrites a default Boolean - yes: true, - // Sets an otherwise empty string - namePrefix: "aPrefix"), - // Overwrites a default Integer - registry: new RegistrySchema(internalPort: 42)) - - def actualValues = fromMap(expectedValues.toMap()) - - assertThat(actualValues.application.username).isEqualTo(expectedValues.application.username) - assertThat(actualValues.application.yes).isEqualTo(expectedValues.application.yes) - assertThat(actualValues.application.namePrefix).isEqualTo(expectedValues.application.namePrefix) - assertThat(actualValues.registry.internalPort).isEqualTo(expectedValues.registry.internalPort) - } - - @Test - void 'getting Tenantname from Config'() { - testConfig.application.namePrefix = 'testprefix-' - assertThat(testConfig.application.getTenantName()).isEqualTo("testprefix") - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/config/schema/JsonConfigValidatorTest.groovy b/src/test/groovy/com/cloudogu/gitops/config/schema/JsonConfigValidatorTest.groovy deleted file mode 100644 index 8a1433cbb..000000000 --- a/src/test/groovy/com/cloudogu/gitops/config/schema/JsonConfigValidatorTest.groovy +++ /dev/null @@ -1,51 +0,0 @@ -package com.cloudogu.gitops.config.schema - -import static groovy.test.GroovyAssert.shouldFail - -import java.util.stream.Stream - -import org.junit.jupiter.params.ParameterizedTest -import org.junit.jupiter.params.provider.Arguments -import org.junit.jupiter.params.provider.MethodSource - -class JsonConfigValidatorTest { - static Stream validSchemas() { - Stream.Builder ret = Stream.builder() - - ret.add(Arguments.of("multiple values", [features: [argocd: [url: "http://localhost/argocd"]]])) - - return ret.build() - } - - @ParameterizedTest(name = "{0}") - @MethodSource("validSchemas") - void 'test valid schemas'(String description, Map schema) { - - JsonSchemaValidator.validate(schema) - } - - static Stream invalidSchemas() { - Stream.Builder ret = Stream.builder() - - ret.add(Arguments.of("wrong type for registry.internalPort", [registry: [internalPort: "this should be a number"]])) - - ret.add(Arguments.of("invalid additional key within registry", [registry: [url : "", - unexpectedKey: "this should error"]])) - - ret.add(Arguments.of("invalid additional key on root level", [registry : [url: "",], - unexpectedKey: "this should not exist"])) - - ret.add(Arguments.of("specifying dynamic value", [application: [namePrefix : "prefix", - namePrefixForEnvVars: "prefix"],])) - - return ret.build() - } - - @ParameterizedTest(name = "{0}") - @MethodSource("invalidSchemas") - void 'test invalid schemas'(String description, Map schema) { - shouldFail(RuntimeException) { - JsonSchemaValidator.validate(schema) - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/config/schema/JsonSchemaGeneratorTest.groovy b/src/test/groovy/com/cloudogu/gitops/config/schema/JsonSchemaGeneratorTest.groovy deleted file mode 100644 index fae91d55f..000000000 --- a/src/test/groovy/com/cloudogu/gitops/config/schema/JsonSchemaGeneratorTest.groovy +++ /dev/null @@ -1,24 +0,0 @@ -package com.cloudogu.gitops.config.schema - -import static org.assertj.core.api.Assertions.assertThat - -import groovy.json.JsonOutput -import groovy.json.JsonSlurper - -import org.junit.jupiter.api.Test - -class JsonSchemaGeneratorTest { - @Test - void 'test configuration schema is not ouf of date'() { - // slurp and output to ensure consistent formatting - def slurper = new JsonSlurper() - def output = new JsonOutput() - - def expect = output.toJson(slurper.parseText(new JsonSchemaGenerator().createSchema().toString())) - def actual = output.toJson(slurper.parse(new File(System.getProperty("user.dir"), "docs/configuration.schema.json"))) - - assertThat(actual) - .as("Config in docs/configuration.schema.json must be updated. Run GenerateJsonSchema class.") - .isEqualTo(expect) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptorTest.groovy b/src/test/groovy/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptorTest.groovy deleted file mode 100644 index 547d22696..000000000 --- a/src/test/groovy/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptorTest.groovy +++ /dev/null @@ -1,161 +0,0 @@ -package com.cloudogu.gitops.dependencyinjection.okhttp - -import com.github.tomakehurst.wiremock.junit5.WireMockExtension -import okhttp3.OkHttpClient -import okhttp3.Request -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.RegisterExtension - -import javax.net.ssl.HostnameVerifier -import javax.net.ssl.SSLContext -import javax.net.ssl.TrustManager -import javax.net.ssl.X509TrustManager -import java.security.SecureRandom -import java.security.cert.X509Certificate -import java.util.concurrent.TimeUnit - -import static com.github.tomakehurst.wiremock.client.WireMock.* -import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig -import static org.assertj.core.api.Assertions.assertThat - -class RetryInterceptorTest { - - public static final int OKHTTPCLIENT_TIMEOUT = 1000 - - @RegisterExtension - static WireMockExtension wireMock = WireMockExtension.newInstance() - .options(wireMockConfig() - .dynamicPort() - .dynamicHttpsPort()) - .build() - - @BeforeEach - void 'resetWireMock'() { - wireMock.resetAll() - } - - @Test - void 'retries three times on 500'() { - def path = "/retry-500" - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("Retry Scenario") - .whenScenarioStateIs("Started") - .willReturn(aResponse().withStatus(500)) - .willSetStateTo("First Retry")) - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("Retry Scenario") - .whenScenarioStateIs("First Retry") - .willReturn(aResponse().withStatus(500)) - .willSetStateTo("Second Retry")) - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("Retry Scenario") - .whenScenarioStateIs("Second Retry") - .willReturn(aResponse() - .withStatus(200) - .withBody("Successful Result"))) - - def client = createClient() - def response = client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute() - - assertThat(response.body().string()).isEqualTo("Successful Result") - wireMock.verify(3, getRequestedFor(urlEqualTo(path))) - } - - @Test - void 'retries three times on 500 with HTTPS'() { - def path = "/retry-500" - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("HTTPS Retry Scenario") - .whenScenarioStateIs("Started") - .willReturn(aResponse().withStatus(500)) - .willSetStateTo("First Retry")) - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("HTTPS Retry Scenario") - .whenScenarioStateIs("First Retry") - .willReturn(aResponse().withStatus(500)) - .willSetStateTo("Second Retry")) - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("HTTPS Retry Scenario") - .whenScenarioStateIs("Second Retry") - .willReturn(aResponse() - .withStatus(200) - .withBody("Successful Result"))) - - def client = createClient() - def response = client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute() - - assertThat(response.body().string()).isEqualTo("Successful Result") - wireMock.verify(3, getRequestedFor(urlEqualTo(path))) - } - - @Test - void 'retries on timeout'() { - def path = "/timeout-test" - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("Timeout Scenario") - .whenScenarioStateIs("Started") - .willReturn(aResponse() - .withStatus(200) - .withFixedDelay(100)) // Delay longer than read timeout - .willSetStateTo("After Timeout")) - - wireMock.stubFor(get(urlEqualTo(path)) - .inScenario("Timeout Scenario") - .whenScenarioStateIs("After Timeout") - .willReturn(aResponse() - .withStatus(200) - .withBody("Successful Result"))) - - def client = createClient(100) - def response = client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute() - - assertThat(response.body().string()).isEqualTo("Successful Result") - wireMock.verify(2, getRequestedFor(urlEqualTo(path))) - } - - @Test - void 'fails after third retry'() { - def path = "/always-fail" - - wireMock.stubFor(get(urlEqualTo(path)) - .willReturn(aResponse().withStatus(500))) - - def client = createClient() - def response = client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute() - - assertThat(response.code()).isEqualTo(500) - wireMock.verify(4, getRequestedFor(urlEqualTo(path))) // Initial request + 3 retries - } - - private OkHttpClient createClient(int timeout = OKHTTPCLIENT_TIMEOUT) { - // 1. Create a TrustManager that trusts everyone - def trustAllCerts = [new X509TrustManager() { - void checkClientTrusted(X509Certificate[] chain, String authType) {} - - void checkServerTrusted(X509Certificate[] chain, String authType) {} - - X509Certificate[] getAcceptedIssuers() { - return new X509Certificate[0] - } - }] as TrustManager[] - - def sslContext = SSLContext.getInstance("TLS") - sslContext.init(null, trustAllCerts, new SecureRandom()) - - new OkHttpClient.Builder() - .addInterceptor(new RetryInterceptor(retries: 3, waitPeriodInMs: 0)) - .connectTimeout(timeout, TimeUnit.MILLISECONDS) - .readTimeout(timeout, TimeUnit.MILLISECONDS) - .sslSocketFactory(sslContext.socketFactory, trustAllCerts[0] as X509TrustManager) - .hostnameVerifier({ hostname, session -> true } as HostnameVerifier) - .build() - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/destroy/DestroyerDependencyInjectionTest.groovy b/src/test/groovy/com/cloudogu/gitops/destroy/DestroyerDependencyInjectionTest.groovy deleted file mode 100644 index e60b98eb0..000000000 --- a/src/test/groovy/com/cloudogu/gitops/destroy/DestroyerDependencyInjectionTest.groovy +++ /dev/null @@ -1,25 +0,0 @@ -package com.cloudogu.gitops.destroy - -import com.cloudogu.gitops.config.Config - -import io.micronaut.context.ApplicationContext - -import org.assertj.core.api.Assertions -import org.junit.jupiter.api.Test - -class DestroyerDependencyInjectionTest { - @Test - void 'can create bean'() { - def destroyer = ApplicationContext.run() - .registerSingleton(Config.fromMap([scm : [scmManager: [url : 'http://localhost:9091/scm', - username: 'admin', - password: 'admin']], - jenkins : [url : 'http://localhost:9090', - username: 'admin', - password: 'admin',], - application: [insecure: true]])) - .getBean(Destroyer) - - Assertions.assertThat(destroyer.destructionHandlers).hasSize(3) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategyTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategyTest.groovy deleted file mode 100644 index 27fd09e49..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategyTest.groovy +++ /dev/null @@ -1,131 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.ScmTenantSchema -import com.cloudogu.gitops.config.scm.ScmTenantSchema.ScmManagerTenantConfig -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.yaml.YamlSlurper -import org.junit.jupiter.api.Test - -import static org.assertj.core.api.Assertions.assertThat - -class ArgoCdApplicationStrategyTest { - private File localTempDir - GitHandler gitHandler = new GitHandlerForTests(new Config(), new ScmManagerMock()) - - @Test - void 'deploys feature using argo CD'() { - def strategy = createStrategy() - File valuesYaml = File.createTempFile('values', 'yaml') - strategy.deployFeature("repoURL", "repoName", "chartName", "version", - "foo-namespace", "releaseName", valuesYaml.toPath()) - - def argoCdApplicationYaml = new File("$localTempDir/apps/argocd/applications/releaseName.yaml") - assertThat(argoCdApplicationYaml.text).isEqualTo("""--- -apiVersion: "argoproj.io/v1alpha1" -kind: "Application" -metadata: - name: "repoName" - namespace: "foo-argocd" -spec: - destination: - server: "https://kubernetes.default.svc" - namespace: "foo-namespace" - project: "cluster-resources" - sources: - - repoURL: "repoURL" - chart: "chartName" - targetRevision: "version" - helm: - releaseName: "releaseName" - valueFiles: - - "\$values/apps/repoName/repoName-gop-helm.yaml" - - "\$values/apps/repoName/repoName-user-values.yaml" - ignoreMissingValueFiles: true - - repoURL: "http://scmm.scm-manager.svc.cluster.local/scm/repo/argocd/cluster-resources.git" - targetRevision: "main" - ref: "values" - path: "apps/repoName" - directory: - recurse: true - syncPolicy: - automated: - prune: true - selfHeal: true - syncOptions: - - "ServerSideApply=true" - - "CreateNamespace=true" -""") - } - - @Test - void 'deploys feature using argo CD from git repo'() { - def strategy = createStrategy() - File valuesYaml = File.createTempFile('values', 'yaml') - strategy.deployFeature("repoURL", "repoName", "chartName", "version", - "namespace", "releaseName", valuesYaml.toPath(), DeploymentStrategy.RepoType.GIT) - - def argoCdApplicationYaml = new File("$localTempDir/apps/argocd/applications/releaseName.yaml") - def result = new YamlSlurper().parse(argoCdApplicationYaml) - def sources = result['spec']['sources'] as List - assertThat(sources[0] as Map).containsKey('path') - assertThat(sources[0]['path']).isEqualTo('chartName') - } - - @Test - void 'deploys feature with argocdOperator true, setting CreateNamespace to false'() { - def strategy = createStrategy(true) - File valuesYaml = File.createTempFile('values', 'yaml') - valuesYaml.text = """ - param1: value1 - param2: value2 - """ - strategy.deployFeature("repoURL", "repoName", "chartName", "version", - "namespace", "releaseName", valuesYaml.toPath()) - - def argoCdApplicationYaml = new File("$localTempDir/apps/argocd/applications/releaseName.yaml") - assertThat(argoCdApplicationYaml.text).contains("CreateNamespace=false") - } - - @Test - void 'deploys feature with argocdOperator false, setting CreateNamespace to true'() { - def strategy = createStrategy(false) - File valuesYaml = File.createTempFile('values', 'yaml') - valuesYaml.text = """ - param1: value1 - param2: value2 - """ - strategy.deployFeature("repoURL", "repoName", "chartName", "version", - "namespace", "releaseName", valuesYaml.toPath()) - - def argoCdApplicationYaml = new File("$localTempDir/apps/argocd/applications/releaseName.yaml") - assertThat(argoCdApplicationYaml.text).contains("CreateNamespace=true") - } - - private ArgoCdApplicationStrategy createStrategy(boolean argocdOperator = false) { - Config config = new Config(application: new Config.ApplicationSchema(namePrefix: 'foo-', - gitName: 'Cloudogu', - gitEmail: 'hello@cloudogu.com'), - scm: new ScmTenantSchema(scmManager: new ScmManagerTenantConfig(username: "dont-care-username", - password: "dont-care-password")), - features: new Config.FeaturesSchema(argocd: new Config.ArgoCDSchema(operator: argocdOperator))) - - def repoProvider = new TestGitRepoFactory(config, new FileSystemUtils()) { - @Override - GitRepo getRepo(String repoTarget, GitProvider gitProvider) { - def repo = super.getRepo(repoTarget, gitProvider) - localTempDir = new File(repo.getAbsoluteLocalRepoTmpDir()) - - return repo - } - } - - return new ArgoCdApplicationStrategy(config, new FileSystemUtils(), repoProvider, gitHandler) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/DeployerTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/DeployerTest.groovy deleted file mode 100644 index 4de3660d0..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/DeployerTest.groovy +++ /dev/null @@ -1,43 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import com.cloudogu.gitops.config.Config -import org.junit.jupiter.api.Test - -import java.nio.file.Path - -import static org.mockito.ArgumentMatchers.any -import static org.mockito.ArgumentMatchers.anyString -import static org.mockito.Mockito.* - -class DeployerTest { - private ArgoCdApplicationStrategy argoCdStrat = mock(ArgoCdApplicationStrategy.class) - private HelmStrategy helmStrat = mock(HelmStrategy.class) - - @Test - void 'When argocd disabled, deploys imperatively via helm'() { - def deployer = createDeployer(false) - - deployer.deployFeature("repoURL", "repoName", "chart", "version", "namespace", "releaseName", Path.of("values.yaml")) - - verify(argoCdStrat, never()).deployFeature(anyString(), anyString(), anyString(), anyString(), anyString(), anyString(), any(Path)) - verify(helmStrat).deployFeature("repoURL", "repoName", "chart", "version", "namespace", - "releaseName", Path.of("values.yaml"), DeploymentStrategy.RepoType.HELM) - } - - @Test - void 'When Argo CD enabled, deploys natively via Argo CD'() { - def deployer = createDeployer(true) - - deployer.deployFeature("repoURL", "repoName", "chart", "version", "namespace", "releaseName", Path.of("values.yaml")) - - verify(argoCdStrat).deployFeature("repoURL", "repoName", "chart", "version", "namespace", - "releaseName", Path.of("values.yaml"), DeploymentStrategy.RepoType.HELM) - verify(helmStrat, never()).deployFeature(anyString(), anyString(), anyString(), anyString(), anyString(), anyString(), any(Path)) - } - - private Deployer createDeployer(boolean argoCDActive) { - Config config = new Config(features: new Config.FeaturesSchema(argocd: new Config.ArgoCDSchema(active: argoCDActive))) - - return new Deployer(config, argoCdStrat, helmStrat) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/HelmStrategyTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/HelmStrategyTest.groovy deleted file mode 100644 index 2b47ceb90..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/deployment/HelmStrategyTest.groovy +++ /dev/null @@ -1,45 +0,0 @@ -package com.cloudogu.gitops.infrastructure.deployment - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import org.junit.jupiter.api.Test - -import java.nio.file.Files -import java.nio.file.Path - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.Mockito.mock -import static org.mockito.Mockito.verify - -class HelmStrategyTest { - - HelmClient helmClient = mock(HelmClient) - - @Test - void 'deploys feature using helm client'() { - Path valuesYaml = Files.createTempFile('', '') - - createStrategy().deployFeature("repoURL", "repoName", "chart", "version", "foo-namespace", "releaseName", valuesYaml) - - verify(helmClient).addRepo("repoName", "repoURL") - verify(helmClient).upgrade("releaseName", "repoName/chart", [namespace: "foo-namespace", - version : "version", - values : valuesYaml.toString()]) - } - - @Test - void 'Fails to deploy from git'() { - def exception = shouldFail(RuntimeException) { - createStrategy().deployFeature("http://repoURL", "repoName", "chart", "version", "namespace", - "releaseName", Path.of("values.yaml"), DeploymentStrategy.RepoType.GIT) - } - assertThat(exception.message).isEqualTo("Unable to deploy helm chart via Helm CLI from Git URL, because helm does not support this out of the box.\n" + - "Repo URL: http://repoURL") - - } - - protected HelmStrategy createStrategy() { - new HelmStrategy(new Config(application: new Config.ApplicationSchema(namePrefix: "foo-")), helmClient) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/GitRepoTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/git/GitRepoTest.groovy deleted file mode 100644 index 62c509d99..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/GitRepoTest.groovy +++ /dev/null @@ -1,206 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.Scope -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.utils.FileSystemUtils -import org.eclipse.jgit.api.Git -import org.eclipse.jgit.lib.Ref -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.mockito.Mock - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat - -class GitRepoTest { - - public static final String expectedNamespace = "namespace" - public static final String expectedRepo = "repo" - Config config = Config.fromMap([application: [gitName : "Cloudogu", - gitEmail: "hello@cloudogu.com"], - scm : [scmManager: [username: "dont-care-username", - password: "dont-care-password"]]]) - - TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, new FileSystemUtils()) - - @Mock - GitProvider gitProvider - - ScmManagerMock scmManagerMock - - @BeforeEach - void setup() { - scmManagerMock = new ScmManagerMock() - } - - @Test - void "writes file"() { - def repo = getRepo("", scmManagerMock) - repo.writeFile("test.txt", "the file's content") - - def expectedFile = new File("$repo.absoluteLocalRepoTmpDir/test.txt") - assertThat(expectedFile.getText()).is("the file's content") - } - - @Test - void "overwrites file"() { - def repo = getRepo("", scmManagerMock) - def tempDir = repo.absoluteLocalRepoTmpDir - - def existingFile = new File("$tempDir/already-exists.txt") - existingFile.createNewFile() - existingFile.text = "already existing content" - - repo.writeFile("already-exists.txt", "overwritten content") - - def expectedFile = new File("$tempDir/already-exists.txt") - assertThat(expectedFile.getText()).is("overwritten content") - } - - @Test - void "writes file and creates subdirectory"() { - def repo = getRepo("", scmManagerMock) - def tempDir = repo.absoluteLocalRepoTmpDir - repo.writeFile("subdirectory/test.txt", "the file's content") - - def expectedFile = new File("$tempDir/subdirectory/test.txt") - assertThat(expectedFile.getText()).is("the file's content") - } - - @Test - void "throws error when directory conflicts with existing file"() { - def repo = getRepo("", scmManagerMock) - def tempDir = repo.absoluteLocalRepoTmpDir - new File("$tempDir/test.txt").mkdir() - - shouldFail(FileNotFoundException) { - repo.writeFile("test.txt", "the file's content") - } - } - - @Test - void 'Creates repo with empty name-prefix'() { - def repo = getRepo('expectedRepoTarget', scmManagerMock) - assertThat(repo.repoTarget).isEqualTo('expectedRepoTarget') - } - - @Test - void 'Creates repo with name-prefix'() { - config.application.namePrefix = 'abc-' - def repo = getRepo('expectedRepoTarget', scmManagerMock) - assertThat(repo.repoTarget).isEqualTo('abc-expectedRepoTarget') - } - - @Test - void 'Creates repo with name-prefix when in namespace 3rd-party-deps'() { - config.application.namePrefix = 'abc-' - def repo = getRepo("${GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES}/foo", scmManagerMock) - assertThat(repo.repoTarget).isEqualTo("${config.application.namePrefix}${GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES}/foo".toString()) - } - - @Test - void 'Clones and checks out main'() { - def repo = getRepo("", scmManagerMock) - - repo.cloneRepo() - def HEAD = new File(repo.absoluteLocalRepoTmpDir, '.git/HEAD') - assertThat(HEAD.text).isEqualTo("ref: refs/heads/main\n") - assertThat(new File(repo.absoluteLocalRepoTmpDir, 'README.md')).exists() - } - - @Test - void 'pushes changes to remote directory'() { - def repo = getRepo("", scmManagerMock) - - repo.cloneRepo() - def readme = new File(repo.absoluteLocalRepoTmpDir, 'README.md') - readme.text = 'This text should be in the readme afterwards' - repo.commitAndPush("The commit message") - - def commits = Git.open(new File(repo.absoluteLocalRepoTmpDir)).log().setMaxCount(1).all().call().collect() - assertThat(commits.size()).isEqualTo(1) - assertThat(commits[0].fullMessage).isEqualTo("The commit message") - assertThat(commits[0].authorIdent.emailAddress).isEqualTo('hello@cloudogu.com') - assertThat(commits[0].authorIdent.name).isEqualTo('Cloudogu') - assertThat(commits[0].committerIdent.emailAddress).isEqualTo('hello@cloudogu.com') - assertThat(commits[0].committerIdent.name).contains("Cloudogu - GOP v") - - List tags = Git.open(new File(repo.absoluteLocalRepoTmpDir)).tagList().call() - assertThat(tags.size()).isEqualTo(0) - } - - @Test - void 'pushes changes to remote directory with tag'() { - def repo = getRepo("", scmManagerMock) - def expectedTag = '1.0' - - repo.cloneRepo() - def readme = new File(repo.absoluteLocalRepoTmpDir, 'README.md') - readme.text = 'This text should be in the readme afterwards' - // Create existing tag to test for idempotence - Git.open(new File(repo.absoluteLocalRepoTmpDir)).tag().setName(expectedTag).call() - - repo.commitAndPush("The commit message", expectedTag) - - List tags = Git.open(new File(repo.absoluteLocalRepoTmpDir)).tagList().call() - assertThat(tags.size()).isEqualTo(1) - assertThat(tags[0].name).isEqualTo("refs/tags/$expectedTag".toString()) - // It would be a good idea to check if the git tag is set on the commit. - // However, it's extremely complicated with jgit - // The "official" example code throws an exception here: Ref peeledRef = repository.getRefDatabase().peel(ref) - // https://github.com/centic9/jgit-cookbook/blob/d923e18b2ce2e55761858fd2e8e402dd252e0766/src/main/java/org/dstadler/jgit/porcelain/ListTags.java - // 🤷 - } - - @Test - void 'creates repository and sets permission when new and username present'() { - - def repoTarget = "foo/bar" - def repo = getRepo(repoTarget, scmManagerMock) - scmManagerMock.nextCreateResults = [true] // simulate "new repo" - scmManagerMock.gitOpsUsername = 'foo-gitops' // username available - - def created = repo.createRepositoryAndSetPermission('testdescription', true) - - assertThat(created).isTrue() - - // Verify that repo was created - assertThat(scmManagerMock.createdRepos).containsExactly(repoTarget) - - // Verify permission call - assertThat(scmManagerMock.permissionCalls).hasSize(1) - def call = scmManagerMock.permissionCalls[0] - assertThat(call.repoTarget).isEqualTo(repoTarget) - assertThat(call.principal).isEqualTo('foo-gitops') - assertThat(call.role).isEqualTo(AccessRole.WRITE) - assertThat(call.scope).isEqualTo(Scope.USER) - } - - @Test - void 'does not set permission when no GitOps username is configured'() { - def repoTarget = "foo/bar" - def scmManagerMock = new ScmManagerMock() - def repo = getRepo(repoTarget, scmManagerMock) - - scmManagerMock.nextCreateResults = [true] // repo is new - scmManagerMock.gitOpsUsername = null // no username - - def created = repo.createRepositoryAndSetPermission('desc', true) - - assertThat(created).isTrue() - - // Repo created - assertThat(scmManagerMock.createdRepos).containsExactly(repoTarget) - - // No permission calls because username missing - assertThat(scmManagerMock.permissionCalls).isEmpty() - } - - private GitRepo getRepo(String repoTarget = "${expectedNamespace}/${expectedRepo}", ScmManagerMock scmManagerMock) { - return repoProvider.getRepo(repoTarget, scmManagerMock) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerTest.groovy deleted file mode 100644 index 0a0bae13c..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerTest.groovy +++ /dev/null @@ -1,166 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.config.scm.util.ScmManagerConfig -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope -import com.cloudogu.gitops.infrastructure.git.providers.Scope -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.RepositoryApi -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.utils.NetworkingUtils -import okhttp3.internal.http.RealResponseBody -import okio.BufferedSource -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.ExtendWith -import org.junit.jupiter.api.function.Executable -import org.mockito.Mock -import org.mockito.junit.jupiter.MockitoExtension -import retrofit2.Call -import retrofit2.Response - -import static org.junit.jupiter.api.Assertions.* -import static org.mockito.ArgumentMatchers.* -import static org.mockito.Mockito.* - -@ExtendWith(MockitoExtension) -class ScmManagerTest { - - private Config config - - @Mock - ScmManagerConfig scmmCfg - @Mock - K8sClient k8s - @Mock - NetworkingUtils net - @Mock - ScmManagerUrlResolver urls - @Mock - ScmManagerApiClient apiClient - @Mock - RepositoryApi repoApi - - @BeforeEach - void setup() { - config = new Config(application: new Config.ApplicationSchema(insecure: false, - namePrefix: "fv40-", - runningInsideK8s: true)) - - lenient().when(scmmCfg.getCredentials()).thenReturn(new Credentials("user", "password")) - lenient().when(scmmCfg.getGitOpsUsername()).thenReturn("gitops-bot") - - lenient().when(urls.inClusterBase()).thenReturn(new URI("http://scmm.ns.svc.cluster.local/scm")) - lenient().when(urls.inClusterRepoPrefix()).thenReturn("http://scmm.ns.svc.cluster.local/scm/repo/fv40-") - lenient().when(urls.clientApiBase()).thenReturn(new URI("http://nodeport/scm/api/v2/")) - - lenient().when(apiClient.repositoryApi()).thenReturn(repoApi) - } - - private ScmManager newSchManager() { - return new ScmManager(config, scmmCfg, urls, apiClient) - } - - private static Call callReturningSuccess(int code) { - def call = mock(Call) - when(call.execute()).thenReturn(Response.success(code, null)) - call - } - - private static Call callReturningError(int code) { - def call = mock(Call) - def body = new RealResponseBody('ignored', 0, mock(BufferedSource)) - when(call.execute()).thenReturn(Response.error(code, body)) - call - } - - @Test - void 'createRepository returns true on 201 and false on subsequent 409 for the same repo'() { - def scmManager = newSchManager() - - def created = callReturningSuccess(201) - def conflict = callReturningError(409) - def seen = new HashSet() - - when(repoApi.create(any(Repository), anyBoolean())) - .thenAnswer(inv -> { - Repository r = inv.getArgument(0) - if (seen.contains(r.fullRepoName)) return conflict - seen.add(r.fullRepoName) - return created - }) - - assertTrue(scmManager.createRepository("team/demo", "Demo repo", true)) - assertFalse(scmManager.createRepository("team/demo", "Demo repo", true)) // 409 - assertTrue(scmManager.createRepository("team/other", null, false)) // neuer Name -> 201 - - verify(repoApi, times(3)).create(any(Repository), anyBoolean()) - } - - @Test - void 'setRepositoryPermission maps MAINTAIN to WRITE and handles 201 409'() { - def scmManager = newSchManager() - - def created = callReturningSuccess(201) - def conflict = callReturningError(409) - def seen = new HashSet() - // key: ns/name - - when(repoApi.createPermission(anyString(), anyString(), any(Permission))) - .thenAnswer(inv -> { - String namespace = inv.getArgument(0) - String repoName = inv.getArgument(1) - String key = namespace + "/" + repoName - if (seen.contains(key)) return conflict - seen.add(key) - return created - }) - - assertDoesNotThrow({ -> scmManager.setRepositoryPermission("namespace/repo1", "devs", AccessRole.MAINTAIN, Scope.GROUP) - } as Executable) - - assertDoesNotThrow({ -> scmManager.setRepositoryPermission("namespace/repo1", "devs", AccessRole.MAINTAIN, Scope.GROUP) - } as Executable) - verify(repoApi, atLeastOnce()) - .createPermission(eq("namespace"), eq("repo1"), argThat { Permission p -> p.groupPermission && p.role == Permission.Role.WRITE }) - } - - @Test - void 'url, repoPrefix, repoUrl variants, protocol and host come from UrlResolver'() { - when(urls.inClusterRepoUrl(anyString())).thenAnswer(a -> "http://scmm.ns.svc.cluster.local/scm/repo/" + a.getArgument(0)) - when(urls.clientRepoUrl(anyString())).thenAnswer(a -> "http://nodeport/scm/repo/" + a.getArgument(0)) - - def scmManager = newSchManager() - - assertEquals("http://scmm.ns.svc.cluster.local/scm", scmManager.url) - assertEquals("http://scmm.ns.svc.cluster.local/scm/repo/fv40-", scmManager.repoPrefix()) - - assertEquals("http://scmm.ns.svc.cluster.local/scm/repo/team/app", - scmManager.repoUrl("team/app", RepoUrlScope.IN_CLUSTER)) - assertEquals("http://nodeport/scm/repo/team/app", - scmManager.repoUrl("team/app", RepoUrlScope.CLIENT)) - - assertEquals("http", scmManager.protocol) - assertEquals("scmm.ns.svc.cluster.local", scmManager.host) - } - - @Test - void 'prometheusMetricsEndpoint is delegated to UrlResolver'() { - when(urls.prometheusEndpoint()).thenReturn(new URI("http://nodeport/scm/api/v2/metrics/prometheus")) - def scmManager = newSchManager() - assertEquals(new URI("http://nodeport/scm/api/v2/metrics/prometheus"), scmManager.prometheusMetricsEndpoint()) - } - - // Credentials & GitOps-User - @Test - void 'credentials and gitOpsUsername come from ScmManagerConfig'() { - def scmManager = newSchManager() - assertEquals("user", scmManager.credentials.username) - assertEquals("password", scmManager.credentials.password) - assertEquals("gitops-bot", scmManager.gitOpsUsername) - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolverTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolverTest.groovy deleted file mode 100644 index 14343630d..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolverTest.groovy +++ /dev/null @@ -1,156 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.ScmTenantSchema -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.utils.NetworkingUtils -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.ExtendWith -import org.mockito.Mock -import org.mockito.junit.jupiter.MockitoExtension - -import static org.junit.jupiter.api.Assertions.* -import static org.mockito.ArgumentMatchers.any -import static org.mockito.ArgumentMatchers.eq -import static org.mockito.Mockito.* - -@ExtendWith(MockitoExtension.class) -class ScmManagerUrlResolverTest { - private Config config - - @Mock - private K8sClient k8s - @Mock - private NetworkingUtils net - - @BeforeEach - void setUp() { - config = new Config(application: new Config.ApplicationSchema(namePrefix: 'fv40-', - runningInsideK8s: false)) - } - - private ScmManagerUrlResolver resolverWith(Map args = [:]) { - def scmmCofig = new ScmTenantSchema.ScmManagerTenantConfig() - scmmCofig.internal = (args.containsKey('internal') ? args.internal : true) - scmmCofig.namespace = (args.containsKey('namespace') ? args.namespace : "scm-manager") - scmmCofig.url = (args.containsKey('url') ? args.url : "") - scmmCofig.ingress = (args.containsKey('ingress') ? args.ingress : "") - - return new ScmManagerUrlResolver(config, scmmCofig, k8s, net) - } - - // ---------- Client base & API ---------- - @Test - void "clientBase(): internal + outside K8s uses NodePort and appends 'scm' (no trailing slash) and only resolves NodePort once"() { - when(k8s.waitForNodePort(eq('scmm'), any())).thenReturn("30080") - when(net.findClusterBindAddress()).thenReturn("10.0.0.1") - - def r = resolverWith() - URI base1 = r.clientBase() - URI base2 = r.clientBase() - - assertEquals("http://10.0.0.1:30080/scm", base1.toString()) - assertEquals(base1, base2) - - verify(k8s, times(1)).waitForNodePort("scmm", "scm-manager") - verify(net, times(1)).findClusterBindAddress() - verifyNoMoreInteractions(k8s, net) - } - - @Test - void "clientApiBase(): appends 'api' to the client base"() { - when(k8s.waitForNodePort("scmm", "scm-manager")).thenReturn("30080") - when(net.findClusterBindAddress()).thenReturn("10.0.0.1") - - var urlResolver = resolverWith() - assertEquals("http://10.0.0.1:30080/scm/api/", urlResolver.clientApiBase().toString()) - } - - // ---------- Repo base & URLs ---------- - @Test - void "clientRepoUrl(): trims repoTarget and removes trailing slash"() { - when(k8s.waitForNodePort("scmm", "scm-manager")).thenReturn("30080") - when(net.findClusterBindAddress()).thenReturn("10.0.0.1") - - var urlResolver = resolverWith() - assertEquals("http://10.0.0.1:30080/scm/repo/ns/project", - urlResolver.clientRepoUrl(" ns/project ")) - } - - // ---------- In-cluster base & URLs ---------- - @Test - void "inClusterBase(): internal uses service DNS "() { - def r = resolverWith(namespace: "custom-ns", internal: true) - assertEquals("http://scmm.custom-ns.svc.cluster.local/scm", r.inClusterBase().toString()) - } - - @Test - void "inClusterBase(): external uses external base + 'scm'"() { - var r = resolverWith(internal: false, url: "https://scmm.external") - assertEquals("https://scmm.external/scm", r.inClusterBase().toString()) - } - - @Test - void "inClusterRepoUrl(): builds full in-cluster repo URL without trailing slash"() { - var urlResolver = resolverWith() - assertEquals("http://scmm.scm-manager.svc.cluster.local/scm/repo/admin/admin", - urlResolver.inClusterRepoUrl("admin/admin")) - } - - @Test - void "inClusterRepoPrefix(): includes configured namePrefix (empty prefix yields base path)"() { - // with non-empty namePrefix - config.application.namePrefix = 'fv40-' - def r1 = resolverWith() - assertEquals('http://scmm.scm-manager.svc.cluster.local/scm/repo/fv40-', r1.inClusterRepoPrefix()) - - // with empty/blank namePrefix - config.application.namePrefix = ' ' - def r2 = resolverWith() - assertEquals('http://scmm.scm-manager.svc.cluster.local/scm/repo/', r2.inClusterRepoPrefix()) - } - - // ---------- externalBase selection & error ---------- - @Test - void "externalBase(): prefers 'url' over 'ingress'"() { - def r = resolverWith(internal: false, url: 'https://scmm.external', ingress: 'ingress.example.org') - assertEquals('https://scmm.external/scm', r.inClusterBase().toString()) - } - - @Test - void "externalBase(): uses 'ingress' when 'url' is missing"() { - def r = resolverWith(internal: false, url: null, ingress: 'ingress.example.org') - assertEquals('http://ingress.example.org/scm', r.inClusterBase().toString()) - } - - @Test - void "externalBase(): throws when neither 'url' nor 'ingress' is set"() { - def r = resolverWith(internal: false, url: null, ingress: null) - def ex = assertThrows(IllegalArgumentException) { r.inClusterBase() } - assertTrue(ex.message.contains('Either scmm.url or scmm.ingress must be set when internal=false')) - } - - @Test - void "nodePortBase(): falls back to default namespace 'scm-manager' when none provided"() { - when(k8s.waitForNodePort(eq('scmm'), eq('scm-manager'))).thenReturn("30080") - when(net.findClusterBindAddress()).thenReturn('10.0.0.1') - - def r = resolverWith(namespace: null) - assertEquals('http://10.0.0.1:30080/scm', r.clientBase().toString()) - } - - // ---------- helpers behavior ---------- - @Test - void "ensureScm(): adds 'scm' if missing and keeps it if present"() { - def r1 = resolverWith(internal: false, url: 'https://scmm.localhost') - assertEquals('https://scmm.localhost/scm', r1.clientBase().toString()) - } - - // ---------- prometheus endpoint ---------- - @Test - void "prometheusEndpoint(): resolves "() { - def r = resolverWith(internal: false, url: 'https://scmm.localhost') - assertEquals('https://scmm.localhost/scm/api/v2/metrics/prometheus', r.prometheusEndpoint().toString()) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApiTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApiTest.groovy deleted file mode 100644 index 3c48d5a5e..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApiTest.groovy +++ /dev/null @@ -1,68 +0,0 @@ -package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api - -import com.cloudogu.gitops.config.Credentials -import com.github.tomakehurst.wiremock.junit5.WireMockExtension -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.RegisterExtension - -import javax.net.ssl.SSLHandshakeException - -import static com.github.tomakehurst.wiremock.client.WireMock.* -import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat - -class UsersApiTest { - - @RegisterExtension - static WireMockExtension wireMock = WireMockExtension.newInstance() - .options(wireMockConfig() - .dynamicPort() - .dynamicHttpsPort()) - .build() - - private Credentials credentials = new Credentials("user", "pass") - - @Test - void 'allows self-signed certificates when using insecure option'() { - wireMock.stubFor(delete(urlPathEqualTo("/scm/api/v2/users/test-user")) - .willReturn(aResponse().withStatus(204))) - - def api = usersApi(true, true) - // insecure=true, useHttps=true - def resp = api.delete('test-user').execute() - - assertThat(resp.isSuccessful()).isTrue() - wireMock.verify(1, deleteRequestedFor(urlPathEqualTo("/scm/api/v2/users/test-user"))) - } - - @Test - void 'does not allow self-signed certificates by default'() { - wireMock.stubFor(delete(urlPathEqualTo("/scm/api/v2/users/test-user")) - .willReturn(aResponse().withStatus(204))) - - def api = usersApi(false, true) - // insecure=false, useHttps=true - - shouldFail(SSLHandshakeException) { - api.delete('test-user').execute() - } - - wireMock.verify(0, deleteRequestedFor(urlPathEqualTo("/scm/api/v2/users/test-user"))) - } - - private UsersApi usersApi(boolean insecure, boolean useHttps = false) { - def client = new ScmManagerApiClient(apiBaseUrl(useHttps), credentials, insecure) - return client.usersApi() - } - - private String apiBaseUrl(boolean useHttps) { - if (useHttps) { - // Use the proper HTTPS port from WireMock - def httpsPort = wireMock.httpsPort - return "https://localhost:${httpsPort}/scm/api/" - } else { - return "${wireMock.baseUrl()}/scm/api/" - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManagerTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManagerTest.groovy deleted file mode 100644 index de3f98b4e..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManagerTest.groovy +++ /dev/null @@ -1,85 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import org.junit.jupiter.api.Test - -import static groovy.test.GroovyAssert.shouldFail -import static org.mockito.ArgumentMatchers.anyString -import static org.mockito.Mockito.* - -class GlobalPropertyManagerTest { - @Test - void 'sets global property'() { - def client = mock(JenkinsApiClient) - def propertyManager = new GlobalPropertyManager(client) - - when(client.runScript(anyString())).thenReturn("Done") - propertyManager.setGlobalProperty('the-key', 'the-value') - - verify(client).runScript(""" - instance = Jenkins.getInstance() - globalNodeProperties = instance.getGlobalNodeProperties() - envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) - - def newEnvVarsNodeProperty - def envVars - - if ( envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0 ) { - newEnvVarsNodeProperty = new hudson.slaves.EnvironmentVariablesNodeProperty() - globalNodeProperties.add(newEnvVarsNodeProperty) - envVars = newEnvVarsNodeProperty.getEnvVars() - } else { - envVars = envVarsNodePropertyList.get(0).getEnvVars() - - } - - envVars.put("the-key", "the-value") - - instance.save() - print("Done") - """) - } - - @Test - void 'throws when there was an error when creating global property'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]") - - shouldFail(RuntimeException) { - new GlobalPropertyManager(client).setGlobalProperty("the-key", "the-value") - } - } - - @Test - void 'deletes global property'() { - def client = mock(JenkinsApiClient) - def propertyManager = new GlobalPropertyManager(client) - - when(client.runScript(anyString())).thenReturn("Nothing to do") - propertyManager.deleteGlobalProperty('the-key') - - verify(client).runScript(""" - def instance = Jenkins.getInstance() - def globalNodeProperties = instance.getGlobalNodeProperties() - def envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) - - if (envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0) { - print("Nothing to do") - return - } - - envVars = envVarsNodePropertyList.get(0).getEnvVars() - envVars.remove("the-key") - print("Done") - """) - } - - @Test - void 'throws when there was an error when deleting global property'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]") - - shouldFail(RuntimeException) { - new GlobalPropertyManager(client).deleteGlobalProperty("the-key") - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClientTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClientTest.groovy deleted file mode 100644 index c045ea2ad..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClientTest.groovy +++ /dev/null @@ -1,266 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import com.cloudogu.gitops.config.Config -import com.github.tomakehurst.wiremock.junit5.WireMockExtension -import io.micronaut.context.ApplicationContext -import okhttp3.FormBody -import okhttp3.JavaNetCookieJar -import okhttp3.OkHttpClient -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.RegisterExtension - -import javax.net.ssl.SSLContext -import javax.net.ssl.SSLSocketFactory -import javax.net.ssl.TrustManager -import javax.net.ssl.X509TrustManager -import java.security.SecureRandom -import java.security.cert.X509Certificate - -import static com.github.tomakehurst.wiremock.client.WireMock.* -import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat - -class JenkinsApiClientTest { - - @RegisterExtension - static WireMockExtension wireMock = WireMockExtension.newInstance() - .options(wireMockConfig() - .dynamicPort() - .dynamicHttpsPort()) - .build() - - @Test - void 'runs script with crumb'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-crumb", "crumbRequestField": "Jenkins-Crumb"}'))) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) - .willReturn(aResponse() - .withStatus(200) - .withBody("ok"))) - - def httpClient = getUnsafeOkHttpClient().newBuilder().cookieJar(new JavaNetCookieJar(new CookieManager())).build() - def apiClient = new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl()}/jenkins")), - httpClient) - - def result = apiClient.runScript("println('ok')") - assertThat(result).isEqualTo("ok") - - wireMock.verify(1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .withHeader("Authorization", matching("Basic .*"))) - - wireMock.verify(1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText")) - .withHeader("Authorization", matching("Basic .*")) - .withHeader("Jenkins-Crumb", equalTo("the-crumb"))) - } - - @Test - void 'adds crumb to sendRequest'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-crumb", "crumbRequestField": "Jenkins-Crumb"}'))) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/foobar")) - .willReturn(aResponse().withStatus(200))) - - def client = new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl()}/jenkins")), - getUnsafeOkHttpClient()) - client.postRequestWithCrumb("foobar") - - wireMock.verify(1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))) - wireMock.verify(1, postRequestedFor(urlPathEqualTo("/jenkins/foobar")) - .withHeader("Jenkins-Crumb", equalTo("the-crumb"))) - } - - @Test - void 'adds crumb and post data to sendRequest'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-crumb", "crumbRequestField": "Jenkins-Crumb"}'))) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/foobar")) - .willReturn(aResponse().withStatus(200))) - - def client = new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl()}/jenkins")), - getUnsafeOkHttpClient()) - client.postRequestWithCrumb("foobar", new FormBody.Builder().add('key', 'value with spaces').build()) - - wireMock.verify(1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))) - wireMock.verify(1, postRequestedFor(urlPathEqualTo("/jenkins/foobar")) - .withHeader("Jenkins-Crumb", equalTo("the-crumb")) - .withFormParam("key", equalTo("value with spaces"))) - - } - - @Test - void 'allows self-signed certificates when using insecure'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-crumb", "crumbRequestField": "Jenkins-Crumb"}'))) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) - .willReturn(aResponse() - .withStatus(200) - .withBody("ok"))) - - def apiClient = ApplicationContext.run() - .registerSingleton(new Config(application: new Config.ApplicationSchema(insecure: true), - jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl().replace('http://', 'https://')}/jenkins"))) - .getBean(JenkinsApiClient) - - def result = apiClient.runScript("println('ok')") - assertThat(result).isEqualTo("ok") - - wireMock.verify(1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .withHeader("Authorization", matching("Basic .*"))) - - wireMock.verify(1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText")) - .withHeader("Authorization", matching("Basic .*")) - .withHeader("Jenkins-Crumb", equalTo("the-crumb"))) - } - - @Test - void 'retries on invalid crumb'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .inScenario("Invalid Crumb Retry") - .whenScenarioStateIs("Started") - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-invalid-crumb", "crumbRequestField": "Jenkins-Crumb"}')) - .willSetStateTo("First Crumb")) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) - .inScenario("Invalid Crumb Retry") - .whenScenarioStateIs("First Crumb") - .withHeader("Jenkins-Crumb", equalTo("the-invalid-crumb")) - .willReturn(aResponse() - .withStatus(403) - .withBody('{"servlet":"Stapler", "message":"No valid crumb was included in the request", "url":"/scriptText", "status":"403"}')) - .willSetStateTo("Invalid Crumb Response")) - - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .inScenario("Invalid Crumb Retry") - .whenScenarioStateIs("Invalid Crumb Response") - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-second-crumb", "crumbRequestField": "Jenkins-Crumb"}')) - .willSetStateTo("Second Crumb")) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) - .inScenario("Invalid Crumb Retry") - .whenScenarioStateIs("Second Crumb") - .withHeader("Jenkins-Crumb", equalTo("the-second-crumb")) - .willReturn(aResponse() - .withStatus(200) - .withBody("ok"))) - - def httpClient = getUnsafeOkHttpClient() - def apiClient = new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl()}/jenkins")), - httpClient) - apiClient.setMaxRetries(3) - apiClient.setWaitPeriodInMs(0) - - def result = apiClient.runScript("println('ok')") - assertThat(result).isEqualTo("ok") - - wireMock.verify(2, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))) - wireMock.verify(2, postRequestedFor(urlPathEqualTo("/jenkins/scriptText"))) - } - - @Test - void 'retries on invalid crumb are limited'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-invalid-crumb", "crumbRequestField": "Jenkins-Crumb"}'))) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) - .willReturn(aResponse() - .withStatus(403) - .withBody('{"servlet":"Stapler", "message":"No valid crumb was included in the request", "url":"/scriptText", "status":"403"}'))) - - def httpClient = getUnsafeOkHttpClient() - def apiClient = new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl()}/jenkins")), - httpClient) - apiClient.setMaxRetries(3) - apiClient.setWaitPeriodInMs(0) - - shouldFail(RuntimeException) { - apiClient.runScript("println('ok')") - } - - wireMock.verify(3, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))) - wireMock.verify(3, postRequestedFor(urlPathEqualTo("/jenkins/scriptText"))) - } - - @Test - void 'retries when fetching crumb fails'() { - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .inScenario("Crumb Fetch Retry") - .whenScenarioStateIs("Started") - .willReturn(aResponse() - .withStatus(401) - .withBody("error")) - .willSetStateTo("First Attempt Failed")) - - wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .inScenario("Crumb Fetch Retry") - .whenScenarioStateIs("First Attempt Failed") - .willReturn(aResponse() - .withStatus(200) - .withBody('{"crumb": "the-invalid-crumb", "crumbRequestField": "Jenkins-Crumb"}'))) - - wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) - .willReturn(aResponse() - .withStatus(200) - .withBody("ok"))) - - def httpClient = getUnsafeOkHttpClient() - def apiClient = new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: "${wireMock.baseUrl()}/jenkins")), - httpClient) - apiClient.setMaxRetries(3) - apiClient.setWaitPeriodInMs(0) - - def result = apiClient.runScript("println('ok')") - assertThat(result).isEqualTo("ok") - - wireMock.verify(2, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))) - wireMock.verify(1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText"))) - } - - private static OkHttpClient getUnsafeOkHttpClient() { - try { - // Create a trust manager that does not validate certificate chains - final TrustManager[] trustAllCerts = [new X509TrustManager() { - @Override - void checkClientTrusted(X509Certificate[] chain, String authType) {} - - @Override - void checkServerTrusted(X509Certificate[] chain, String authType) {} - - @Override - X509Certificate[] getAcceptedIssuers() { - return new X509Certificate[0] - } - }] as TrustManager[] - - // Install the all-trusting trust manager - final SSLContext sslContext = SSLContext.getInstance("SSL") - sslContext.init(null, trustAllCerts, new SecureRandom()) - final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory() - - return new OkHttpClient.Builder() - .sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0]) - .hostnameVerifier { hostname, session -> true } - .build() - } catch (Exception e) { - throw new RuntimeException(e) - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/JobManagerTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/JobManagerTest.groovy deleted file mode 100644 index dce50dd0f..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/JobManagerTest.groovy +++ /dev/null @@ -1,258 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import com.cloudogu.gitops.config.Config -import com.github.tomakehurst.wiremock.WireMockServer -import okhttp3.OkHttpClient -import org.junit.jupiter.api.Test - -import static com.github.tomakehurst.wiremock.client.WireMock.* -import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.options -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.anyString -import static org.mockito.Mockito.mock -import static org.mockito.Mockito.when - -class JobManagerTest { - - @Test - void 'creates credential'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathMatching(".*createCredentials.*")) - .willReturn(ok())) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - jobManager.createCredential('the-jobname', 'the-id', 'the-username', 'the-password', 'some description') - - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname/credentials/store/folder/domain/_/createCredentials"))) - - def requests = wireMockServer.findAll(postRequestedFor(urlPathMatching(".*createCredentials.*"))) - assertThat(requests).hasSize(1) - - def requestBody = requests[0].bodyAsString - assertThat(URLDecoder.decode(requestBody, "utf-8")) - .isEqualTo('json={"credentials":{"scope":"GLOBAL","id":"the-id","username":"the-username","password":"the-password","description":"some description","$class":"com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"}}') - - } finally { - wireMockServer.stop() - } - } - - @Test - void 'throw when creating credential fails'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathMatching(".*createCredentials.*")) - .willReturn(aResponse().withStatus(404))) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - def exception = shouldFail(RuntimeException) { - jobManager.createCredential('the-jobname', 'the-id', 'the-username', 'the-password', 'some description') - } - assertThat(exception.getMessage()).isEqualTo('Could not create credential id=the-id,job=the-jobname. StatusCode: 404') - } finally { - wireMockServer.stop() - } - } - - @Test - void 'starts job'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathMatching("/jenkins/job/the-jobname/build.*")) - .willReturn(ok())) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - jobManager.startJob('the-jobname') - - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname/build")) - .withQueryParam("delay", equalTo("0sec"))) - - } finally { - wireMockServer.stop() - } - } - - @Test - void 'throw when starting job fails'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathMatching("/jenkins/job/the-jobname/build.*")) - .willReturn(aResponse().withStatus(400))) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - def exception = shouldFail(RuntimeException) { - jobManager.startJob('the-jobname') - } - assertThat(exception.getMessage()).isEqualTo('Could not trigger build of Jenkins job: the-jobname. StatusCode: 400') - } finally { - wireMockServer.stop() - } - } - - @Test - void 'throws when job contains invalid characters'() { - def client = mock(JenkinsApiClient) - def jobManager = new JobManager(client) - - def exception = shouldFail(RuntimeException) { - jobManager.deleteJob("foo'foo") - } - assertThat(exception.getMessage()).isEqualTo('Job name cannot contain quotes.') - } - - @Test - void 'throws when job deletion fails'() { - def client = mock(JenkinsApiClient) - def jobManager = new JobManager(client) - - def exception = shouldFail(RuntimeException) { - jobManager.deleteJob("foo-foo") - } - assertThat(exception.getMessage()).isEqualTo('Could not delete job foo-foo') - } - - @Test - void 'deletes job'() { - def client = mock(JenkinsApiClient) - def jobManager = new JobManager(client) - - when(client.runScript(anyString())).thenReturn("null") - jobManager.deleteJob("foo") - org.mockito.Mockito.verify(client).runScript("print(Jenkins.instance.getItem('foo')?.delete())") - } - - @Test - void 'checks existing Job'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) - .willReturn(ok())) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - def exists = jobManager.jobExists('the-jobname') - - assertThat(exists).isEqualTo(true) - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))) - } finally { - wireMockServer.stop() - } - } - - @Test - void 'checks non-existing Job'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) - .willReturn(aResponse().withStatus(404))) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - def exists = jobManager.jobExists('the-jobname') - assertThat(exists).isEqualTo(false) - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))) - } finally { - wireMockServer.stop() - } - } - - @Test - void 'creates Job'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) - .willReturn(aResponse().withStatus(404))) - wireMockServer.stubFor(post(urlPathMatching("/jenkins/createItem.*")) - .willReturn(ok())) - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - def created = jobManager.createJob('the-jobname', 'http://scm', 'ns', 'creds') - - assertThat(created).isEqualTo(true) - - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))) - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/createItem")) - .withQueryParam("name", equalTo("the-jobname")) - .withRequestBody(containing('http://scm')) - .withRequestBody(containing('ns')) - .withRequestBody(containing('creds'))) - - } finally { - wireMockServer.stop() - } - } - - @Test - void 'ignores existing Job'() { - def wireMockServer = new WireMockServer(options().dynamicPort()) - wireMockServer.start() - - try { - wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) - .willReturn(okJson('{"crumb":"the-crumb"}'))) - - wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) - .willReturn(ok())) // 200 OK means "Job Exists" - - def jobManager = new JobManager(new JenkinsApiClient(new Config(jenkins: new Config.JenkinsSchema(url: wireMockServer.baseUrl() + "/jenkins")), - new OkHttpClient())) - - def created = jobManager.createJob('the-jobname', 'http://scm', 'ns', 'creds') - - assertThat(created).isEqualTo(false) - wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))) - wireMockServer.verify(0, postRequestedFor(urlPathEqualTo("/jenkins/createItem"))) - - } finally { - wireMockServer.stop() - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/UserManagerTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/UserManagerTest.groovy deleted file mode 100644 index 851e764f5..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/jenkins/UserManagerTest.groovy +++ /dev/null @@ -1,124 +0,0 @@ -package com.cloudogu.gitops.infrastructure.jenkins - -import org.junit.jupiter.api.Test - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.anyString -import static org.mockito.Mockito.* - -class UserManagerTest { - @Test - void 'creates user successfully'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("the-user") - - new UserManager(client).createUser("the-user", "hunter2") - verify(client).runScript(anyString()) - } - - @Test - void 'creates user with quotes successfully'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("the-'user") - - new UserManager(client).createUser("the-'user", "code''injection") - verify(client).runScript(""" - def realm = Jenkins.getInstance().getSecurityRealm() - def user = realm.createAccount('the-\\'user', 'code\\'\\'injection') - - print(user) - """) - } - - @Test - void 'throws when backslashes are passed'() { - def client = mock(JenkinsApiClient) - shouldFail(IllegalArgumentException) { - new UserManager(client).createUser("the-\\'user", "hunter2") - } - } - - @Test - void 'throws when there was an error'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]") - - shouldFail(RuntimeException) { - new UserManager(client).createUser("the-user", "hunter2") - } - } - - @Test - void 'grants permission for user'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("true") - when(client.runScript("print(Jenkins.getInstance().getAuthorizationStrategy().class)")).thenReturn("class hudson.security.GlobalMatrixAuthorizationStrategy") - - new UserManager(client).grantPermission("the-'user", UserManager.Permissions.METRICS_VIEW) - - verify(client).runScript("""print(Jenkins.getInstance().getAuthorizationStrategy().class)""") - verify(client).runScript(""" - import org.jenkinsci.plugins.matrixauth.PermissionEntry - import org.jenkinsci.plugins.matrixauth.AuthorizationType - - def permissions = Jenkins.getInstance().getAuthorizationStrategy().getGrantedPermissionEntries() - permissions.computeIfAbsent(jenkins.metrics.api.Metrics.VIEW) { - new HashSet<>() - } - print(permissions[jenkins.metrics.api.Metrics.VIEW].add(new PermissionEntry(AuthorizationType.USER, 'the-\\'user'))) - """) - } - - @Test - void 'throws when granting permission failed'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]") - - shouldFail(RuntimeException) { - new UserManager(client).grantPermission("the-'user", UserManager.Permissions.METRICS_VIEW) - } - } - - @Test - void 'checks whether matrix based authorization is enabled'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("class hudson.security.GlobalMatrixAuthorizationStrategy") - - assertThat(new UserManager(client).isUsingMatrixBasedPermissions()).isTrue() - } - - @Test - void 'checks whether matrix based authorization is disabled'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("class hudson.security.FullControlOnceLoggedInAuthorizationStrategy") - - assertThat(new UserManager(client).isUsingMatrixBasedPermissions()).isFalse() - } - - @Test - void 'checks whether cas security realm is used'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("class org.jenkinsci.plugins.cas.CasSecurityRealm") - - assertThat(new UserManager(client).isUsingCasSecurityRealm()).isTrue() - } - - @Test - void 'checks whether cas security realm is not used'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("class hudson.security.HudsonPrivateSecurityRealm") - - assertThat(new UserManager(client).isUsingCasSecurityRealm()).isFalse() - } - - @Test - void 'throws when determining security realm errors'() { - def client = mock(JenkinsApiClient) - when(client.runScript(anyString())).thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]") - - shouldFail(RuntimeException) { - new UserManager(client).isUsingCasSecurityRealm() - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientTest.groovy deleted file mode 100644 index 857e31522..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientTest.groovy +++ /dev/null @@ -1,1386 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.api - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials - -import java.nio.file.Files -import java.nio.file.Path -import groovy.json.JsonSlurper - -import io.fabric8.kubernetes.api.model.* -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer -import io.fabric8.openshift.api.model.ProjectBuilder -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.io.TempDir - -@EnableKubernetesMockClient -class K8sClientTest { - - KubernetesMockServer server - KubernetesClient client - - K8sClient k8sApiClient - - @TempDir - Path tempDir - - @BeforeEach - void setup() { - k8sApiClient = new K8sClient() - k8sApiClient.client = client - k8sApiClient.SLEEPTIME = 10 // Speed up tests - k8sApiClient.DEFAULT_RETRIES = 3 - } - - // ======================================== - // Node Operations Tests - // ======================================== - - @Test - void 'waitForNode returns first node name'() { - // Given - def node = new NodeBuilder() - .withNewMetadata() - .withName("test-node-1") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/nodes") - .andReturn(200, new NodeListBuilder().withItems(node).build()) - .once() - - // When - String nodeName = k8sApiClient.waitForNode() - - // Then - assertThat(nodeName).isEqualTo("test-node-1") - } - - @Test - void 'waitForNode retries when no nodes available'() { - // Given - server.expect() - .get() - .withPath("/api/v1/nodes") - .andReturn(200, new NodeListBuilder().build()) - .times(2) - - def node = new NodeBuilder() - .withNewMetadata() - .withName("test-node-1") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/nodes") - .andReturn(200, new NodeListBuilder().withItems(node).build()) - .once() - - // When - String nodeName = k8sApiClient.waitForNode() - - // Then - assertThat(nodeName).isEqualTo("test-node-1") - } - - @Test - void 'waitForNode throws exception after max retries'() { - // Given - server.expect() - .get() - .withPath("/api/v1/nodes") - .andReturn(200, new NodeListBuilder().build()) - .times(k8sApiClient.DEFAULT_RETRIES + 1) - - // When/Then - def exception = shouldFail(RuntimeException) { - k8sApiClient.waitForNode() - } - assertThat(exception.message).contains("Failed to retrieve node") - } - - @Test - void 'waitForInternalNodeIp returns node internal IP'() { - // Given - First call for waitForNode - def node = new NodeBuilder() - .withNewMetadata() - .withName("test-node-1") - .endMetadata() - .withNewStatus() - .addNewAddress() - .withType("InternalIP") - .withAddress("192.168.1.100") - .endAddress() - .endStatus() - .build() - - server.expect() - .get() - .withPath("/api/v1/nodes") - .andReturn(200, new NodeListBuilder().withItems(node).build()) - .once() - - // Second call for waitForInternalNodeIp - server.expect() - .get() - .withPath("/api/v1/nodes/test-node-1") - .andReturn(200, node) - .once() - - // When - String ip = k8sApiClient.waitForInternalNodeIp() - - // Then - assertThat(ip).isEqualTo("192.168.1.100") - } - - @Test - void 'waitForInternalNodeIp ignores IPv6 addresses'() { - // Given - def node = new NodeBuilder() - .withNewMetadata() - .withName("test-node-1") - .endMetadata() - .withNewStatus() - .addNewAddress() - .withType("InternalIP") - .withAddress("192.168.1.100") - .endAddress() - .addNewAddress() - .withType("InternalIP") - .withAddress("fe80::1") - .endAddress() - .endStatus() - .build() - - server.expect() - .get() - .withPath("/api/v1/nodes") - .andReturn(200, new NodeListBuilder().withItems(node).build()) - .once() - - server.expect() - .get() - .withPath("/api/v1/nodes/test-node-1") - .andReturn(200, node) - .once() - - // When - String ip = k8sApiClient.waitForInternalNodeIp() - - // Then - assertThat(ip).isEqualTo("192.168.1.100") - } - - // ======================================== - // Service Operations Tests - // ======================================== - - @Test - void 'waitForNodePort returns service nodePort'() { - // Given - def service = new ServiceBuilder() - .withNewMetadata() - .withName("test-service") - .withNamespace("test-ns") - .endMetadata() - .withNewSpec() - .addNewPort() - .withPort(8080) - .withNodePort(30080) - .endPort() - .endSpec() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/services/test-service") - .andReturn(200, service) - .once() - - // When - String nodePort = k8sApiClient.waitForNodePort("test-service", "test-ns") - - // Then - assertThat(nodePort).isEqualTo("30080") - } - - @Test - void 'createServiceNodePort creates service with nodePort'() { - // Given - // createOrReplace() tries POST first - server.expect() - .post() - .withPath("/api/v1/namespaces/default/services") - .andReturn(201, new ServiceBuilder() - .withNewMetadata() - .withName("my-service") - .withNamespace("default") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createServiceNodePort("my-service", "8080:80", "30000", "") - - // Then - Verify the request was made (mock server expectation will fail if not) - } - - @Test - void 'createServiceNodePort creates service without explicit nodePort'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/test-ns/services") - .andReturn(201, new ServiceBuilder() - .withNewMetadata() - .withName("my-service") - .withNamespace("test-ns") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createServiceNodePort("my-service", "8080:80", "", "test-ns") - - // Then - Verify the request was made - } - - @Test - void 'patchServiceNodePort updates service port'() { - // Given - def service = new ServiceBuilder() - .withNewMetadata() - .withName("test-service") - .withNamespace("test-ns") - .endMetadata() - .withNewSpec() - .addNewPort() - .withName("http") - .withPort(8080) - .withNodePort(30080) - .endPort() - .endSpec() - .build() - - // patchServiceNodePort makes a GET, then patch() makes another GET followed by PATCH - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/services/test-service") - .andReturn(200, service) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/services/test-service") - .andReturn(200, service) - .once() - - server.expect() - .patch() - .withPath("/api/v1/namespaces/test-ns/services/test-service") - .andReturn(200, service) - .once() - - // When - k8sApiClient.patchServiceNodePort("test-service", "test-ns", "http", 30090) - - // Then - Verify patch was called - } - - @Test - void 'patchServiceNodePort throws exception for invalid parameters'() { - // When/Then - def exception = shouldFail(IllegalArgumentException) { - k8sApiClient.patchServiceNodePort("", "test-ns", "http", 30000) - } - assertThat(exception.message).contains("Service name") - } - - @Test - void 'patchServiceNodePort throws exception when port not found'() { - // Given - def service = new ServiceBuilder() - .withNewMetadata() - .withName("test-service") - .withNamespace("test-ns") - .endMetadata() - .withNewSpec() - .addNewPort() - .withName("http") - .withPort(8080) - .endPort() - .endSpec() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/services/test-service") - .andReturn(200, service) - .once() - - // When/Then - def exception = shouldFail(RuntimeException) { - k8sApiClient.patchServiceNodePort("test-service", "test-ns", "https", 30000) - } - assertThat(exception.message).contains("Port with name https not found") - } - - // ======================================== - // Namespace Operations Tests - // ======================================== - - @Test - void 'createNamespace creates new namespace'() { - // Given - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns") - .andReturn(404, "") - .once() - - server.expect() - .post() - .withPath("/api/v1/namespaces") - .andReturn(201, new NamespaceBuilder() - .withNewMetadata() - .withName("test-ns") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createNamespace("test-ns") - - // Then - Verify namespace was created - } - - @Test - void 'createNamespace creates OpenShift project when openshift config is enabled'() { - // Given - Config config = Config.fromMap([application: [openshift: true]]) - k8sApiClient.gopConfig = config - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-project") - .andReturn(404, "") - .once() - - server.expect() - .post() - .withPath("/apis/project.openshift.io/v1/projects") - .andReturn(201, new ProjectBuilder() - .withNewMetadata() - .withName("test-project") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createNamespace("test-project") - - // Then - def requestBody = new JsonSlurper().parseText(server.getLastRequest().getUtf8Body()) as Map - assertThat(requestBody["kind"]).isEqualTo("Project") - assertThat(requestBody["metadata"]["name"]).isEqualTo("test-project") - } - - @Test - void 'createNamespace creates Kubernetes namespace when openshift config is disabled'() { - // Given - Config config = Config.fromMap([application: [openshift: false]]) - k8sApiClient.gopConfig = config - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns") - .andReturn(404, "") - .once() - - server.expect() - .post() - .withPath("/api/v1/namespaces") - .andReturn(201, new NamespaceBuilder() - .withNewMetadata() - .withName("test-ns") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createNamespace("test-ns") - - // Then - def requestBody = new JsonSlurper().parseText(server.getLastRequest().getUtf8Body()) as Map - assertThat(requestBody["kind"]).isEqualTo("Namespace") - assertThat(requestBody["metadata"]["name"]).isEqualTo("test-ns") - } - - @Test - void 'createNamespace does not create existing namespace'() { - // Given - def namespace = new NamespaceBuilder() - .withNewMetadata() - .withName("test-ns") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns") - .andReturn(200, namespace) - .once() - - // When - k8sApiClient.createNamespace("test-ns") - - // Then - assertThat(server.getLastRequest().method).isEqualTo("GET") - assertThat(server.getLastRequest().path).isEqualTo("/api/v1/namespaces/test-ns") - } - - @Test - void 'createNamespace creates Kubernetes namespace when config is null'() { - // Given - k8sApiClient.gopConfig = null - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns") - .andReturn(404, "") - .once() - - server.expect() - .post() - .withPath("/api/v1/namespaces") - .andReturn(201, new NamespaceBuilder() - .withNewMetadata() - .withName("test-ns") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createNamespace("test-ns") - - // Then - def requestBody = new JsonSlurper().parseText(server.getLastRequest().getUtf8Body()) as Map - assertThat(requestBody["kind"]).isEqualTo("Namespace") - assertThat(requestBody["metadata"]["name"]).isEqualTo("test-ns") - } - - @Test - void 'createNamespace does not create OpenShift project when namespace already exists'() { - // Given - Config config = Config.fromMap([application: [openshift: true]]) - k8sApiClient.gopConfig = config - - def namespace = new NamespaceBuilder() - .withNewMetadata() - .withName("existing-project") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/existing-project") - .andReturn(200, namespace) - .once() - - // When - k8sApiClient.createNamespace("existing-project") - - // Then - assertThat(server.getLastRequest().method).isEqualTo("GET") - assertThat(server.getLastRequest().path).isEqualTo("/api/v1/namespaces/existing-project") - } - - @Test - void 'createNamespace throws exception for invalid name'() { - // When/Then - def exception = shouldFail(IllegalArgumentException) { - k8sApiClient.createNamespace("") - } - assertThat(exception.message).contains("Namespace name must be provided") - } - - @Test - void 'createNamespaces creates multiple namespaces'() { - // Given - server.expect() - .get() - .withPath("/api/v1/namespaces/ns1") - .andReturn(404, "") - .once() - - server.expect() - .post() - .withPath("/api/v1/namespaces") - .andReturn(201, new NamespaceBuilder().withNewMetadata().withName("ns1").endMetadata().build()) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/ns2") - .andReturn(404, "") - .once() - - server.expect() - .post() - .withPath("/api/v1/namespaces") - .andReturn(201, new NamespaceBuilder().withNewMetadata().withName("ns2").endMetadata().build()) - .once() - - // When - k8sApiClient.createNamespaces(["ns1", "ns2"]) - - // Then - Verify both namespaces were created - } - - @Test - void 'namespaceExists returns true for existing namespace'() { - // Given - def namespace = new NamespaceBuilder() - .withNewMetadata() - .withName("test-ns") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns") - .andReturn(200, namespace) - .once() - - // When - boolean exists = k8sApiClient.namespaceExists("test-ns") - - // Then - assertThat(exists).isTrue() - } - - @Test - void 'namespaceExists returns false for non-existing namespace'() { - // Given - server.expect() - .get() - .withPath("/api/v1/namespaces/non-existing") - .andReturn(404, "") - .once() - - // When - boolean exists = k8sApiClient.namespaceExists("non-existing") - - // Then - assertThat(exists).isFalse() - } - - // ======================================== - // Secret Operations Tests - // ======================================== - - @Test - void 'createSecret creates generic secret'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/test-ns/secrets") - .andReturn(201, new SecretBuilder() - .withNewMetadata() - .withName("my-secret") - .withNamespace("test-ns") - .endMetadata() - .withType("Opaque") - .build()) - .once() - - // When - k8sApiClient.createSecret("Opaque", "my-secret", "test-ns", - new Tuple2("username", "admin"), - new Tuple2("password", "secret")) - - // Then - Verify secret was created - } - - @Test - void 'createImagePullSecret creates docker registry secret'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/default/secrets") - .andReturn(201, new SecretBuilder() - .withNewMetadata() - .withName("my-registry") - .withNamespace("default") - .endMetadata() - .withType("kubernetes.io/dockerconfigjson") - .build()) - .once() - - // When - k8sApiClient.createImagePullSecret("my-registry", "", "docker.io", "user", "pass") - - // Then - Verify secret was created - } - - @Test - void 'getArgoCDNamespacesSecret retrieves secret data'() { - // Given - def secret = new SecretBuilder() - .withNewMetadata() - .withName("argocd-secret") - .withNamespace("argocd") - .endMetadata() - .withData(["namespaces": Base64.encoder.encodeToString("ns1,ns2".bytes)]) - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/argocd/secrets/argocd-secret") - .andReturn(200, secret) - .once() - - // When - String data = k8sApiClient.getArgoCDNamespacesSecret("argocd-secret", "argocd") - - // Then - assertThat(data).isEqualTo(Base64.encoder.encodeToString("ns1,ns2".bytes)) - } - - @Test - void 'getCredentialsFromSecret extracts username and password'() { - // Given - def secret = new SecretBuilder() - .withNewMetadata() - .withName("my-secret") - .withNamespace("test-ns") - .endMetadata() - .withData(["username": Base64.encoder.encodeToString("admin".bytes), - "password": Base64.encoder.encodeToString("secret123".bytes)]) - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/secrets/my-secret") - .andReturn(200, secret) - .once() - - // When - Credentials creds = k8sApiClient.getCredentialsFromSecret("my-secret", "test-ns") - - // Then - assertThat(creds.username).isEqualTo("admin") - assertThat(creds.password).isEqualTo("secret123") - } - - @Test - void 'getCredentialsFromSecret with Credentials object'() { - // Given - def inputCreds = new Credentials(secretName: "my-secret", - secretNamespace: "test-ns", - usernameKey: "user", - passwordKey: "pass") - - def secret = new SecretBuilder() - .withNewMetadata() - .withName("my-secret") - .withNamespace("test-ns") - .endMetadata() - .withData(["user": Base64.encoder.encodeToString("testuser".bytes), - "pass": Base64.encoder.encodeToString("testpass".bytes)]) - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/secrets/my-secret") - .andReturn(200, secret) - .once() - - // When - Credentials result = k8sApiClient.getCredentialsFromSecret(inputCreds) - - // Then - assertThat(result.username).isEqualTo("testuser") - assertThat(result.password).isEqualTo("testpass") - } - - // ======================================== - // ConfigMap Operations Tests - // ======================================== - - @Test - void 'createConfigMapFromFile creates configmap'() { - // Given - Path testFile = tempDir.resolve("test.txt") - Files.writeString(testFile, "test content") - - server.expect() - .post() - .withPath("/api/v1/namespaces/default/configmaps") - .andReturn(201, new ConfigMapBuilder() - .withNewMetadata() - .withName("my-config") - .withNamespace("default") - .endMetadata() - .build()) - .once() - - // When - k8sApiClient.createConfigMapFromFile("my-config", "", testFile.toString()) - - // Then - Verify configmap was created - } - - @Test - void 'createConfigMapFromFile throws exception for non-existing file'() { - // When/Then - def exception = shouldFail(RuntimeException) { - k8sApiClient.createConfigMapFromFile("my-config", "", "/non/existing/file.txt") - } - assertThat(exception.message).contains("File not found") - } - - @Test - void 'getConfigMap retrieves value from configmap'() { - // Given - def configMap = new ConfigMapBuilder() - .withNewMetadata() - .withName("my-config") - .withNamespace("default") - .endMetadata() - .withData(["key1": "value1", "key2": "value2"]) - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/configmaps/my-config") - .andReturn(200, configMap) - .once() - - // When - String value = k8sApiClient.getConfigMap("my-config", "key1") - - // Then - assertThat(value).isEqualTo("value1") - } - - @Test - void 'getConfigMap throws exception for non-existing key'() { - // Given - def configMap = new ConfigMapBuilder() - .withNewMetadata() - .withName("my-config") - .withNamespace("default") - .endMetadata() - .withData(["key1": "value1"]) - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/configmaps/my-config") - .andReturn(200, configMap) - .once() - - // When/Then - def exception = shouldFail(RuntimeException) { - k8sApiClient.getConfigMap("my-config", "non-existing-key") - } - assertThat(exception.message).contains("Could not fetch non-existing-key") - } - - // ======================================== - // Resource Management Tests - // ======================================== - - @Test - void 'applyYaml applies resources from file'() { - // Given - Path yamlFile = tempDir.resolve("test.yaml") - Files.writeString(yamlFile, """ -apiVersion: v1 -kind: Namespace -metadata: - name: test-ns -""") - - server.expect() - .post() - .withPath("/api/v1/namespaces") - .andReturn(201, new NamespaceBuilder() - .withNewMetadata() - .withName("test-ns") - .endMetadata() - .build()) - .once() - - // When - String result = k8sApiClient.applyYaml(yamlFile.toString()) - - // Then - assertThat(result).contains("Applied 1 resource(s)") - } - - @Test - void 'applyYaml throws exception for non-existing file or directory'() { - // When/Then - def exception = shouldFail(RuntimeException) { - k8sApiClient.applyYaml("/non/existing/file.yaml") - } - - assertThat(exception.message).contains("File or directory not found") - assertThat(exception.message).contains("/non/existing/file.yaml") - } - - @Test - void 'label adds labels to resource'() { - // Given - def pod = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("default") - .withLabels(["existing": "label"]) - .endMetadata() - .build() - - // label() makes a GET, then replace() makes another GET followed by PUT - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - server.expect() - .put() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - // When - k8sApiClient.label("pod", "test-pod", "default", - new Tuple2("app", "myapp"), - new Tuple2("version", "1.0")) - - // Then - Verify labels were updated - } - - @Test - void 'labelRemove removes labels from resource'() { - // Given - def pod = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("default") - .withLabels(["app": "myapp", "version": "1.0"]) - .endMetadata() - .build() - - // label() makes a GET, then replace() makes another GET followed by PUT - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - server.expect() - .put() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - // When - k8sApiClient.labelRemove("pod", "test-pod", "default", "version") - - // Then - Verify label was removed - } - - @Test - void 'patch patches resource with strategic merge'() { - // Given - def pod = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("default") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - server.expect() - .patch() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - // When - k8sApiClient.patch("pod", "test-pod", "default", "strategic", ["metadata": ["labels": ["new": "label"]]]) - - // Then - Verify patch was applied - } - - @Test - void 'delete removes resources by label selector'() { - // Given - server.expect() - .delete() - .withPath("/api/v1/namespaces/test-ns/pods?labelSelector=app%3Dmyapp") - .andReturn(200, new StatusBuilder().build()) - .once() - - // When - k8sApiClient.delete("pod", "test-ns", new Tuple2("app", "myapp")) - - // Then - Verify delete was called - } - - @Test - void 'delete removes specific resource by name'() { - // Given - server.expect() - .delete() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, new StatusBuilder().build()) - .once() - - // When - k8sApiClient.delete("pod", "test-ns", "test-pod") - - // Then - Verify delete was called - } - - @Test - void 'run creates pod with image'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/default/pods") - .andReturn(201, new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .endMetadata() - .build()) - .once() - - // When - String result = k8sApiClient.run("test-pod", "nginx:latest", "", [:]) - - // Then - assertThat(result).contains("pod/test-pod created") - } - - @Test - void 'run applies pod overrides instead of generated parameter values'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/jenkins/pods") - .andReturn(201, new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .endMetadata() - .build()) - .once() - - String overrideImage = "bash:42" - Map overrides = [spec: [containers : [[name : "override-container", - image : "${overrideImage}", - args : ["cat", "/etc/group"], - volumeMounts: [[name: "group", mountPath: "/etc/group", readOnly: true]]]], - nodeSelector: [node: "jenkins"], - volumes : [[name: "group", hostPath: [path: "/etc/group"]]]]] - - // When - k8sApiClient.run("test-pod", "nginx:latest", "jenkins", overrides) - - // Then - def requestBody = new JsonSlurper().parseText(server.getLastRequest().getUtf8Body()) as Map - assertThat(requestBody["metadata"]["name"]).isEqualTo("test-pod") - assertThat(requestBody["metadata"]["namespace"]).isEqualTo("jenkins") - assertThat(requestBody["spec"]["nodeSelector"]["node"]).isEqualTo("jenkins") - - List containers = requestBody["spec"]["containers"] as List - assertThat(containers).hasSize(1) - Map container = containers[0] as Map - assertThat(container["name"]).isEqualTo("override-container") - assertThat(container["image"]).isEqualTo("bash:42") - assertThat(container["args"] as List).containsExactly("cat", "/etc/group") - - List volumeMounts = container["volumeMounts"] as List - Map volumeMount = volumeMounts[0] as Map - assertThat(volumeMount["mountPath"]).isEqualTo("/etc/group") - assertThat(volumeMount["readOnly"]).isEqualTo(true) - - List volumes = requestBody["spec"]["volumes"] as List - Map volume = volumes[0] as Map - assertThat((volume["hostPath"] as Map)["path"]).isEqualTo("/etc/group") - } - - @Test - void 'run returns pod logs and removes pod for interactive rm mode'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/jenkins/pods") - .andReturn(201, new PodBuilder() - .withNewMetadata() - .withName("gid-pod") - .endMetadata() - .build()) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/jenkins/pods/gid-pod") - .andReturn(200, new PodBuilder() - .withNewMetadata() - .withName("gid-pod") - .endMetadata() - .withNewStatus() - .withPhase("Succeeded") - .endStatus() - .build()) - .once() - - def succeededPod = new PodBuilder() - .withNewMetadata() - .withName("gid-pod") - .endMetadata() - .withNewStatus() - .withPhase("Succeeded") - .endStatus() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/jenkins/pods?fieldSelector=metadata.name%3Dgid-pod") - .andReturn(200, new PodListBuilder().withItems(succeededPod).build()) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/jenkins/pods/gid-pod/log?pretty=false") - .andReturn(200, "root:x:0:\ndocker:x:42:\n") - .once() - - server.expect() - .delete() - .withPath("/api/v1/namespaces/jenkins/pods/gid-pod") - .andReturn(200, new StatusBuilder().build()) - .once() - - // When - String result = k8sApiClient.run("gid-pod", "bash:42", "jenkins", [:], "--restart=Never", "-ti", "--rm", "--quiet") - - // Then - assertThat(result).isEqualTo("root:x:0:\ndocker:x:42:\n") - - def createRequest = new JsonSlurper().parseText(server.takeRequest().getUtf8Body()) as Map - assertThat(createRequest["spec"]["restartPolicy"]).isEqualTo("Never") - } - - // ======================================== - // Query Operations Tests - // ======================================== - - @Test - void 'getCustomResource returns list of custom resources'() { - // Given - Mock server setup for generic resources is complex, simplifying - // When/Then - This would need more sophisticated mocking - // Skipping detailed test due to complexity with genericKubernetesResources - } - - @Test - void 'getAnnotation retrieves annotation value'() { - // Given - def pod = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("default") - .withAnnotations(["key1": "value1", "key2": "value2"]) - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - // When - String value = k8sApiClient.getAnnotation("pod", "test-pod", "key1", "default") - - // Then - assertThat(value).isEqualTo("value1") - } - - @Test - void 'getAnnotation returns null for non-existing annotation'() { - // Given - def pod = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("default") - .withAnnotations(["key1": "value1"]) - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/default/pods/test-pod") - .andReturn(200, pod) - .once() - - // When - String value = k8sApiClient.getAnnotation("pod", "test-pod", "non-existing", "default") - - // Then - assertThat(value).isNull() - } - - @Test - void 'getCurrentContext returns context name'() { - // When - String context = k8sApiClient.getCurrentContext() - - // Then - assertThat(context).isNotNull() - // Note: Actual value depends on mock client configuration - } - - // ======================================== - // Wait Operations Tests - // ======================================== - - @Test - void 'waitForResourcePhase waits for pod to reach Running phase'() { - // Given - def podRunning = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("test-ns") - .endMetadata() - .withNewStatus() - .withPhase("Running") - .endStatus() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, podRunning) - .once() - - // When - k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 5, 1) - - // Then - No exception means success - } - - @Test - void 'waitForResourcePhase retries until phase is reached'() { - // Given - def podPending = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("test-ns") - .endMetadata() - .withNewStatus() - .withPhase("Pending") - .endStatus() - .build() - - def podRunning = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("test-ns") - .endMetadata() - .withNewStatus() - .withPhase("Running") - .endStatus() - .build() - - // First two requests return Pending - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, podPending) - .once() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, podPending) - .once() - - // Third request returns Running - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, podRunning) - .once() - - // When - k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 10, 1) - - // Then - No exception means success - } - - @Test - void 'waitForResourcePhase throws exception on timeout'() { - // Given - def podPending = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("test-ns") - .endMetadata() - .withNewStatus() - .withPhase("Pending") - .endStatus() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, podPending) - .always() - - // When/Then - def exception = shouldFail(RuntimeException) { - k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 2, 1) - } - assertThat(exception.message).contains("Timeout reached") - } - - @Test - void 'waitForResourcePhase with default timeout'() { - // Given - def podRunning = new PodBuilder() - .withNewMetadata() - .withName("test-pod") - .withNamespace("test-ns") - .endMetadata() - .withNewStatus() - .withPhase("Running") - .endStatus() - .build() - - server.expect() - .get() - .withPath("/api/v1/namespaces/test-ns/pods/test-pod") - .andReturn(200, podRunning) - .always() - - // When - k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running") - - // Then - No exception means success - } - - @Test - void 'waitForResourcePhase validates parameters'() { - // When/Then - def exception = shouldFail(IllegalArgumentException) { - k8sApiClient.waitForResourcePhase("", "test-pod", "test-ns", "Running", 60, 1) - } - assertThat(exception.message).contains("Resource type") - - exception = shouldFail(IllegalArgumentException) { - k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 0, 1) - } - assertThat(exception.message).contains("Timeout") - - exception = shouldFail(IllegalArgumentException) { - k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 60, 0) - } - assertThat(exception.message).contains("check interval") - } - - // ======================================== - // Edge Cases and Error Handling Tests - // ======================================== - - @Test - void 'resolves default namespace for empty string'() { - // Given - server.expect() - .post() - .withPath("/api/v1/namespaces/default/secrets") - .andReturn(201, new SecretBuilder() - .withNewMetadata() - .withName("test-secret") - .withNamespace("default") - .endMetadata() - .withType("Opaque") - .build()) - .once() - - // When - k8sApiClient.createSecret("Opaque", "test-secret", "", new Tuple2("key", "value")) - - // Then - Verify default namespace was used - } - - @Test - void 'handles multiple resource types in getResourceClient'() { - // Test covered indirectly by other tests, but we can verify deployment - // Given - def deployment = new io.fabric8.kubernetes.api.model.apps.DeploymentBuilder() - .withNewMetadata() - .withName("test-deploy") - .withNamespace("default") - .endMetadata() - .build() - - server.expect() - .get() - .withPath("/apis/apps/v1/namespaces/default/deployments/test-deploy") - .andReturn(200, deployment) - .once() - - server.expect() - .delete() - .withPath("/apis/apps/v1/namespaces/default/deployments/test-deploy") - .andReturn(200, new StatusBuilder().build()) - .once() - - // When - k8sApiClient.delete("deployment", "default", "test-deploy") - - // Then - Verify delete was called for deployment - } - - @Test - void 'CustomResource class is immutable'() { - // When - def cr = new K8sClient.CustomResource("test-ns", "test-name") - - // Then - assertThat(cr.namespace).isEqualTo("test-ns") - assertThat(cr.name).isEqualTo("test-name") - } -} diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sJavaApiClientTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sJavaApiClientTest.groovy deleted file mode 100644 index e69de29bb..000000000 diff --git a/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinitionTest.groovy b/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinitionTest.groovy deleted file mode 100644 index 04f2ba518..000000000 --- a/src/test/groovy/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinitionTest.groovy +++ /dev/null @@ -1,299 +0,0 @@ -package com.cloudogu.gitops.infrastructure.kubernetes.rbac - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.yaml.YamlSlurper -import org.junit.jupiter.api.Test - -import static org.assertj.core.api.Assertions.assertThat -import static org.junit.jupiter.api.Assertions.assertThrows - -class RbacDefinitionTest { - - private final Config config = Config.fromMap([scm : [scmManager: [username: 'user', - password: 'pass', - protocol: 'http', - host : 'localhost',],], - application: [namePrefix: '', - insecure : false, - gitName : 'Test User', - gitEmail : 'test@example.com']]) - - private final GitRepo repo = new GitRepo(config, null, "my-repo", new FileSystemUtils()) - - @Test - void 'generates at least one RBAC YAML file'() { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("access") - .withNamespace("testing") - .withServiceAccountsFrom("testing", ["reader"]) - .withRepo(repo) - .withConfig(config) - .generate() - - File outputDir = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac") - File[] yamlFiles = outputDir.listFiles({ file -> file.name.endsWith(".yaml") } as FileFilter) - List fileNames = yamlFiles.collect { it.name } - - assertThat(yamlFiles).isNotEmpty() - assertThat(fileNames).anyMatch { it.contains("role") || it.contains("rolebinding") } - } - - @Test - void 'fails if name is missing'() { - def ex = assertThrows(IllegalArgumentException) { - new RbacDefinition(Role.Variant.ARGOCD) - .withNamespace("testing") - .withServiceAccountsFrom("testing", ["reader"]) - .withRepo(repo) - .withConfig(config) - .generate() - } - - assertThat(ex.message).contains("name must not be blank") - } - - @Test - void 'fails if namespace is missing'() { - def ex = assertThrows(IllegalArgumentException) { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("access") - .withServiceAccountsFrom("testing", ["reader"]) - .withRepo(repo) - .withConfig(config) - .generate() - } - - assertThat(ex.message).contains("namespace must not be blank") - } - - @Test - void 'fails if service accounts are empty'() { - def ex = assertThrows(IllegalArgumentException) { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("access") - .withNamespace("testing") - .withRepo(repo) - .withConfig(config) - .withServiceAccounts([]) // leer übergeben - .generate() - } - assertThat(ex.message).contains("At least one service account") - } - - @Test - void 'accepts service accounts via withServiceAccounts directly'() { - def sa = new ServiceAccountRef("myns", "mysa") - - new RbacDefinition(Role.Variant.ARGOCD) - .withName("direct") - .withNamespace("myns") - .withServiceAccounts([sa]) - .withRepo(repo) - .withConfig(config) - .generate() - - File f = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac/rolebinding-direct-myns.yaml") - assertThat(f).exists() - } - - @Test - void 'custom subfolder is respected'() { - String custom = "custom-dir" - new RbacDefinition(Role.Variant.ARGOCD) - .withName("custom") - .withNamespace("testing") - .withSubfolder(custom) - .withServiceAccountsFrom("testing", ["reader"]) - .withRepo(repo) - .withConfig(config) - .generate() - - File out = new File(repo.getAbsoluteLocalRepoTmpDir(), custom) - File[] yamlFiles = out.listFiles({ file -> file.name.endsWith(".yaml") } as FileFilter) - List fileNames = yamlFiles.collect { it.name } - - assertThat(yamlFiles).isNotEmpty() - assertThat(fileNames).anyMatch { it.contains("role") || it.contains("rolebinding") } - } - - @Test - void 'multiple service accounts are rendered correctly'() { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("multi") - .withNamespace("testing") - .withServiceAccountsFrom("testing", ["reader", "writer", "admin"]) - .withRepo(repo) - .withConfig(config) - .generate() - - File[] files = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac").listFiles() - List fileNames = files.collect { it.name } - assertThat(fileNames).anyMatch { it.contains("role") } - } - - @Test - void 'custom role and binding file names are rendered'() { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("myrole") - .withNamespace("custom-ns") - .withServiceAccountsFrom("custom-ns", ["sa1"]) - .withRepo(repo) - .withConfig(config) - .generate() - - File outputDir = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac") - List fileNames = outputDir.listFiles().collect { it.name } - - assertThat(fileNames).contains("role-myrole-custom-ns.yaml", "rolebinding-myrole-custom-ns.yaml") - } - - @Test - void 'subfolder can be nested'() { - String nested = "some/nested/path" - new RbacDefinition(Role.Variant.ARGOCD) - .withName("nestedtest") - .withNamespace("ns") - .withServiceAccountsFrom("ns", ["sa1"]) - .withSubfolder(nested) - .withRepo(repo) - .withConfig(config) - .generate() - - File outputDir = new File(repo.getAbsoluteLocalRepoTmpDir(), nested) - List fileNames = outputDir.listFiles().collect { it.name } - - assertThat(fileNames).contains("role-nestedtest-ns.yaml", "rolebinding-nestedtest-ns.yaml") - } - - @Test - void 'fails if repo is not set'() { - IllegalStateException ex = assertThrows(IllegalStateException) { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("failtest") - .withNamespace("ns") - .withServiceAccountsFrom("ns", ["sa1"]) - .withConfig(config) - .generate() - } - - assertThat(ex.message).contains("SCMM repo must be set using withRepo() before calling generate()") - } - - @Test - void 'rendered rolebinding yaml contains correct service accounts'() { - List saList = ["reader", "writer"] - String ns = "rbac-test" - - new RbacDefinition(Role.Variant.ARGOCD) - .withName("test") - .withNamespace(ns) - .withServiceAccountsFrom(ns, saList) - .withRepo(repo) - .withConfig(config) - .generate() - - String path = "rbac/rolebinding-test-${ns}.yaml".toString() - File file = new File(repo.getAbsoluteLocalRepoTmpDir(), path) - Map yaml = new YamlSlurper().parse(file) as Map - - assertThat(yaml["metadata"]["name"]).isEqualTo("test") - assertThat(yaml["metadata"]["namespace"]).isEqualTo(ns) - - List names = yaml["subjects"].collect { it['name'] as String } - assertThat(names).containsExactlyInAnyOrderElementsOf(saList) - - List namespaces = yaml["subjects"].collect { it['namespace'] as String } - assertThat(namespaces).containsOnly(ns) - - assertThat(yaml["roleRef"]["name"]).isEqualTo("test") - assertThat(yaml["roleRef"]["kind"]).isEqualTo("Role") - } - - @Test - void 'rendered role yaml contains correct metadata'() { - String name = "myrole" - String ns = "custom-ns" - - new RbacDefinition(Role.Variant.ARGOCD) - .withName(name) - .withNamespace(ns) - .withServiceAccountsFrom(ns, ["sa1"]) - .withRepo(repo) - .withConfig(config) - .generate() - - String path = "rbac/role-${name}-${ns}.yaml".toString() - File file = new File(repo.getAbsoluteLocalRepoTmpDir(), path) - Map yaml = new YamlSlurper().parse(file) as Map - - assertThat(yaml["metadata"]["name"]).isEqualTo(name) - assertThat(yaml["metadata"]["namespace"]).isEqualTo(ns) - } - - @Test - void 'renders node access rules in argocd-role only when not on OpenShift'() { - config.application.openshift = false - - GitRepo tempRepo = new GitRepo(config, null, "rbac-test", new FileSystemUtils()) - - new RbacDefinition(Role.Variant.ARGOCD) - .withName("nodecheck") - .withNamespace("monitoring") - .withServiceAccountsFrom("monitoring", ["sa1"]) - .withRepo(tempRepo) - .withConfig(config) - .generate() - - File roleFile = new File(tempRepo.getAbsoluteLocalRepoTmpDir(), "rbac/role-nodecheck-monitoring.yaml") - Map yaml = new YamlSlurper().parse(roleFile) as Map - List rules = yaml["rules"] as List - - assertThat(rules).anyMatch { rule -> - List resources = rule["resources"] as List - List verbs = rule["verbs"] as List - resources.containsAll(["nodes", "nodes/metrics"]) && verbs.containsAll(["get", "list", "watch"]) - } - } - - @Test - void 'does not render node access rules in argocd-role when on OpenShift'() { - config.application.openshift = true - - GitRepo tempRepo = new GitRepo(config, null, "rbac-test", new FileSystemUtils()) - - new RbacDefinition(Role.Variant.ARGOCD) - .withName("nodecheck") - .withNamespace("monitoring") - .withServiceAccountsFrom("monitoring", ["sa1"]) - .withRepo(tempRepo) - .withConfig(config) - .generate() - - File roleFile = new File(tempRepo.getAbsoluteLocalRepoTmpDir(), "rbac/role-nodecheck-monitoring.yaml") - Map yaml = new YamlSlurper().parse(roleFile) as Map - List rules = yaml["rules"] as List - - assertThat(rules).noneMatch { rule -> - List resources = rule["resources"] as List - resources.contains("nodes") && resources.contains("nodes/metrics") - } - } - - @Test - void 'fails if config is not set'() { - def ex = assertThrows(IllegalArgumentException) { - new RbacDefinition(Role.Variant.ARGOCD) - .withName("failtest") - .withNamespace("ns") - .withServiceAccountsFrom("ns", ["sa"]) - .withRepo(repo) - .generate() - } - - assertThat(ex.message).contains("Config must not be null") - // oder je nach deiner tatsächlichen Exception-Message - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/TestK8sHelper.groovy b/src/test/groovy/com/cloudogu/gitops/integration/TestK8sHelper.groovy deleted file mode 100644 index b901efa7b..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/TestK8sHelper.groovy +++ /dev/null @@ -1,373 +0,0 @@ -package com.cloudogu.gitops.integration - -import static org.assertj.core.api.Assertions.fail - -import java.nio.charset.StandardCharsets -import java.util.concurrent.CountDownLatch -import java.util.concurrent.TimeUnit -import java.util.concurrent.atomic.AtomicReference -import groovy.util.logging.Slf4j - -import io.fabric8.kubernetes.api.model.ContainerStatus -import io.fabric8.kubernetes.api.model.Namespace -import io.fabric8.kubernetes.api.model.Pod -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.KubernetesClientBuilder -import io.fabric8.kubernetes.client.KubernetesClientException -import io.fabric8.kubernetes.client.dsl.ExecListener -import io.fabric8.kubernetes.client.dsl.ExecWatch -import org.awaitility.Awaitility - -/** - * This class contains helper methods for k8s communication.*/ -@Slf4j -class TestK8sHelper { - - static final int DEFAULT_WAIT_MINUTES = 5 - static final int DEFAULT_POLL_SECONDS = 5 - static final String RUNNING = 'Running' - static final String FAILED = 'Failed' - static final String SUCCEEDED = 'Succeeded' - static final String COMPLETED = 'Completed' - static final Set FATAL_CONTAINER_WAITING_REASONS = ['CrashLoopBackOff', - 'CreateContainerConfigError', - 'CreateContainerError', - 'ErrImagePull', - 'ImageInspectError', - 'ImagePullBackOff', - 'InvalidImageName', - 'RunContainerError'] as Set - - /** - * This method logs Namespace and contining Pods to namespace.*/ - static void dumpNamespacesAndPods() { - StringBuffer sb = new StringBuffer('##### K8s Dump ##### \n') - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - def pods = client.pods().inAnyNamespace().list().getItems() - - // sort: namespace, pod-name - pods.sort { a, b -> (a.metadata?.namespace <=> b.metadata?.namespace) ?: (a.metadata?.name <=> b.metadata?.name) - } - - // group by namespace - def podsByNs = pods.groupBy { it.metadata?.namespace ?: "" } - - podsByNs.each { ns, nsPods -> - sb.append("\n=== Namespace: ${ns} (${nsPods.size()}) ===\n") - nsPods.each { pod -> - def name = pod.metadata?.name - def phase = pod.status?.phase - def node = pod.spec?.nodeName ?: "-" - def startTime = pod.status?.startTime ?: "-" - def restarts = (pod.status?.containerStatuses ?: []).sum { it?.restartCount ?: 0 } ?: 0 - - sb.append(String.format(" %-60s phase=%-10s restarts=%-3s node=%-25s start=%s", - name, phase, restarts, node, startTime)) - sb.append("\n") - } - } - } - log.info sb.toString() - } - - /** - * Executes command on container and returns result. - * @param client - * @param ns - * @param pod - * @param container - * @param cmd - * @return - */ - static String execAndGetStdout(KubernetesClient client, - String ns, - String pod, - String container, - String... cmd) { - - ByteArrayOutputStream out = new ByteArrayOutputStream() - ByteArrayOutputStream err = new ByteArrayOutputStream() - - CountDownLatch finished = new CountDownLatch(1) - AtomicReference failure = new AtomicReference<>() - - ExecListener listener = new ExecListener() { - - @Override - void onClose(int code, String reason) { - finished.countDown() - } - } - - try (ExecWatch watch = client.pods() - .inNamespace(ns) - .withName(pod) - .inContainer(container) - .writingOutput(out) - .writingError(err) - .usingListener(listener) - .exec(cmd)) { - - Awaitility.await() - .atMost(5, TimeUnit.MINUTES) - .pollInterval(500, TimeUnit.MILLISECONDS) - .until(() -> finished.getCount() == 0) - - } catch (Exception e) { - throw new RuntimeException("Exec failed/timeout for pod " + ns + "/" + pod, e) - } - - if (failure.get() != null) { - throw new RuntimeException("Exec failure", failure.get()) - } - - String stderr = err.toString(StandardCharsets.UTF_8) - if (!stderr.isBlank()) { - log.error(stderr) - throw new RuntimeException(stderr) - } - - return out.toString(StandardCharsets.UTF_8) - } - - /** - * Checks the current Kubernetes state once and verifies that every matching pod is running. - * Use a waitFor... variant when the tested resource may still be rolling out. - * @param namespace - * @param podNameStartsWith optional pod name prefix. Empty string matches all pods in the namespace. - */ - static boolean checkAllPodsRunningInNamespace(String namespace, String podNameStartsWith = '') { - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - // Check Pod - List actualPods = client.pods().inNamespace(namespace).list().items.findAll { Pod pod -> pod.metadata.name.startsWith(podNameStartsWith) - } - assert !actualPods.empty: "No pods found in namespace: ${namespace} with name ${podNameStartsWith}" - failOnFatalPods(namespace, actualPods) - List notRunningPods = actualPods.findAll { Pod pod -> !isPodRunning(pod) } - - assert notRunningPods.empty: "These pods in ${namespace} are not yet running: ${describePods(notRunningPods)}" - return true - } catch (KubernetesClientException ex) { - fail('Unexpected Kubernetes exception', ex) - return false - } - } - - /** - * Waits until at least one matching pod exists and all matching pods are running. */ - static boolean waitForAllPodsRunningInNamespace(String namespace, - String podNameStartsWith = '', - int timeout = DEFAULT_WAIT_MINUTES, - TimeUnit timeoutUnit = TimeUnit.MINUTES) { - Awaitility.await() - .atMost(timeout, timeoutUnit) - .pollInterval(DEFAULT_POLL_SECONDS, TimeUnit.SECONDS) - .untilAsserted { - checkAllPodsRunningInNamespace(namespace, podNameStartsWith) - } - return true - } - - /** - * Checks the current Kubernetes state once and verifies one running pod for each expected name prefix. - * Extra pods in the namespace are ignored, which keeps the check stable during rollouts. */ - static boolean checkPodPrefixesRunningInNamespace(String namespace, List expectedPodPrefixes) { - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - List actualPods = client.pods().inNamespace(namespace).list().items - expectedPodPrefixes.each { String prefix -> - List matchingPods = actualPods.findAll { Pod pod -> pod.metadata.name.startsWith(prefix) } - failIfOnlyFatalPodsMatch(namespace, prefix, matchingPods) - } - - List missingPods = expectedPodPrefixes.findAll { String prefix -> !actualPods.any { Pod pod -> pod.metadata.name.startsWith(prefix) } - } - assert missingPods.empty: "Missing these pods in ${namespace}: ${missingPods}" - - List notRunningPodPrefixes = expectedPodPrefixes.findAll { String prefix -> - List matchingPods = actualPods.findAll { Pod pod -> pod.metadata.name.startsWith(prefix) } - !matchingPods.any { Pod pod -> isPodRunning(pod) } - } - assert notRunningPodPrefixes.empty: "No running pod found in ${namespace} for: ${notRunningPodPrefixes}. Current pods: ${describePods(actualPods)}" - return true - } catch (KubernetesClientException ex) { - fail('Unexpected Kubernetes exception', ex) - return false - } - } - - /** - * Waits until each expected pod name prefix has at least one running pod. */ - static boolean waitForPodPrefixesRunningInNamespace(String namespace, - List expectedPodPrefixes, - int timeout = DEFAULT_WAIT_MINUTES, - TimeUnit timeoutUnit = TimeUnit.MINUTES) { - Awaitility.await() - .atMost(timeout, timeoutUnit) - .pollInterval(DEFAULT_POLL_SECONDS, TimeUnit.SECONDS) - .untilAsserted { - checkPodPrefixesRunningInNamespace(namespace, expectedPodPrefixes) - } - return true - } - - /** - * Checks the current Kubernetes state once using named pod matchers. - * Use this when simple prefixes are ambiguous, for example when one pod name is a prefix of another. */ - static boolean checkPodsMatchingRunningInNamespace(String namespace, Map> expectedPods) { - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - List actualPods = client.pods().inNamespace(namespace).list().items - expectedPods.each { String expectedPod, Closure podNameMatches -> - List matchingPods = actualPods.findAll { Pod pod -> podNameMatches.call(pod.metadata.name) } - failIfOnlyFatalPodsMatch(namespace, expectedPod, matchingPods) - } - - List missingPods = expectedPods.findAll { - String expectedPod, Closure podNameMatches -> !actualPods.any { Pod pod -> podNameMatches.call(pod.metadata.name) } - }.keySet() as List - assert missingPods.empty: "Missing these pods in ${namespace}: ${missingPods}" - - List notRunningPods = expectedPods.findAll { String expectedPod, Closure podNameMatches -> - List matchingPods = actualPods.findAll { Pod pod -> podNameMatches.call(pod.metadata.name) } - !matchingPods.any { Pod pod -> isPodRunning(pod) } - }.keySet() as List - assert notRunningPods.empty: "No running pod found in ${namespace} for: ${notRunningPods}. Current pods: ${describePods(actualPods)}" - return true - } catch (KubernetesClientException ex) { - fail('Unexpected Kubernetes exception', ex) - return false - } - } - - /** - * Waits until every named pod matcher resolves to at least one running pod. */ - static boolean waitForPodsMatchingRunningInNamespace(String namespace, - Map> expectedPods, - int timeout = DEFAULT_WAIT_MINUTES, - TimeUnit timeoutUnit = TimeUnit.MINUTES) { - Awaitility.await() - .atMost(timeout, timeoutUnit) - .pollInterval(DEFAULT_POLL_SECONDS, TimeUnit.SECONDS) - .untilAsserted { - checkPodsMatchingRunningInNamespace(namespace, expectedPods) - } - return true - } - - /** - * Checks the current Kubernetes state once and verifies that all expected namespaces exist. */ - static boolean checkNamespacesExist(List expectedNamespaces) { - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - List currentNamespaces = client.namespaces().list().items - List missingNamespaces = expectedNamespaces.findAll { - String expectedNamespace -> !currentNamespaces.any { Namespace currentNamespace -> currentNamespace.metadata.name == expectedNamespace } - } - assert missingNamespaces.empty: "Missing these Namespaces: ${missingNamespaces}" - return true - } catch (KubernetesClientException ex) { - fail('Unexpected Kubernetes exception', ex) - return false - } - } - - /** - * Waits until all expected namespaces exist. */ - static boolean waitForNamespaces(List expectedNamespaces, - int timeout = DEFAULT_WAIT_MINUTES, - TimeUnit timeoutUnit = TimeUnit.MINUTES) { - Awaitility.await() - .atMost(timeout, timeoutUnit) - .pollInterval(DEFAULT_POLL_SECONDS, TimeUnit.SECONDS) - .untilAsserted { - checkNamespacesExist(expectedNamespaces) - } - return true - } - - private static void failOnFatalPods(String namespace, Collection pods) { - Collection fatalPods = pods.findAll { Pod pod -> isPodFatal(pod) } - if (!fatalPods.empty) { - throw new IllegalStateException("Pods in ${namespace} reached a terminal or unrecoverable state: ${describePods(fatalPods)}") - } - } - - private static void failIfOnlyFatalPodsMatch(String namespace, String expectedPod, Collection matchingPods) { - if (matchingPods.empty || matchingPods.any { Pod pod -> isPodRunning(pod) }) { - return - } - - if (matchingPods.every { Pod pod -> isPodFatal(pod) }) { - throw new IllegalStateException("No recoverable pod found in ${namespace} for ${expectedPod}. Matching pods: ${describePods(matchingPods)}") - } - } - - private static boolean isPodRunning(Pod pod) { - return ( pod.status?.phase == RUNNING || pod.status?.phase == COMPLETED ) && !hasFatalContainerState(pod) - } - - private static boolean isPodFatal(Pod pod) { - String phase = pod.status?.phase - return phase == FAILED || phase == SUCCEEDED || hasFatalContainerState(pod) - } - - private static boolean hasFatalContainerState(Pod pod) { - return containerStatusesFor(pod).any { ContainerStatus status -> - def waiting = status.state?.waiting - def terminated = status.getState()?.getTerminated() - (waiting != null && FATAL_CONTAINER_WAITING_REASONS.contains(waiting.reason)) || (terminated != null && terminated.exitCode != null && terminated.exitCode != 0) - } - } - - private static List containerStatusesFor(Pod pod) { - List statuses = [] - statuses.addAll(pod.status?.initContainerStatuses ?: []) - statuses.addAll(pod.status?.containerStatuses ?: []) - return statuses - } - - private static String describePods(Collection pods) { - return pods.collect { Pod pod -> - String podName = pod.getMetadata().getName() - String phase = pod.getStatus()?.getPhase() ?: "" - List containerStatuses = pod.getStatus()?.getContainerStatuses() - String readyContainers = containerStatuses == null ? '0/0' : - "${containerStatuses.count { ContainerStatus status -> Boolean.TRUE == status.getReady() }}/${containerStatuses.size()}" - String details = podProblemDetails(pod) - "${podName}:${phase}:ready=${readyContainers}${details ? ":${details}" : ""}" - }.join(', ') - } - - private static String podProblemDetails(Pod pod) { - List details = [] - if (pod.status?.reason) { - details << 'reason=' + pod.status.reason - } - if (pod.status?.message) { - details << 'message=' + shorten(pod.status.message) - } - containerStatusesFor(pod).each { ContainerStatus status -> - String containerState = describeContainerState(status) - if (containerState) { - details << containerState - } - } - return details.empty ? '' : "details=[${details.join('; ')}]" - } - - private static String describeContainerState(ContainerStatus status) { - def waiting = status.state?.waiting - if (waiting != null) { - return "container=${status.name} waiting=${waiting.reason ?: ''}${waiting.message ? " message=${shorten(waiting.message)}" : ''}" - } - - def terminated = status.getState()?.getTerminated() - if (terminated != null) { - return "container=${status.name} terminated=${terminated.reason ?: ''} exit=${terminated.exitCode}" - } - - return null - } - - private static String shorten(String value) { - return value.length() <= 160 ? value : "${value.take(157)}..." - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/features/CertManagerTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/features/CertManagerTestIT.groovy deleted file mode 100644 index 4de72c781..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/features/CertManagerTestIT.groovy +++ /dev/null @@ -1,61 +0,0 @@ -package com.cloudogu.gitops.integration.features - -import com.cloudogu.gitops.integration.TestK8sHelper - -import groovy.util.logging.Slf4j - -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This class checks if cert-manager is started well. - * Cert-Manager contains own namespace ('cert-manager') which owns and 3 Pods:*/ -@Slf4j -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full") -//TODO: why not in ArgoCD Operator? Clearify -class CertManagerTestIT extends KubenetesApiTestSetup { - - String namespace = 'cert-manager' - - @Override - boolean isReadyToStartTests() { - try { - return TestK8sHelper.checkPodsMatchingRunningInNamespace(namespace, expectedCertManagerPods()) - } catch (AssertionError ignored) { - return false - } - } - - @BeforeAll - static void labelTest() { - println "###### CERT-MANAGER ######" - } - - @Test - void ensureNamespaceExists() { - TestK8sHelper.waitForNamespaces([namespace]) - } - - @Test - void ensureAllCertManagerPodsAreExist() { - TestK8sHelper.waitForPodsMatchingRunningInNamespace(namespace, expectedCertManagerPods()) - } - - @Test - void ensureExpectedCertManagerPodsAreRunning() { - TestK8sHelper.waitForPodsMatchingRunningInNamespace(namespace, expectedCertManagerPods()) - } - - private static Map> expectedCertManagerPods() { - [ - 'cert-manager' : { String podName -> - podName.startsWith('cert-manager-') && - !podName.startsWith('cert-manager-cainjector') && - !podName.startsWith('cert-manager-webhook') - }, - 'cert-manager-cainjector': { String podName -> podName.startsWith('cert-manager-cainjector') }, - 'cert-manager-webhook' : { String podName -> podName.startsWith('cert-manager-webhook') }, - ] - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/features/KubenetesApiTestSetup.groovy b/src/test/groovy/com/cloudogu/gitops/integration/features/KubenetesApiTestSetup.groovy deleted file mode 100644 index 202332365..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/features/KubenetesApiTestSetup.groovy +++ /dev/null @@ -1,86 +0,0 @@ -package com.cloudogu.gitops.integration.features - -import static org.assertj.core.api.Assertions.assertThat -import static org.assertj.core.api.Assertions.fail - -import java.time.Duration -import java.time.Instant -import java.util.function.Supplier - -import io.kubernetes.client.openapi.ApiClient -import io.kubernetes.client.openapi.Configuration -import io.kubernetes.client.openapi.apis.CoreV1Api -import io.kubernetes.client.util.ClientBuilder -import io.kubernetes.client.util.KubeConfig -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.BeforeEach - -abstract class KubenetesApiTestSetup { - static String kubeConfigPath - CoreV1Api api - int TIME_TO_WAIT = 12 - int RETRY_SECONDS = 30 - - /** - * Gets path to kubeconfig*/ - @BeforeAll - static void setupKubeconfig() { - kubeConfigPath = System.getenv("HOME") + "/.kube/config" - if (!new File(kubeConfigPath).exists()) { - kubeConfigPath = System.getenv("KUBECONFIG") - } - assertThat(kubeConfigPath) isNotBlank() - } - - /** - * establish connection to kubernetes and create API to use.*/ - @BeforeEach - void setupConnection() { - ApiClient client = - ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath))).build() - // set the global default api-client to the out-of-cluster one from above - Configuration.setDefaultApiClient(client) - - // the CoreV1Api loads default api-client from global configuration. - api = new CoreV1Api() - waitForCondition(() -> waitingCondition(), - maxWaitTimeInMinutes(TIME_TO_WAIT), - pollIntervallSeconds(RETRY_SECONDS)) - } - - static void waitForCondition(Supplier condition, Duration timeout, Duration pollInterval) { - Instant end = Instant.now().plus(timeout) - while (Instant.now().isBefore(end)) { - if (condition.get()) { - return - } - try { - Thread.sleep(pollInterval.toMillis()) - } catch (InterruptedException e) { - Thread.currentThread().interrupt() - throw new RuntimeException("break polling", e) - } - } - fail('Wait condition not fulfilled in time') - } - - private Duration pollIntervallSeconds(int time) { - return Duration.ofSeconds(time) - } - - private Duration maxWaitTimeInMinutes(int time) { - return Duration.ofMinutes(time) - } - - boolean waitingCondition() { - println 'waiting for pods' - return isReadyToStartTests() - } - - /** - * This condition is to override, if test has to wait, i.e. ArgoCD has to do its GitOps magic. - * @return - */ - - abstract boolean isReadyToStartTests() -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/features/MonitoringTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/features/MonitoringTestIT.groovy deleted file mode 100644 index 8056cff4b..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/features/MonitoringTestIT.groovy +++ /dev/null @@ -1,74 +0,0 @@ -package com.cloudogu.gitops.integration.features - -import static org.assertj.core.api.Assertions.assertThat - -import com.cloudogu.gitops.integration.TestK8sHelper - -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Disabled -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This class checks if Prometheus is started well. - * Prometheus contains own namespace ('monitoring') which owns and 3 Pods: - * - Grafana - * - Operator - * - prometheus-stack*/ -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full") -class MonitoringTestIT extends KubenetesApiTestSetup { - - String namespace = 'monitoring' - String grafanaPod = 'kube-prometheus-stack-grafana' - String operatorPod = 'kube-prometheus-stack-operator' - String prometheusPod = 'prometheus-kube-prometheus-stack-prometheus' - - @Override - boolean isReadyToStartTests() { - try { - return TestK8sHelper.checkAllPodsRunningInNamespace(namespace, grafanaPod) - } catch (AssertionError ignored) { - return false - } - } - - @BeforeAll - static void labelTest() { - println "###### PROMETHEUS ######" - } - - @Test - void ensureNamespaceExists() { - TestK8sHelper.waitForNamespaces([namespace]) - } - - @Test - void ensureGrafanaIsStarted() { - TestK8sHelper.waitForAllPodsRunningInNamespace(namespace, grafanaPod) - } - - @Test - void ensureOperatorIsStarted() { - TestK8sHelper.waitForAllPodsRunningInNamespace(namespace, operatorPod) - } - - @Disabled("not start on jenkins") - @Test - void ensureMonitoringIsStarted() { - - def pods = api.listNamespacedPod(namespace).execute() - assertThat(pods).isNotNull() - assertThat(pods.getItems().isEmpty()).isFalse() - - def prometheus = pods.items.find { it.getMetadata().name.contains(prometheusPod) } - assertThat(prometheus).isNotNull() - assertThat(prometheus.status.phase).isEqualTo("Running") - } - - @Disabled("jenkins got only 2") - @Test - void ensureNamespaceGot3Pods() { - def pods = api.listNamespacedPod(namespace).execute() - assertThat(pods.getItems().size()).isEqualTo(3) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/ArgoCDOperatorProfileTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/ArgoCDOperatorProfileTestIT.groovy deleted file mode 100644 index 615d8b445..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/ArgoCDOperatorProfileTestIT.groovy +++ /dev/null @@ -1,79 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import static org.assertj.core.api.Assertions.assertThat -import static org.assertj.core.api.Assertions.fail - -import com.cloudogu.gitops.integration.TestK8sHelper - -import java.util.concurrent.TimeUnit - -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.KubernetesClientBuilder -import io.fabric8.kubernetes.client.KubernetesClientException -import org.awaitility.Awaitility -import org.awaitility.core.ConditionTimeoutException -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This tests can only be successfull, if one of theses profiles used. - * - * To run locally: add -Dmicronaut.environments=operator-full to your execute configuration*/ - -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "operator-full|operator-minimal") -class ArgoCDOperatorProfileTestIT extends ProfileTestSetup { - - static String namespaceOperator = 'argocd-operator-system' - static String namespaceArgocd = 'argocd' - - @BeforeAll - static void labelTest() { - println "###### Integration ArgoCD Operator test ######" - try { - Awaitility.await() - .atMost(40, TimeUnit.MINUTES) - .pollInterval(5, TimeUnit.SECONDS) - .untilAsserted { - assert TestK8sHelper.checkAllPodsRunningInNamespace(namespaceOperator, 'argocd-operator-controller') && TestK8sHelper.checkAllPodsRunningInNamespace(namespaceArgocd, 'argocd-server') - } - } catch (ConditionTimeoutException timeoutEx) { - TestK8sHelper.dumpNamespacesAndPods() - fail('Cluster not ready, sth false.') - } - } - - @Test - void ensureNamespaceExists() { - - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - - def argocdNamespace = client.namespaces().withName(namespaceOperator).get() - - assertThat(argocdNamespace).isNotNull() - assert namespaceOperator.startsWith(argocdNamespace.metadata.name) - - } catch (KubernetesClientException ex) { - // Handle exception - assert fail("not expected exception was thrown. ", ex) - } - - } - - @Test - void ensureOperatorNamespaceExists() { - - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - - def argocdNamespace = client.namespaces().withName(namespaceArgocd).get() - - assertThat(argocdNamespace).isNotNull() - - } catch (KubernetesClientException ex) { - // Handle exception - assert fail("not expected exception was thrown. ", ex) - } - - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/ArgoCDProfileTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/ArgoCDProfileTestIT.groovy deleted file mode 100644 index 06ab8611c..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/ArgoCDProfileTestIT.groovy +++ /dev/null @@ -1,44 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import com.cloudogu.gitops.integration.TestK8sHelper - -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This tests can only be successfull, if one of theses profiles used. - * - * To run locally: add -Dmicronaut.environments=full to your execute configuration*/ - -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full|minimal|operator-full|content-examples|operator-minimal|operator-content-examples") -class ArgoCDProfileTestIT extends ProfileTestSetup { - - String namespace = 'argocd' - - @BeforeAll - static void labelTest() { - println "###### Integration ArgoCD test ######" - } - - @Test - void ensureNamespaceExists() { - TestK8sHelper.waitForNamespaces([namespace], 40) - } - - /** - * chechs that ArgoCD pods running **/ - @Test - void ensureArgoCDIsOnlineAndPodsAreRunning() { - String expectedPod1 = "argocd-application-controller" - String expectedPod2 = "argocd-applicationset-controller" - // String expectedPod3 = "argocd-notifications-controller" // not stable - String expectedPod4 = "argocd-redis" - String expectedPod5 = "argocd-repo-server" - String expectedPod6 = "argocd-server" - - List expectedPods = [expectedPod1, expectedPod2, /* expectedPod3,*/ expectedPod4, expectedPod5, expectedPod6,] - - TestK8sHelper.waitForPodPrefixesRunningInNamespace(namespace, expectedPods, 40) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/FullProfileTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/FullProfileTestIT.groovy deleted file mode 100644 index 0f1463272..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/FullProfileTestIT.groovy +++ /dev/null @@ -1,116 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import com.cloudogu.gitops.integration.TestK8sHelper - -import java.util.concurrent.TimeUnit -import groovy.util.logging.Slf4j - -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This test ensures all Pods and Namespaces are available, runnning at a startet GOP with - more or less - defaulöt values. - * - * * To run locally: add -Dmicronaut.environments=full to your execute configuration - **/ -@Slf4j -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full") -class FullProfileTestIT extends ProfileTestSetup { - - /** - * Gets path to kubeconfig */ - static final String EXAMPLE_APPS_NAMESPACE = 'example-apps-staging' - - @BeforeAll - static void labelMyTest() { - log.info '########### K8S SMOKE TESTS PROFILE full ###########' - waitUntilAllPodsRunning() - } - - private static void waitUntilAllPodsRunning() { - // if cert-manager is online, argocd is online, too! - TestK8sHelper.waitForAllPodsRunningInNamespace(EXAMPLE_APPS_NAMESPACE, "", 40, TimeUnit.MINUTES) - } - - @Test - void ensureJenkinsPodIsStarted() { - TestK8sHelper.waitForAllPodsRunningInNamespace('jenkins', 'jenkins') - } - - @Test - void ensureArgoCDIsOnlineAndPodsAreRunning() { - String expectedPod1 = "argocd-application-controller" - String expectedPod2 = "argocd-applicationset-controller" - // String expectedPod3 = "argocd-notifications-controller" // not stable - String expectedPod4 = "argocd-redis" - String expectedPod5 = "argocd-repo-server" - String expectedPod6 = "argocd-server" - - List expectedPods = [expectedPod1, expectedPod2, /* expectedPod3,*/ expectedPod4, expectedPod5, expectedPod6,] - - TestK8sHelper.waitForPodPrefixesRunningInNamespace('argocd', expectedPods) - } - - @Test - void ensureScmmPodIsStarted() { - - TestK8sHelper.waitForAllPodsRunningInNamespace('scm-manager') - } - - @Test - void ensureNamespacesExists() { - List expectedNamespaces = ["argocd", - "cert-manager", - "jenkins", - "registry", - "scm-manager", - "default", - "example-apps-production", - "example-apps-staging", - "ingress", - "kube-node-lease", - "kube-public", - "kube-system", - "monitoring", - "secrets"] as List - - TestK8sHelper.waitForNamespaces(expectedNamespaces) - } - - /** - * tests searches for ingress services and ensure ingress is used as loadbalancer*/ - @Test - void ensureIngressIsOnline() { - TestK8sHelper.waitForAllPodsRunningInNamespace('ingress', 'traefik') - } - - @Test - void ensureCertManagerIsOnline() { - TestK8sHelper.waitForAllPodsRunningInNamespace('cert-manager') - } - - @Test - void ensureVaultIsOnline() { - TestK8sHelper.waitForAllPodsRunningInNamespace('secrets', 'vault-0') - } - - @Test - void ensureRegistryIsOnline() { - TestK8sHelper.waitForAllPodsRunningInNamespace('registry', 'docker-registry') - } - - @Test - void ensureExternalSecretsPodsRunning() { - TestK8sHelper.waitForPodsMatchingRunningInNamespace('secrets', [ - 'external-secrets' : { String podName -> - podName.startsWith('external-secrets-') && - !podName.startsWith('external-secrets-webhook') && - !podName.startsWith('external-secrets-cert-controller') - }, - 'external-secrets-webhook' : { String podName -> podName.startsWith('external-secrets-webhook') }, - 'external-secrets-cert-controller': { String podName -> podName.startsWith('external-secrets-cert-controller') }, - ]) - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/MandantProfileTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/MandantProfileTestIT.groovy deleted file mode 100644 index 7101a3d7d..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/MandantProfileTestIT.groovy +++ /dev/null @@ -1,122 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import static org.assertj.core.api.Assertions.fail - -import com.cloudogu.gitops.integration.TestK8sHelper - -import java.util.concurrent.TimeUnit -import groovy.util.logging.Slf4j - -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.KubernetesClientBuilder -import io.fabric8.kubernetes.client.KubernetesClientException -import org.awaitility.Awaitility -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.DisabledIfSystemProperty -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This test ensures all Pods and Namespaces are available, runnning at a startet GOP with - more or less - defaulöt values. - * - * * To run locally: add -Dmicronaut.environments=full to your execute configuration - **/ -@Slf4j -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") -class MandantProfileTestIT extends ProfileTestSetup { - - /** - * Gets path to kubeconfig */ - static final String RUNNING = "Running" - static final String TENANT_POD_FOR_CONDITION = 'argocd-application-controller' - static final String TENANT_NAMESPACE_ARGOCD = 'tenant1-argocd' - static final String TENANT_NAMESPACE_REGISTRY = 'tenant1-registry' - static final String TENANT_NAMESPACE_SCM = 'tenant1-scm-manager' - - @BeforeAll - static void labelMyTest() { - log.info '########### PROFILE Operator-Mandants ###########' - waitUntilTenantIsReady() - } - - private static void waitUntilTenantIsReady() { - // tenant is created very late after running GOP twice! - Awaitility.await().atMost(40, TimeUnit.MINUTES).pollInterval(5, TimeUnit.SECONDS).untilAsserted { - assert TestK8sHelper.checkAllPodsRunningInNamespace(TENANT_NAMESPACE_REGISTRY, "docker-registry") && TestK8sHelper.checkAllPodsRunningInNamespace(TENANT_NAMESPACE_SCM, 'scmm-') - } - } - - @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") - // just local - @Test - void ensureJenkinsPodIsStartedOnTenant() { - TestK8sHelper.waitForAllPodsRunningInNamespace('tenant1-jenkins', 'jenkins') - } - - @Test - void ensureRegistryPodIsStartedOnTenant() { - TestK8sHelper.waitForAllPodsRunningInNamespace('tenant1-registry', 'docker-registry') - } - - @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") - // just local - @Test - void ensureArgocdPodsAreStartedOnTenant() { - def argocdNamespace = TENANT_NAMESPACE_ARGOCD - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-application-controller') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-applicationset-controller') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-redis') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-repo-server') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-server') - } - - @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") - // just local - @Test - void ensureArgocdPodsAreStartedOnCentral() { - def argocdNamespace = 'argocd' - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-application-controller') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-applicationset-controller') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-redis') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-repo-server') - TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, 'argocd-server') - } - - @Test - void ensureScmmPodIsStarted() { - - TestK8sHelper.waitForAllPodsRunningInNamespace('scm-manager') - } - - @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") - // just local - @Test - void ensureNamespacesExists() { - List expectedNamespaces = ["argocd", - "argocd-operator-system", - "scm-manager", - "default", - "tenant1-argocd", - "tenant1-jenkins", - "tenant1-registry", - "tenant1-example-apps-staging", - "tenant1-example-apps-staging", - "tenant1-scm-manager", - "kube-node-lease", - "kube-public", - "kube-system"] as List - - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - - def currentNames = client.namespaces().list().getItems() - - // 1. Verify all expected pods are present - def missingNamespace = expectedNamespaces.findAll { prefix -> !currentNames.any { it.getMetadata().getName().startsWith(prefix) } - } - assert missingNamespace.isEmpty(): "Missing these Namespace: ${missingNamespace}" - - } catch (KubernetesClientException ex) { - fail("Unexpected Kubernetes exception", ex) - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/PetclinicProfileTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/PetclinicProfileTestIT.groovy deleted file mode 100644 index 2bce01154..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/PetclinicProfileTestIT.groovy +++ /dev/null @@ -1,91 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import static org.assertj.core.api.Assertions.assertThat -import static org.assertj.core.api.Assertions.fail - -import com.cloudogu.gitops.integration.TestK8sHelper - -import java.util.concurrent.TimeUnit -import groovy.util.logging.Slf4j - -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.KubernetesClientBuilder -import io.fabric8.kubernetes.client.KubernetesClientException -import org.awaitility.core.ConditionTimeoutException -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.DisabledIfSystemProperty -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This tests can only be successfull, if one of theses profiles used. - * - * * To run locally: add -Dmicronaut.environments=content-examples to your execute configuration*/ -@Slf4j -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full|operator-full|content-examples") -class PetclinicProfileTestIT extends ProfileTestSetup { - - static String exampleStagingNs = 'example-apps-staging' - - @BeforeAll - static void labelTest() { - println "###### Testing Petclinic ######" - // petclinic need most of time to run. If online, we can start all tests. - try { - TestK8sHelper.waitForAllPodsRunningInNamespace(exampleStagingNs, "", 40, TimeUnit.MINUTES) - } catch (ConditionTimeoutException timeoutEx) { - TestK8sHelper.dumpNamespacesAndPods() - fail('Cluster not ready, sth false.', timeoutEx) - } - } - - @Test - void ensurePetclinicIsRunningOnStages() { - TestK8sHelper.waitForAllPodsRunningInNamespace(exampleStagingNs) - } - - @DisabledIfSystemProperty(named = "micronaut.environments", matches = "full|operator-full|content-examples") - @Test - void ensurePetclinicIngressIsOnline() { - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - def nameOfServiceAndIngress = "spring-petclinic-plain" - // check Ingress - def ingress = client.network() - .v1() - .ingresses() - .inNamespace(exampleStagingNs) - .withName(nameOfServiceAndIngress) - .get() - - assert ingress != null: "Ingress '${nameOfServiceAndIngress}' not found in '${exampleStagingNs}'" - - def hosts = (ingress.spec?.rules ?: []) - .collect { it?.host } - .findAll { it } - - assert hosts.get(0).contains("petclinic") // in this case, petclinic do not care about prefix - } catch (KubernetesClientException ex) { - fail("Unexpected Kubernetes exception", ex) - } - } - - @DisabledIfSystemProperty(named = "micronaut.environments", matches = "full|operator-full|content-examples") - @Test - void ensurePetclinicServidsdsdceIsOnline() { - try (KubernetesClient client = new KubernetesClientBuilder().build()) { - - // Check Service - def nameOfServiceAndIngress = "spring-petclinic-plain" - def service = client.services() - .inNamespace(exampleStagingNs) - .withName(nameOfServiceAndIngress) - .get() - - assertThat(service).isNotNull() - - } catch (KubernetesClientException ex) { - fail("Unexpected Kubernetes exception", ex) - } - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/PrefixProfileTestIT.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/PrefixProfileTestIT.groovy deleted file mode 100644 index 801d68b3c..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/PrefixProfileTestIT.groovy +++ /dev/null @@ -1,73 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import static org.assertj.core.api.Assertions.fail - -import com.cloudogu.gitops.integration.TestK8sHelper - -import java.util.concurrent.TimeUnit -import groovy.util.logging.Slf4j - -import org.awaitility.core.ConditionTimeoutException -import org.junit.jupiter.api.BeforeAll -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.condition.EnabledIfSystemProperty - -/** - * This tests can only be successfull, if one of theses profiles used. - * * To run locally: add -Dmicronaut.environments=full-prefix to your execute configuration*/ -@Slf4j -@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full-prefix") -class PrefixProfileTestIT extends ProfileTestSetup { - // is used for pre-condition - static String exampleStagingNs = 'my-prefix-example-apps-staging' - static String argocdNs = 'my-prefix-argocd' - String scmManagerNs = 'my-prefix-scm-manager' - String registryNs = 'my-prefix-registry' - String ingressNs = 'my-prefix-ingress' - /* Jenking can not start ingress*/ - static String certManagerNs = 'my-prefix-cert-manager' - String jenkinsNs = 'my-prefix-jenkins' - static String monitoringNs = 'my-prefix-monitoring' - String secretsNs = 'my-prefix-secrets' - String exampleProductionNs = 'my-prefix-example-apps-production' - - @BeforeAll - static void labelTest() { - log.info "###### Integration test for Prefix ######" - - try { - TestK8sHelper.waitForAllPodsRunningInNamespace(exampleStagingNs, "", 40, TimeUnit.MINUTES) - } catch (ConditionTimeoutException timeoutEx) { - TestK8sHelper.dumpNamespacesAndPods() - fail('Cluster not ready, sth false.', timeoutEx) - } - } - - @Test - void ensureNamespacesExistWithPrefix() { - List expectedNamespaces = [argocdNs, - scmManagerNs, - registryNs, - ingressNs, - certManagerNs, - jenkinsNs, - monitoringNs, - secretsNs, - exampleProductionNs, - exampleStagingNs] - - TestK8sHelper.waitForNamespaces(expectedNamespaces) - } - - @Test - void ensurePodsAreRunningInPrefixedNamespaces() { - List namespacesToCheck = [argocdNs, - scmManagerNs, - registryNs, - certManagerNs, - monitoringNs] - namespacesToCheck.each { String ns -> - TestK8sHelper.waitForAllPodsRunningInNamespace(ns) - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/integration/profiles/ProfileTestSetup.groovy b/src/test/groovy/com/cloudogu/gitops/integration/profiles/ProfileTestSetup.groovy deleted file mode 100644 index 8e6e2e43b..000000000 --- a/src/test/groovy/com/cloudogu/gitops/integration/profiles/ProfileTestSetup.groovy +++ /dev/null @@ -1,35 +0,0 @@ -package com.cloudogu.gitops.integration.profiles - -import com.cloudogu.gitops.integration.TestK8sHelper - -import groovy.util.logging.Slf4j - -import org.junit.jupiter.api.AfterAll -import org.junit.jupiter.api.extension.ExtensionContext -import org.junit.jupiter.api.extension.RegisterExtension -import org.junit.jupiter.api.extension.TestWatcher - -/** - * Common setup to dump K88s content after failing tests.*/ -@Slf4j -class ProfileTestSetup implements TestWatcher { - - private static boolean anyTestFailed = false - @RegisterExtension - final TestWatcher watcher = this - - @Override - void testFailed(ExtensionContext context, Throwable cause) { - anyTestFailed = true - } - - @AfterAll - static void afterAllOnlyOnFailure() { - // if one test fails, logging is necessary - if (anyTestFailed) { - log.info "############## K8s dump ##############" - TestK8sHelper.dumpNamespacesAndPods() - } - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/TestLogger.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/TestLogger.groovy deleted file mode 100644 index b04b37d58..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/TestLogger.groovy +++ /dev/null @@ -1,73 +0,0 @@ -package com.cloudogu.gitops.testhelper - -import ch.qos.logback.classic.Level -import ch.qos.logback.classic.Logger -import ch.qos.logback.classic.LoggerContext -import ch.qos.logback.classic.spi.ILoggingEvent -import ch.qos.logback.core.read.ListAppender -import org.slf4j.LoggerFactory - -import java.util.stream.Collectors - -class TestLogger { - - private Class loggerInClass - private MemoryAppender memoryAppender - - TestLogger(Class clazz, Level loglevel = Level.DEBUG) { - this.loggerInClass = clazz - Logger logger = (Logger) LoggerFactory.getLogger(loggerInClass) - memoryAppender = new MemoryAppender() - memoryAppender.setContext((LoggerContext) LoggerFactory.getILoggerFactory()) - logger.setLevel(loglevel) - logger.addAppender(memoryAppender) - memoryAppender.start() - } - - void changeLogLevel(Level loglevel) { - Logger logger = (Logger) LoggerFactory.getLogger(loggerInClass) - logger.setLevel(loglevel) - } - - MemoryAppender getLogs() { - return memoryAppender - } -} - -class MemoryAppender extends ListAppender { - - void reset() { - list.clear(); - } - - boolean contains(String string, Level level) { - return list.stream() - .anyMatch(event -> event.toString().contains(string) && event.getLevel().equals(level)); - } - - int countEventsForLogger(String loggerName) { - return (int) list.stream() - .filter(event -> event.getLoggerName().contains(loggerName)) - .count(); - } - - List search(String string) { - return list.stream() - .filter(event -> event.toString().contains(string)) - .collect(Collectors.toList()) as List; - } - - List search(String string, Level level) { - return list.stream() - .filter(event -> event.toString().contains(string) && event.getLevel().equals(level)) - .collect(Collectors.toList()) as List; - } - - int getSize() { - return list.size(); - } - - List getLoggedEvents() { - return Collections.unmodifiableList(list); - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/git/GitHandlerForTests.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/git/GitHandlerForTests.groovy deleted file mode 100644 index 5dfbad32f..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/git/GitHandlerForTests.groovy +++ /dev/null @@ -1,63 +0,0 @@ -package com.cloudogu.gitops.testhelper.git - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.K8sClientForTest -import com.cloudogu.gitops.utils.NetworkingUtils - -import static org.mockito.Mockito.mock - -class GitHandlerForTests extends GitHandler { - private final GitProvider tenantProvider - private final GitProvider centralProvider - - GitHandlerForTests(Config config, GitProvider tenantProvider, GitProvider centralProvider = null) { - super(config, mock(HelmStrategy), new FileSystemUtils(), new K8sClientForTest(), new NetworkingUtils()) - this.tenantProvider = tenantProvider - this.centralProvider = centralProvider - } - - @Override - void enable() { - // Inject the test providers into the base class before running the real logic - this.tenant = tenantProvider - this.central = centralProvider - - // Mirror the production side effect: set namespace for internal SCMM - if (this.config?.scm?.scmManager != null) { - this.config.scm.scmManager.namespace = "${config.application.namePrefix}scm-manager".toString() - } - - // === Run ONLY the repo setup logic (NO provider construction here) === - final String namePrefix = (config?.application?.namePrefix ?: "").trim() - if (this.central) { - setupRepos(this.central, namePrefix) - setupRepos(this.tenant, namePrefix) - } else { - setupRepos(this.tenant, namePrefix) - } - - } - - @Override - void validate() {} - - @Override - GitProvider getTenant() { - return tenantProvider - } - - @Override - GitProvider getCentral() { - return centralProvider - } - - @Override - GitProvider getResourcesScm() { - return centralProvider ?: tenantProvider - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/git/GitlabMock.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/git/GitlabMock.groovy deleted file mode 100644 index 27f071d2c..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/git/GitlabMock.groovy +++ /dev/null @@ -1,86 +0,0 @@ -package com.cloudogu.gitops.testhelper.git - -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope -import com.cloudogu.gitops.infrastructure.git.providers.Scope - -class GitlabMock implements GitProvider { - URI base = new URI("https://example.com/group") - // from config.scm.gitlab.url - String namePrefix = "" - // prefix if you use tenant mode - - final List createdRepos = [] - final List permissionCalls = [] - - @Override - boolean createRepository(String repoTarget, String description, boolean initialize) { - createdRepos << repoTarget - return true - } - - @Override - boolean createRepository(String repoTarget, String description) { - return createRepository(repoTarget, description, true) - } - - @Override - void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { - permissionCalls << [repoTarget: repoTarget, principal: principal, role: role, scope: scope] - } - - @Override - String repoUrl(String repoTarget, RepoUrlScope scope) { - def cleaned = base.toString().replaceAll('/+$', '') - return "${cleaned}/${repoTarget}.git" - } - - @Override - String repoPrefix() { - def cleaned = base.toString().replaceAll('/+$', '') - return "${cleaned}/${namePrefix ?: ''}".toString() - } - - // trivial passthroughs - @Override - URI prometheusMetricsEndpoint() { - return base - } - - @Override - Credentials getCredentials() { - return new Credentials("gitops", "gitops") - } - - @Override - void deleteRepository(String n, String r, boolean p) {} - - @Override - void deleteUser(String name) {} - - @Override - void setDefaultBranch(String target, String branch) {} - - @Override - String getUrl() { - return base.toString() - } - - @Override - String getProtocol() { - return base.scheme - } - - @Override - String getHost() { - return base.host - } - - @Override - String getGitOpsUsername() { - return "gitops" - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/git/ScmManagerMock.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/git/ScmManagerMock.groovy deleted file mode 100644 index bdd74d318..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/git/ScmManagerMock.groovy +++ /dev/null @@ -1,130 +0,0 @@ -package com.cloudogu.gitops.testhelper.git - -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.infrastructure.git.providers.AccessRole -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope -import com.cloudogu.gitops.infrastructure.git.providers.Scope - -/** - * Lightweight test double for ScmManager/GitProvider. - * - Configurable in-cluster and client bases - * - Optional namePrefix to model “tenant” behavior - * - Records createRepository / setRepositoryPermission calls for assertions*/ -class ScmManagerMock implements GitProvider { - - private final Set initOnceRepos = [] as Set - private final Map createCalls = [:].withDefault { 0 } - - void initOnceRepo(String fullName) { - initOnceRepos << fullName - } - - void clearInitOnce() { - initOnceRepos.clear(); createCalls.clear() - } - - // --- configurable --- - URI inClusterBase = new URI("http://scmm.scm-manager.svc.cluster.local/scm") - URI clientBase = new URI("http://localhost:8080/scm") - String namePrefix = "" - // e.g., "fv40-" for tenant mode - Credentials credentials = new Credentials("gitops", "gitops") - String gitOpsUsername = "gitops" - URI prometheus = new URI("http://localhost:8080/scm/api/v2/metrics/prometheus") - - // --- call recordings for assertions --- - final List createdRepos = [] - final List permissionCalls = [] - /** Optional sequence to control createRepository() return values per call */ - List nextCreateResults = [] - // empty -> default true - - @Override - boolean createRepository(String repoTarget, String description, boolean initialize) { - if (initOnceRepos.contains(repoTarget)) { - return ++createCalls[repoTarget] == 1 // 1. call true, then false - } - createdRepos << repoTarget - // Pretend repository was created successfully. - // If you need idempotency checks, examine createdRepos.count(repoTarget) in your tests. - return nextCreateResults ? nextCreateResults.remove(0) : true - } - - @Override - void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { - permissionCalls << [repoTarget: repoTarget, - principal : principal, - role : role, - scope : scope] - } - - /** …/scm/repo// */ - @Override - String repoUrl(String repoTarget, RepoUrlScope scope) { - URI base = (scope == RepoUrlScope.CLIENT) ? clientBase : inClusterBase - def cleanedBase = withoutTrailingSlash(base).toString() - return "${cleanedBase}/repo/${repoTarget}" - } - - /** In-cluster repo prefix: …/scm/repo/[] */ - @Override - String repoPrefix() { - def base = withoutTrailingSlash(inClusterBase).toString() - def prefix = (namePrefix ?: "").strip() - return "${base}/repo/${prefix}" - } - - @Override - Credentials getCredentials() { - return credentials - } - - /** …/scm/api/v2/metrics/prometheus */ - @Override - URI prometheusMetricsEndpoint() { - return prometheus - } - - @Override - void deleteRepository(String namespace, String repository, boolean prefixNamespace) { - - } - - @Override - void deleteUser(String name) { - - } - - @Override - void setDefaultBranch(String repoTarget, String branch) { - - } - - /** In-cluster base …/scm (without trailing slash) */ - @Override - String getUrl() { - return inClusterBase.toString() - } - - @Override - String getProtocol() { - return inClusterBase.scheme // e.g., "http" - } - - @Override - String getHost() { - return inClusterBase.host // e.g., "scmm.ns.svc.cluster.local" - } - - @Override - String getGitOpsUsername() { - return gitOpsUsername - } - - // --- helpers --- - private static URI withoutTrailingSlash(URI uri) { - def s = uri.toString() - return new URI(s.endsWith("/") ? s.substring(0, s.length() - 1) : s) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestGitProvider.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestGitProvider.groovy deleted file mode 100644 index 2c7348c22..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestGitProvider.groovy +++ /dev/null @@ -1,22 +0,0 @@ -package com.cloudogu.gitops.testhelper.git - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider - -class TestGitProvider { - static Map buildProviders(Config cfg) { - if (cfg.scm.scmProviderType?.toString() == 'GITLAB') { - def gitlab = new GitlabMock(base: new URI(cfg.scm.gitlab.url), - namePrefix: cfg.application.namePrefix) - return [tenant: gitlab, central: cfg.multiTenant.useDedicatedInstance ? gitlab : null] - } - - def serviceDns = "http://scmm.${cfg.application.namePrefix}scm-manager.svc.cluster.local/scm" - String tenantInCluster = (cfg.scm.scmManager?.url ?: serviceDns) as String - String centralInCluster = (cfg.multiTenant.scmManager?.url ?: tenantInCluster) as String - - def tenant = new ScmManagerMock(inClusterBase: new URI(tenantInCluster), namePrefix: cfg.application.namePrefix) - def central = cfg.multiTenant.useDedicatedInstance ? new ScmManagerMock(inClusterBase: new URI(centralInCluster), namePrefix: cfg.application.namePrefix) : null - return [tenant: tenant, central: central] - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestGitRepoFactory.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestGitRepoFactory.groovy deleted file mode 100644 index 0c4c99820..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestGitRepoFactory.groovy +++ /dev/null @@ -1,64 +0,0 @@ -package com.cloudogu.gitops.testhelper.git - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.GitRepoFactory -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.utils.FileSystemUtils -import org.apache.commons.io.FileUtils - -import static org.mockito.Mockito.doAnswer -import static org.mockito.Mockito.spy - -class TestGitRepoFactory extends GitRepoFactory { - Map repos = [:] - GitProvider defaultProvider - - TestGitRepoFactory(Config config, FileSystemUtils fileSystemUtils) { - super(config, fileSystemUtils) - } - - @Override - GitRepo getRepo(String repoTarget, GitProvider scm) { - def effectiveProvider = scm ?: defaultProvider - - if (!effectiveProvider) { - throw new IllegalStateException("No GitProvider provided for repo '${repoTarget}' and defaultProvider is null.") - } - - if (repos[repoTarget]) { - return repos[repoTarget] - } - - GitRepo repoNew = new GitRepo(config, scm, repoTarget, fileSystemUtils) { - String remoteGitRepoUrl = '' - - @Override - String getGitRepositoryUrl() { - if (!remoteGitRepoUrl) { - - def tempDir = File.createTempDir('gitops-playground-repocopy') - tempDir.deleteOnExit() - def originalRepo = System.getProperty("user.dir") + "/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/" - - FileUtils.copyDirectory(new File(originalRepo), tempDir) - remoteGitRepoUrl = 'file://' + tempDir.absolutePath - } - return remoteGitRepoUrl - } - } - - GitRepo spyRepo = spy(repoNew) - - // Test-only: remove local clone target before cloning to avoid "not empty" errors - doAnswer { invocation -> - File target = new File(spyRepo.absoluteLocalRepoTmpDir) - if (target?.exists()) { - FileUtils.deleteDirectory(target) - } - invocation.callRealMethod() - }.when(spyRepo).cloneRepo() - repos.put(repoTarget, spyRepo) - return spyRepo - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestScmManagerApiClient.groovy b/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestScmManagerApiClient.groovy deleted file mode 100644 index eac3a6ae1..000000000 --- a/src/test/groovy/com/cloudogu/gitops/testhelper/git/TestScmManagerApiClient.groovy +++ /dev/null @@ -1,77 +0,0 @@ -package com.cloudogu.gitops.testhelper.git - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.Credentials -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.Permission -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.RepositoryApi -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient -import okhttp3.internal.http.RealResponseBody -import okio.BufferedSource -import org.mockito.ArgumentMatchers -import retrofit2.Call -import retrofit2.Response - -import static org.mockito.ArgumentMatchers.anyBoolean -import static org.mockito.ArgumentMatchers.anyString -import static org.mockito.Mockito.mock -import static org.mockito.Mockito.when - -class TestScmManagerApiClient extends ScmManagerApiClient { - - RepositoryApi repositoryApi = mock(RepositoryApi) - Set createdRepos = new HashSet<>() - Set createdPermissions = new HashSet<>() - - TestScmManagerApiClient(Config config) { - super(config.scm.scmManager.url, new Credentials(config.scm.scmManager.username, config.scm.scmManager.password), null) - } - - @Override - RepositoryApi repositoryApi() { - return repositoryApi - } - - /** - * Make all repo API calls return created on the first call and exists on subsequent calls for each repo.*/ - void mockRepoApiBehaviour() { - def responseCreated = mockSuccessfulResponse(201) - def responseExists = mockErrorResponse(409) - - when(repositoryApi.create(ArgumentMatchers.any(Repository), anyBoolean())) - .thenAnswer { invocation -> - Repository repo = invocation.getArgument(0) - if (createdRepos.contains(repo.fullRepoName)) { - return responseExists - } else { - createdRepos.add(repo.fullRepoName) - return responseCreated - } - } - when(repositoryApi.createPermission(anyString(), anyString(), ArgumentMatchers.any(Permission))) - .thenAnswer { invocation -> - String namespace = invocation.getArgument(0) - String name = invocation.getArgument(1) - if (createdPermissions.contains("${namespace}/${name}".toString())) { - return responseExists - } else { - createdPermissions.add("${namespace}/${name}".toString()) - return responseCreated - } - } - } - - static Call mockSuccessfulResponse(int expectedReturnCode) { - def expectedCall = mock(Call) - when(expectedCall.execute()).thenReturn(Response.success(expectedReturnCode, null)) - expectedCall - } - - static Call mockErrorResponse(int expectedReturnCode) { - def expectedCall = mock(Call) - // Response is a final class that cannot be mocked 😠 - Response errorResponse = Response.error(expectedReturnCode, new RealResponseBody('dontcare', 0, mock(BufferedSource))) - when(expectedCall.execute()).thenReturn(errorResponse) - expectedCall - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/CertManagerTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/CertManagerTest.groovy deleted file mode 100644 index 35c5df47c..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/CertManagerTest.groovy +++ /dev/null @@ -1,164 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.K8sClientForTest -import groovy.yaml.YamlSlurper -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.ExtendWith -import org.mockito.ArgumentCaptor -import org.mockito.Mock -import org.mockito.junit.jupiter.MockitoExtension - -import java.nio.file.Files -import java.nio.file.Path - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.verify -import static org.mockito.Mockito.when - -@ExtendWith(MockitoExtension.class) -class CertManagerTest { - String chartVersion = "1.19.4" - Config config = Config.fromMap([features: [certManager: [active: true, - helm : [chart : 'cert-manager', - repoURL: 'https://charts.jetstack.io', - version: chartVersion,],],],]) - - Path temporaryYamlFile - FileSystemUtils fileSystemUtils = new FileSystemUtils() - - @Mock - DeploymentStrategy deploymentStrategy - @Mock - AirGappedUtils airGappedUtils - @Mock - GitHandler gitHandler - @Mock - GitProvider gitProvider - - @Test - void 'Helm release is installed'() { - createCertManager().install() - - verify(deploymentStrategy).deployFeature('https://charts.jetstack.io', 'cert-manager', - 'cert-manager', chartVersion, 'cert-manager', - 'cert-manager', temporaryYamlFile, RepoType.HELM) - } - - @Test - void 'Sets pod resource limits and requests'() { - config.application.podResources = true - - createCertManager().install() - - assertThat(parseActualYaml()['resources'] as Map).containsKeys('limits', 'requests') - assertThat(parseActualYaml()['cainjector']['resources'] as Map).containsKeys('limits', 'requests') - assertThat(parseActualYaml()['webhook']['resources'] as Map).containsKeys('limits', 'requests') - } - - @Test - void "is disabled via active flag"() { - config.features.certManager.active = false - createCertManager().install() - assertThat(temporaryYamlFile).isNull() - } - - @Test - void 'helm release is installed in air-gapped mode'() { - when(gitHandler.getResourcesScm()).thenReturn(gitProvider) - when(gitProvider.repoUrl(any())).thenReturn("http://scmm.scm-manager.svc.cluster.local/scm/repo/a/b") - - config.application.mirrorRepos = true - when(airGappedUtils.mirrorHelmRepoToGit(any(Config.HelmConfig))).thenReturn('a/b') - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path SourceChart = rootChartsFolder.resolve('cert-manager') - Files.createDirectories(SourceChart) - - Map ChartYaml = [version: chartVersion] - fileSystemUtils.writeYaml(ChartYaml, SourceChart.resolve('Chart.yaml').toFile()) - - createCertManager().install() - - def helmConfig = ArgumentCaptor.forClass(Config.HelmConfig) - verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()) - assertThat(helmConfig.value.chart).isEqualTo('cert-manager') - // check existing value, but its not used in deploy. - assertThat(helmConfig.value.repoURL).isEqualTo('https://charts.jetstack.io') - assertThat(helmConfig.value.version).isEqualTo(chartVersion) - // important check: scmmRepoUrl is overridden with our values. - verify(deploymentStrategy).deployFeature('http://scmm.scm-manager.svc.cluster.local/scm/repo/a/b', - 'cert-manager', '.', chartVersion, 'cert-manager', - 'cert-manager', temporaryYamlFile, RepoType.GIT) - } - - @Test - void 'check images are overriddes'() { - when(gitHandler.getResourcesScm()).thenReturn(gitProvider) - when(gitProvider.repoUrl(any())).thenReturn("http://test") - - // Prep - config.application.mirrorRepos = true - // test values - config.features.certManager.helm.image = "this.is.my.registry:30000/this.is.my.repository/myImage:1" - config.features.certManager.helm.webhookImage = "this.is.my.registry:30000/this.is.my.repository/myWebhook:2" - config.features.certManager.helm.cainjectorImage = "this.is.my.registry:30000/this.is.my.repository/myCainjectorImage:3" - config.features.certManager.helm.acmeSolverImage = "this.is.my.registry:30000/this.is.my.repository/myAcmeSolverImage:4" - config.features.certManager.helm.startupAPICheckImage = "this.is.my.registry:30000/this.is.my.repository/myStartupAPICheckImage:5" - when(airGappedUtils.mirrorHelmRepoToGit(any(Config.HelmConfig))).thenReturn('a/b') - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path SourceChart = rootChartsFolder.resolve('cert-manager') - Files.createDirectories(SourceChart) - - Map ChartYaml = [version: chartVersion] - fileSystemUtils.writeYaml(ChartYaml, SourceChart.resolve('Chart.yaml').toFile()) - createCertManager().install() - - def templateFile = parseActualYaml() - - // Cert-Manager - assertThat(parseActualYaml()['image']['repository'] as String).isEqualTo('this.is.my.registry:30000/this.is.my.repository/myImage') - assertThat(parseActualYaml()['image']['tag'] as String).isEqualTo('1') - // myWebhook - assertThat(parseActualYaml()['webhook']['image']['repository'] as String).isEqualTo('this.is.my.registry:30000/this.is.my.repository/myWebhook') - assertThat(parseActualYaml()['webhook']['image']['tag'] as String).isEqualTo('2') - // cainjectorImage - assertThat(parseActualYaml()['cainjector']['image']['repository'] as String).isEqualTo('this.is.my.registry:30000/this.is.my.repository/myCainjectorImage') - assertThat(parseActualYaml()['cainjector']['image']['tag'] as String).isEqualTo('3') - // myWebhook - assertThat(parseActualYaml()['acmesolver']['image']['repository'] as String).isEqualTo('this.is.my.registry:30000/this.is.my.repository/myAcmeSolverImage') - assertThat(parseActualYaml()['acmesolver']['image']['tag'] as String).isEqualTo('4') - // myWebhook - assertThat(parseActualYaml()['startupapicheck']['image']['repository'] as String).isEqualTo('this.is.my.registry:30000/this.is.my.repository/myStartupAPICheckImage') - assertThat(parseActualYaml()['startupapicheck']['image']['tag'] as String).isEqualTo('5') - - } - - private CertManager createCertManager() { - // We use the real FileSystemUtils and not a mock to make sure file editing works as expected - new CertManager(config, new FileSystemUtils() { - @Override - Path writeTempFile(Map mapValues) { - def ret = super.writeTempFile(mapValues) - temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")) - return ret - } - }, deploymentStrategy, new K8sClientForTest(), airGappedUtils, gitHandler) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFile) as Map - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/ExternalSecretsOperatorTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/ExternalSecretsOperatorTest.groovy deleted file mode 100644 index 0793808a3..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/ExternalSecretsOperatorTest.groovy +++ /dev/null @@ -1,188 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.CommandExecutorForTest -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.yaml.YamlSlurper -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.ExtendWith -import org.mockito.ArgumentCaptor -import org.mockito.Mock -import org.mockito.junit.jupiter.MockitoExtension - -import java.nio.file.Files -import java.nio.file.Path - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.verify -import static org.mockito.Mockito.when - -@ExtendWith(MockitoExtension.class) -@EnableKubernetesMockClient(crud = true) -class ExternalSecretsOperatorTest { - - Config config = new Config(application: new Config.ApplicationSchema(namePrefix: "foo-"), - registry: new Config.RegistrySchema(), - features: new Config.FeaturesSchema(secrets: new Config.SecretsSchema(active: true))) - - CommandExecutorForTest commandExecutor = new CommandExecutorForTest() - FileSystemUtils fileSystemUtils = new FileSystemUtils() - Path temporaryYamlFile - - @Mock - DeploymentStrategy deploymentStrategy - @Mock - AirGappedUtils airGappedUtils - @Mock - GitHandler gitHandler - @Mock - GitProvider gitProvider - - K8sClient k8sClient - KubernetesClient client - - @BeforeEach - void init() { - k8sClient = new K8sClient() - k8sClient.client = client - } - - @Test - void "is disabled via active flag"() { - config.features.secrets.active = false - createExternalSecretsOperator().install() - assertThat(commandExecutor.actualCommands).isEmpty() - } - - @Test - void 'helm release is installed'() { - createExternalSecretsOperator().install() - - verify(deploymentStrategy).deployFeature('https://charts.external-secrets.io', - 'external-secrets-operator', - 'external-secrets', - '0.9.16', - 'foo-secrets', - 'external-secrets', - temporaryYamlFile, - RepoType.HELM) - - assertThat(parseActualYaml()).doesNotContainKeys('resources') - assertThat(parseActualYaml()).doesNotContainKey('imagePullSecrets') - assertThat(parseActualYaml()).doesNotContainKey('certController') - assertThat(parseActualYaml()).doesNotContainKey('webhook') - - assertThat(parseActualYaml()['installCRDs']).isNull() - } - - @Test - void 'Skips CRDs'() { - config.application.skipCrds = true - - createExternalSecretsOperator().install() - - assertThat(parseActualYaml()['installCRDs']).isEqualTo(false) - } - - @Test - void 'helm release is installed with custom images'() { - config.features.secrets.externalSecrets.helm = new Config.SecretsSchema.ESOSchema.ESOHelmSchema([image : 'localhost:5000/external-secrets/external-secrets:v0.6.1', - certControllerImage: 'localhost:5000/external-secrets/external-secrets-certcontroller:v0.6.1', - webhookImage : 'localhost:5000/external-secrets/external-secrets-webhook:v0.6.1']) - createExternalSecretsOperator().install() - - def valuesYaml = parseActualYaml() - assertThat(valuesYaml['image']['repository']).isEqualTo('localhost:5000/external-secrets/external-secrets') - assertThat(valuesYaml['image']['tag']).isEqualTo('v0.6.1') - - assertThat(valuesYaml['certController']['image']['repository']).isEqualTo('localhost:5000/external-secrets/external-secrets-certcontroller') - assertThat(valuesYaml['certController']['image']['tag']).isEqualTo('v0.6.1') - - assertThat(valuesYaml['webhook']['image']['repository']).isEqualTo('localhost:5000/external-secrets/external-secrets-webhook') - assertThat(valuesYaml['webhook']['image']['tag']).isEqualTo('v0.6.1') - } - - @Test - void 'Sets pod resource limits and requests'() { - config.application.podResources = true - - createExternalSecretsOperator().install() - - assertThat(parseActualYaml()['resources'] as Map).containsKeys('limits', 'requests') - assertThat(parseActualYaml()['webhook']['resources'] as Map).containsKeys('limits', 'requests') - assertThat(parseActualYaml()['certController']['resources'] as Map).containsKeys('limits', 'requests') - } - - @Test - void 'helm release is installed in air-gapped mode'() { - when(gitHandler.getResourcesScm()).thenReturn(gitProvider) - when(gitProvider.repoUrl(any())).thenReturn("http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b") - when(airGappedUtils.mirrorHelmRepoToGit(any(Config.HelmConfig))).thenReturn('a/b') - - config.application.mirrorRepos = true - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path SourceChart = rootChartsFolder.resolve('external-secrets') - Files.createDirectories(SourceChart) - - Map ChartYaml = [version: '1.2.3'] - fileSystemUtils.writeYaml(ChartYaml, SourceChart.resolve('Chart.yaml').toFile()) - - createExternalSecretsOperator().install() - - def helmConfig = ArgumentCaptor.forClass(Config.HelmConfig) - verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()) - assertThat(helmConfig.value.chart).isEqualTo('external-secrets') - assertThat(helmConfig.value.repoURL).isEqualTo('https://charts.external-secrets.io') - assertThat(helmConfig.value.version).isEqualTo('0.9.16') - verify(deploymentStrategy).deployFeature('http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b', - 'external-secrets-operator', '.', '1.2.3', 'foo-secrets', - 'external-secrets', temporaryYamlFile, RepoType.GIT) - } - - @Test - void 'deploys image pull secrets for proxy registry'() { - config.registry.createImagePullSecrets = true - config.registry.proxyUrl = 'proxy-url' - config.registry.proxyUsername = 'proxy-user' - config.registry.proxyPassword = 'proxy-pw' - config.registry.proxyPassword = 'proxy-pw' - config.features.secrets.externalSecrets.helm = new Config.SecretsSchema.ESOSchema.ESOHelmSchema([certControllerImage: 'some:thing', - webhookImage : 'some:thing']) - - createExternalSecretsOperator().install() - assertThat(parseActualYaml()['imagePullSecrets']).isEqualTo([[name: 'proxy-registry']]) - assertThat(parseActualYaml()['certController']['imagePullSecrets']).isEqualTo([[name: 'proxy-registry']]) - assertThat(parseActualYaml()['webhook']['imagePullSecrets']).isEqualTo([[name: 'proxy-registry']]) - } - - private ExternalSecretsOperator createExternalSecretsOperator() { - new ExternalSecretsOperator(config, - new FileSystemUtils() { - @Override - Path writeTempFile(Map mergeMap) { - def ret = super.writeTempFile(mergeMap) - temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")) - // Path after template invocation - return ret - } - }, deploymentStrategy, k8sClient, airGappedUtils, gitHandler) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFile) as Map - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/IngressTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/IngressTest.groovy deleted file mode 100644 index a5e7331d4..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/IngressTest.groovy +++ /dev/null @@ -1,207 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.yaml.YamlSlurper -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.junit.jupiter.api.extension.ExtendWith -import org.mockito.ArgumentCaptor -import org.mockito.Mock -import org.mockito.junit.jupiter.MockitoExtension - -import java.nio.file.Files -import java.nio.file.Path - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.verify -import static org.mockito.Mockito.when - -@ExtendWith(MockitoExtension.class) -@EnableKubernetesMockClient(crud = true) -class IngressTest { - - // setting default config values with ingress active - Config config = new Config(application: new Config.ApplicationSchema(namePrefix: 'foo-'), - features: new Config.FeaturesSchema(ingress: new Config.IngressSchema(active: true))) - Path temporaryYamlFile - FileSystemUtils fileSystemUtils = new FileSystemUtils() - - @Mock - DeploymentStrategy deploymentStrategy - @Mock - AirGappedUtils airGappedUtils - @Mock - GitHandler gitHandler - @Mock - GitProvider gitProvider - - K8sClient k8sClient - KubernetesClient client - - @BeforeEach - void init() { - k8sClient = new K8sClient() - k8sClient.client = client - } - - - @Test - void 'Helm release is installed'() { - createIngress().install() - - /* Assert one default value */ - def actual = parseActualYaml() - assertThat(actual['deployment']['replicaCount']).isEqualTo(2) - - verify(deploymentStrategy).deployFeature(config.features.ingress.helm.repoURL, 'traefik', - config.features.ingress.helm.chart, config.features.ingress.helm.version, 'foo-' + config.features.ingress.ingressNamespace, - 'traefik', temporaryYamlFile, RepoType.HELM) - assertThat(parseActualYaml()['deployment']['metrics']).isNull() - assertThat(parseActualYaml()['deployment']['networkPolicy']).isNull() - assertThat(parseActualYaml()).doesNotContainKey('imagePullSecrets') - - } - - @Test - void 'Sets pod resource limits and requests'() { - config.application.podResources = true - - createIngress().install() - - assertThat(parseActualYaml()['deployment']['resources'] as Map).containsKeys('limits', 'requests') - } - - @Test - void 'When Ingress is not enabled, ingress-helm-values yaml has no content'() { - config.features.ingress.active = false - - createIngress().install() - - assertThat(temporaryYamlFile).isNull() - } - - @Test - void 'additional helm values merged with default values'() { - config.features.ingress.helm.values = [controller: [replicaCount: 42, - span : '7,5',]] - - createIngress().install() - def actual = parseActualYaml() - - assertThat(actual['controller']['replicaCount']).isEqualTo(42) - assertThat(actual['controller']['span']).isEqualTo('7,5') - } - - @Test - void 'helm release is installed in air-gapped mode'() { - when(gitHandler.getResourcesScm()).thenReturn(gitProvider) - when(gitProvider.repoUrl(any())).thenReturn("http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b") - when(airGappedUtils.mirrorHelmRepoToGit(any(Config.HelmConfig))).thenReturn('a/b') - - config.application.mirrorRepos = true - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path SourceChart = rootChartsFolder.resolve('traefik') - Files.createDirectories(SourceChart) - - Map ChartYaml = [version: '1.2.3'] - fileSystemUtils.writeYaml(ChartYaml, SourceChart.resolve('Chart.yaml').toFile()) - - createIngress().install() - - def helmConfig = ArgumentCaptor.forClass(Config.HelmConfig) - verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()) - assertThat(helmConfig.value.chart).isEqualTo('traefik') - - assertThat(helmConfig.value.repoURL).isEqualTo('https://traefik.github.io/charts') - assertThat(helmConfig.value.version).isEqualTo('39.0.0') - verify(deploymentStrategy).deployFeature('http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b', - 'traefik', '.', '1.2.3', 'foo-' + config.features.ingress.ingressNamespace, - 'traefik', temporaryYamlFile, RepoType.GIT) - } - - @Test - void 'When Monitoring is enabled, metrics are enabled'() { - config.features.monitoring.active = true - config.application.namePrefix = "heliosphere" - - createIngress().install() - - def actual = parseActualYaml() - - assertThat(actual['metrics']['enabled']).isEqualTo(true) - assertThat(actual['metrics']['prometheus']['serviceMonitor']['enabled']).isEqualTo(true) - assertThat(actual['metrics']['prometheus']['serviceMonitor']['namespace']).isEqualTo("heliospheremonitoring") - } - - @Test - void 'Activates network policies'() { - config.application.netpols = true - - createIngress().install() - - def actual = parseActualYaml() - - assertThat(actual['deployment']['networkPolicy']['enabled']).isEqualTo(true) - } - - @Test - void 'deploys image pull secrets for proxy registry'() { - config.registry.createImagePullSecrets = true - config.registry.proxyUrl = 'proxy-url' - config.registry.proxyUsername = 'proxy-user' - config.registry.proxyPassword = 'proxy-pw' - - createIngress().install() - assertThat(parseActualYaml()['deployment']['imagePullSecrets']).isEqualTo([[name: 'proxy-registry']]) - } - - @Test - void 'Allows overriding the image'() { - config.features.ingress.helm.image = 'localhost/abc:v42' - - createIngress().install() - - def yaml = parseActualYaml() - assertThat(yaml['image']['repository']).isEqualTo('localhost/abc') - assertThat(yaml['image']['tag']).isEqualTo('v42') - assertThat(yaml['image']['digest']).isNull() - } - - @Test - void 'get namespace from feature'() { - assertThat(createIngress().getActiveNamespaceFromFeature()).isEqualTo('foo-' + config.features.ingress.ingressNamespace) - config.features.ingress.active = false - assertThat(createIngress().getActiveNamespaceFromFeature()).isEqualTo(null) - } - - private Ingress createIngress() { - // We use the real FileSystemUtils and not a mock to make sure file editing works as expected - new Ingress(config, new FileSystemUtils() { - @Override - Path writeTempFile(Map mergeMap) { - def ret = super.writeTempFile(mergeMap) - temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")) - // Path after template invocation - return ret - } - }, deploymentStrategy, k8sClient, airGappedUtils, gitHandler) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFile) as Map - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/MonitoringTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/MonitoringTest.groovy deleted file mode 100644 index 6ca36d98d..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/MonitoringTest.groovy +++ /dev/null @@ -1,649 +0,0 @@ -package com.cloudogu.gitops.tools - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.* - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.FileSystemUtils - -import java.nio.file.Files -import java.nio.file.Path -import groovy.yaml.YamlSlurper - -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.mockito.ArgumentCaptor - -@EnableKubernetesMockClient(crud = true) -class MonitoringTest { - Config config = Config.fromMap(registry: [internal : true, - createImagePullSecrets: false], - scm: [scmManager: [internal: true]], - jenkins: [internal : true, - active : true, - metricsUsername: 'metrics', - metricsPassword: 'metrics',], - application: [username : 'abc', - password : '123', - openshift : false, - namePrefix : 'foo-', - mirrorRepos : false, - podResources : false, - skipCrds : false, - namespaceIsolation: false, - gitName : 'Cloudogu', - gitEmail : 'hello@cloudogu.com', - netpols : false, - namespaces : [dedicatedNamespaces: ["test1-default", - "test1-argocd", - "test1-monitoring", - "test1-secrets"] as LinkedHashSet, - tenantNamespaces : ["test1-example-apps-staging", - "test1-example-apps-production"] as LinkedHashSet]], - features: [argocd : [active: true], - monitoring: [active : true, - grafanaUrl : '', - grafanaEmailFrom: 'grafana@example.org', - grafanaEmailTo : 'infra@example.org', - helm : [chart : 'kube-prometheus-stack', - repoURL: 'https://prom', - version: '19.2.2']], - secrets : [active: true], - ingress : [active: true]]) - - K8sClient k8sClient - DeploymentStrategy deploymentStrategy = mock(DeploymentStrategy) - AirGappedUtils airGappedUtils = mock(AirGappedUtils) - Path temporaryYamlFilePrometheus = null - FileSystemUtils fileSystemUtils = new FileSystemUtils() - File clusterResourcesRepoDir - - GitHandler gitHandler = mock(GitHandler.class) - ScmManagerMock scmManagerMock - - KubernetesClient client - //Client to set mock data, gets injected by annotation - KubernetesMockServer server - //Use server for non CRUD - - @BeforeEach - void setup() { - scmManagerMock = new ScmManagerMock() - k8sClient = mock(K8sClient) - k8sClient.client = client - } - - @Test - void "is disabled via active flag"() { - config.features.monitoring.active = false - createStack(scmManagerMock).install() - assertThat(temporaryYamlFilePrometheus).isNull() - verifyNoMoreInteractions(deploymentStrategy) - } - - @Test - void 'When mailServer disabled: Does not include mail configurations into cluster resources'() { - config.features.mail.active = null // user should not do this in real. - createStack(scmManagerMock).install() - - def yaml = parseActualYaml() - assertThat(yaml['grafana']['notifiers']).isNull() - } - - @Test - void 'When mailServer enabled: Includes mail configurations into cluster resources'() { - config.features.mail.active = true - createStack(scmManagerMock).install() - assertThat(parseActualYaml()['grafana']['notifiers']).isNotNull() - } - - @Test - void "When Email Addresses is set"() { - config.features.mail.active = true - config.features.monitoring.grafanaEmailFrom = 'grafana@example.com' - config.features.monitoring.grafanaEmailTo = 'infra@example.com' - createStack(scmManagerMock).install() - - def notifiersYaml = parseActualYaml()['grafana']['notifiers']['notifiers.yaml']['notifiers']['settings'] as List - assertThat(notifiersYaml[0]['addresses']).isEqualTo('infra@example.com') - assertThat(parseActualYaml()['grafana']['env']['GF_SMTP_FROM_ADDRESS']).isEqualTo('grafana@example.com') - } - - @Test - void "When Email Addresses is NOT set"() { - config.features.mail.active = true - createStack(scmManagerMock).install() - - def notifiersYaml = parseActualYaml()['grafana']['notifiers']['notifiers.yaml']['notifiers']['settings'] as List - assertThat(notifiersYaml[0]['addresses']).isEqualTo('infra@example.org') - assertThat(parseActualYaml()['grafana']['env']['GF_SMTP_FROM_ADDRESS']).isEqualTo('grafana@example.org') - } - - @Test - void 'When external Mailserver is set'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpPort = 1010110 - config.features.monitoring.grafanaEmailTo = 'grafana@example.com' - // needed to check that yaml is inserted correctly - - createStack(scmManagerMock).install() - def contactPointsYaml = parseActualYaml() - - assertThat(contactPointsYaml['grafana']['alerting']['contactpoints.yaml']).isEqualTo(new YamlSlurper().parseText(""" -apiVersion: 1 -contactPoints: -- orgId: 1 - name: email - is_default: true - receivers: - - uid: email1 - type: email - settings: - addresses: ${config.features.monitoring.grafanaEmailTo} -""")) - assertThat(contactPointsYaml['grafana']['alerting']['notification-policies.yaml']).isEqualTo(new YamlSlurper().parseText(''' -apiVersion: 1 -policies: -- orgId: 1 - is_default: true - receiver: email - routes: - - receiver: email - group_by: ["grafana_folder", "alertname"] -''')) - - assertThat(contactPointsYaml['grafana']['env']['GF_SMTP_HOST']).isEqualTo('smtp.example.com:1010110') - } - - @Test - void 'When external Mailserver is set with user'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpUser = 'mailserver@example.com' - - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['grafana']['smtp']['existingSecret']).isEqualTo('grafana-email-secret') - } - - @Test - void 'When external Mailserver is set with password'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpPassword = '1101ABCabc&/+*~' - - createStack(scmManagerMock).install() - assertThat(parseActualYaml()['grafana']['smtp']['existingSecret']).isEqualTo('grafana-email-secret') - } - - @Test - void 'When external Mailserver is set without user and password'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['grafana']['valuesFrom']).isNull() - assertThat(parseActualYaml()['grafana']['smtp']).isNull() - } - - @Test - void 'Check if kubernetes secret will be created when external emailservers credential is set'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpUser = 'grafana@example.com' - config.features.mail.smtpPassword = '1101ABCabc&/+*~' - - createStack(scmManagerMock).install() - - } - - @Test - void 'When external Mailserver is set without port'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - - createStack(scmManagerMock).install() - def contactPointsYaml = parseActualYaml() - - assertThat(contactPointsYaml['grafana']['env']['GF_SMTP_HOST']).isEqualTo('smtp.example.com') - } - - @Test - void 'When external Mailserver is NOT set'() { - config.features.mail.active = null // user should not do this in real. - createStack(scmManagerMock).install() - def contactPointsYaml = parseActualYaml() - - assertThat(contactPointsYaml['grafana']['alerting']).isNull() - } - - @Test - void "configures admin user if requested"() { - config.application.username = "my-user" - config.application.password = "hunter2" - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['grafana']['adminUser']).isEqualTo('my-user') - assertThat(parseActualYaml()['grafana']['adminPassword']).isEqualTo('hunter2') - } - - @Test - void 'uses ingress if enabled'() { - config.features.monitoring.grafanaUrl = 'http://grafana.local' - - createStack(scmManagerMock).install() - - def serviceYaml = parseActualYaml()['grafana']['ingress'] - assertThat(serviceYaml['enabled']).isEqualTo(true) - assertThat((serviceYaml['hosts'] as List)[0]).isEqualTo('grafana.local') - } - - @Test - void 'does not use ingress by default'() { - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['grafana'] as Map).doesNotContainKey('ingress') - } - - @Test - void 'cleanupUnusedDashboards removes all dashboards for disabled features'() { - config.features.monitoring.active = true - config.features.ingress.active = false - config.jenkins.active = false - config.scm.scmManager.url = null // triggers scmm dashboard cleanup - - createStack(scmManagerMock).install() - - File dashboardDir = new File(clusterResourcesRepoDir, "apps/prometheusstack/misc/dashboard") - - assertThat(new File(dashboardDir, "traefik-dashboard.yaml")).doesNotExist() - assertThat(new File(dashboardDir, "traefik-dashboard-requests-handling.yaml")).doesNotExist() - assertThat(new File(dashboardDir, "jenkins-dashboard.yaml")).doesNotExist() - assertThat(new File(dashboardDir, "scmm-dashboard.yaml")).doesNotExist() - } - - @Test - void 'Applies Prometheus ServiceMonitor CRD from file before installing (air-gapped mode)'() { - // Arrange - config.features.monitoring.active = true - config.application.mirrorRepos = true - config.application.skipCrds = false - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path crdFile = rootChartsFolder.resolve("${config.features.monitoring.helm.chart}/charts/crds/crds/crd-servicemonitors.yaml") - Files.createDirectories(crdFile.parent) - Files.writeString(crdFile, "dummy") // content can be anything for this test - - Path chartYaml = rootChartsFolder.resolve("${config.features.monitoring.helm.chart}/Chart.yaml") - Files.createDirectories(chartYaml.parent) - Files.writeString(chartYaml, "apiVersion: v2\nname: kube-prometheus-stack\nversion: 42.0.3\n") - - createStack(scmManagerMock).install() - } - - @Test - void 'Applies Prometheus ServiceMonitor CRD from GitHub before installing'() { - config.features.monitoring.active = true - config.application.mirrorRepos = false // optional, but makes intent explicit - config.application.skipCrds = false // optional, but makes intent explicit - - createStack(scmManagerMock).install() - - } - - @Test - void 'does not apply ServiceMonitor CRD when monitoring is disabled'() { - config.features.monitoring.active = false // important - config.application.skipCrds = false // so it would apply if enabled - config.application.mirrorRepos = false // avoid local chart access - - createStack(scmManagerMock).install() - - // no CRD apply should happen at all - } - - @Test - void 'uses remote scmm url if requested'() { - createStack(scmManagerMock).install() - - def additionalScrapeConfigs = parseActualYaml()['prometheus']['prometheusSpec']['additionalScrapeConfigs'] as List - assertThat(((additionalScrapeConfigs[0]['static_configs'] as List)[0]['targets'] as List)[0]).isEqualTo('localhost:8080') - assertThat(additionalScrapeConfigs[0]['metrics_path']).isEqualTo('/scm/api/v2/metrics/prometheus') - assertThat(additionalScrapeConfigs[0]['scheme']).isEqualTo('http') - - // scrape config for jenkins is unchanged - assertThat(((additionalScrapeConfigs[1]['static_configs'] as List)[0]['targets'] as List)[0]).isEqualTo('jenkins.foo-jenkins.svc.cluster.local') - assertThat(additionalScrapeConfigs[1]['scheme']).isEqualTo('http') - assertThat(additionalScrapeConfigs[1]['metrics_path']).isEqualTo('/prometheus') - } - - @Test - void 'uses remote jenkins url if requested'() { - config.jenkins["internal"] = false - config.jenkins["url"] = 'https://localhost:9090/jenkins' - createStack(scmManagerMock).install() - def additionalScrapeConfigs = parseActualYaml()['prometheus']['prometheusSpec']['additionalScrapeConfigs'] as List - - // scrape config for scmm is unchanged - assertThat(((additionalScrapeConfigs[0]['static_configs'] as List)[0]['targets'] as List)[0]).isEqualTo('localhost:8080') - assertThat(additionalScrapeConfigs[0]['scheme']).isEqualTo('http') - assertThat(additionalScrapeConfigs[0]['metrics_path']).isEqualTo('/scm/api/v2/metrics/prometheus') - - assertThat(((additionalScrapeConfigs[1]['static_configs'] as List)[0]['targets'] as List)[0]).isEqualTo('localhost:9090') - assertThat(additionalScrapeConfigs[1]['metrics_path']).isEqualTo('/jenkins/prometheus') - assertThat(additionalScrapeConfigs[1]['scheme']).isEqualTo('https') - } - - @Test - void 'configures custom metrics user for jenkins'() { - config.jenkins["metricsUsername"] = 'external-metrics-username' - config.jenkins["metricsPassword"] = 'hunter2' - createStack(scmManagerMock).install() - - def additionalScrapeConfigs = parseActualYaml()['prometheus']['prometheusSpec']['additionalScrapeConfigs'] as List - assertThat(additionalScrapeConfigs[1]['basic_auth']['username']).isEqualTo('external-metrics-username') - } - - @Test - void "configures custom image for grafana"() { - config.features.monitoring.helm.grafanaImage = "localhost:5000/grafana/grafana:the-tag" - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['grafana']['image']['registry']).isEqualTo('localhost:5000') - assertThat(parseActualYaml()['grafana']['image']['repository']).isEqualTo('grafana/grafana') - assertThat(parseActualYaml()['grafana']['image']['tag']).isEqualTo('the-tag') - } - - @Test - void "configures custom image for grafana-sidecar"() { - config.features.monitoring.helm.grafanaSidecarImage = "localhost:5000/grafana/sidecar:the-tag" - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['grafana']['sidecar']['image']['registry']).isEqualTo('localhost:5000') - assertThat(parseActualYaml()['grafana']['sidecar']['image']['repository']).isEqualTo('grafana/sidecar') - assertThat(parseActualYaml()['grafana']['sidecar']['image']['tag']).isEqualTo('the-tag') - } - - @Test - void "configures custom image for prometheus and operator"() { - config.features.monitoring.helm.prometheusImage = "localhost:5000/prometheus/prometheus:v1" - config.features.monitoring.helm.prometheusOperatorImage = "localhost:5000/prometheus-operator/prometheus-operator:v2" - config.features.monitoring.helm.prometheusConfigReloaderImage = "localhost:5000/prometheus-operator/prometheus-config-reloader:v3" - - createStack(scmManagerMock).install() - - def actualYaml = parseActualYaml() - assertThat(actualYaml['prometheus']['prometheusSpec']['image']['registry']).isEqualTo('localhost:5000') - assertThat(actualYaml['prometheus']['prometheusSpec']['image']['repository']).isEqualTo('prometheus/prometheus') - assertThat(actualYaml['prometheus']['prometheusSpec']['image']['tag']).isEqualTo('v1') - assertThat(actualYaml['prometheusOperator']['image']['registry']).isEqualTo('localhost:5000') - assertThat(actualYaml['prometheusOperator']['image']['repository']).isEqualTo('prometheus-operator/prometheus-operator') - assertThat(actualYaml['prometheusOperator']['image']['tag']).isEqualTo('v2') - assertThat(actualYaml['prometheusOperator']['prometheusConfigReloader']['image']['registry']).isEqualTo('localhost:5000') - assertThat(actualYaml['prometheusOperator']['prometheusConfigReloader']['image']['repository']).isEqualTo('prometheus-operator/prometheus-config-reloader') - assertThat(actualYaml['prometheusOperator']['prometheusConfigReloader']['image']['tag']).isEqualTo('v3') - } - - @Test - void 'deploys image pull secrets for proxy registry'() { - config.registry.createImagePullSecrets = true - config.registry.proxyUrl = 'proxy-url' - config.registry.proxyUsername = 'proxy-user' - config.registry.proxyPassword = 'proxy-pw' - - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['global']['imagePullSecrets']).isEqualTo([[name: 'proxy-registry']]) - } - - @Test - void 'helm release is installed'() { - createStack(scmManagerMock).install() - - verify(deploymentStrategy).deployFeature('https://prom', 'monitoring', - 'kube-prometheus-stack', '19.2.2', 'foo-monitoring', - 'kube-prometheus-stack', temporaryYamlFilePrometheus, RepoType.HELM) - /* This corresponds to - 'helm repo add prometheusstack https://prom' - 'helm upgrade -i kube-prometheus-stack prometheusstack/kube-prometheus-stack --version 19.2.2' + - " --values ${temporaryYamlFile} --namespace foo-monitoring --create-namespace") */ - - def yaml = parseActualYaml() - assertThat(yaml['grafana']['adminUser']).isEqualTo('abc') - assertThat(yaml['grafana']['adminPassword']).isEqualTo(123) - - assertThat(yaml['prometheusOperator'] as Map).doesNotContainKey('resources') - assertThat(yaml['grafana'] as Map).doesNotContainKey('resources') - assertThat(yaml['grafana']['sidecar'] as Map).doesNotContainKey('resources') - assertThat(yaml['prometheus']['prometheusSpec'] as Map).doesNotContainKey('resources') - - assertThat(yaml['prometheusOperator']['securityContext']).isNull() - assertThat(yaml['grafana']['securityContext']).isNull() - assertThat(yaml['prometheus']['prometheusSpec']['securityContext']).isNull() - - assertThat(yaml['kubeApiServer']).isNull() - - assertThat(yaml['prometheusOperator']['admissionWebhooks']['enabled']).isEqualTo(false) - assertThat(yaml['prometheusOperator']['tls']['enabled']).isEqualTo(false) - assertThat(yaml['prometheusOperator']['kubeletService']).isNull() - assertThat(yaml['prometheusOperator']['namespaces']).isNull() - assertThat(yaml).doesNotContainKey('global') - - assertThat(yaml['grafana']['rbac']).isNull() - assertThat(yaml['grafana']['sidecar']['dashboards']['searchNamespace']).isEqualTo('ALL') - - assertThat(yaml['crds']).isNull() - assertThat(new File("$clusterResourcesRepoDir/misc/monitoring/rbac")).doesNotExist() - } - - @Test - void 'Skips CRDs'() { - config.application.skipCrds = true - - createStack(scmManagerMock).install() - - assertThat(parseActualYaml()['crds']['enabled']).isEqualTo(false) - } - - @Test - void 'Sets pod resource limits and requests'() { - config.application.podResources = true - - createStack(scmManagerMock).install() - - def yaml = parseActualYaml() - assertThat(yaml['prometheusOperator']['resources'] as Map).containsKeys('limits', 'requests') - assertThat(yaml['prometheusOperator']['prometheusConfigReloader']['resources'] as Map).containsKeys('limits', 'requests') - assertThat(yaml['grafana']['resources'] as Map) containsKeys('limits', 'requests') - assertThat(yaml['grafana']['sidecar']['resources'] as Map) containsKeys('limits', 'requests') - assertThat(yaml['prometheus']['prometheusSpec']['resources'] as Map) containsKeys('limits', 'requests') - } - - @Test - void 'works with openshift'() { - config.application.openshift = true - when(k8sClient.getAnnotation('namespace', 'foo-monitoring', 'openshift.io/sa.scc.uid-range')) - .thenReturn('1000920000/10000') - createStack(scmManagerMock).install() - - def yaml = parseActualYaml() - assertThat(yaml['prometheusOperator']['securityContext']).isNotNull() - assertThat(yaml['prometheusOperator']['securityContext']['fsGroup']).isNull() - assertThat(yaml['prometheusOperator']['securityContext']['runAsGroup']).isNull() - assertThat(yaml['prometheusOperator']['securityContext']['runAsUser']).isNull() - - assertThat(yaml['grafana']['securityContext']).isNotNull() - assertThat(yaml['grafana']['securityContext']['fsGroup']).isEqualTo(1000920000) - assertThat(yaml['grafana']['securityContext']['runAsGroup']).isEqualTo(1000920000) - assertThat(yaml['grafana']['securityContext']['runAsUser']).isEqualTo(1000920000) - - assertThat(yaml['prometheus']['prometheusSpec']['securityContext']).isNotNull() - assertThat(yaml['prometheus']['prometheusSpec']['securityContext']['fsGroup']).isNull() - assertThat(yaml['prometheus']['prometheusSpec']['securityContext']['runAsGroup']).isNull() - assertThat(yaml['prometheus']['prometheusSpec']['securityContext']['runAsUser']).isNull() - } - - @Test - void 'works with namespaceIsolation'() { - config.application.namespaceIsolation = true - - def prometheusStack = createStack(scmManagerMock) - prometheusStack.install() - - def yaml = parseActualYaml() - assertThat(yaml['global']['rbac']['create']).isEqualTo(false) - - for (String namespace : config.application.namespaces.getActiveNamespaces()) { - def rbacYaml = new File("$clusterResourcesRepoDir/apps/monitoring/misc/rbac/${namespace}.yaml") - assertThat(rbacYaml.text).contains("namespace: ${namespace}") - assertThat(rbacYaml.text).contains(" namespace: foo-monitoring") - } - - assertThat(yaml['kubeApiServer']['enabled']).isEqualTo(false) - - assertThat(yaml['prometheusOperator']['kubeletService']['enabled']).isEqualTo(false) - assertThat(yaml['prometheusOperator']['namespaces']['releaseNamespace']).isEqualTo(false) - assertThat(yaml['prometheusOperator']['namespaces']['additional'] as List).hasSameElementsAs(config.application.namespaces.getActiveNamespaces()) - - assertThat(yaml['grafana']['rbac']['create']).isEqualTo(false) - assertThat(yaml['grafana']['sidecar']['dashboards']['searchNamespace']).isEqualTo(config.application.namespaces.getActiveNamespaces().join(',')) - } - - @Test - void 'network policies are created for prometheus'() { - config.application.netpols = true - //config.application.namespaces.dedicatedNamespaces = ["testnamespace1", "testnamespace2"] - def prometheusStack = createStack(scmManagerMock) - prometheusStack.install() - - for (String namespace : config.application.namespaces.getActiveNamespaces()) { - def netPolsYaml = new File("$clusterResourcesRepoDir/apps/monitoring/misc/netpols/${namespace}.yaml") - assertThat(netPolsYaml.text).contains("namespace: ${namespace}") - } - } - - @Test - void 'helm releases are installed in air-gapped mode'() { - config.application.mirrorRepos = true - when(airGappedUtils.mirrorHelmRepoToGit(any(Config.HelmConfig))).thenReturn('a/b') - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path prometheusSourceChart = rootChartsFolder.resolve('kube-prometheus-stack') - Files.createDirectories(prometheusSourceChart) - - Map prometheusChartYaml = [version: '1.2.3'] - fileSystemUtils.writeYaml(prometheusChartYaml, prometheusSourceChart.resolve('Chart.yaml').toFile()) - - scmManagerMock.inClusterBase = new URI("http://scmm.foo-scm-manager.svc.cluster.local/scm") - createStack(scmManagerMock).install() - - def helmConfig = ArgumentCaptor.forClass(Config.HelmConfig) - verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()) - assertThat(helmConfig.value.chart).isEqualTo('kube-prometheus-stack') - assertThat(helmConfig.value.repoURL).isEqualTo('https://prom') - assertThat(helmConfig.value.version).isEqualTo('19.2.2') - verify(deploymentStrategy).deployFeature('http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b', - 'monitoring', '.', '1.2.3', 'foo-monitoring', - 'kube-prometheus-stack', temporaryYamlFilePrometheus, RepoType.GIT) - } - - @Test - void 'Merges additional helm values merged with default values'() { - config.features.monitoring.helm.values = [key : [some: 'thing', - one : 1], - prometheus: [prometheusSpec: [scrapeConfigSelectorNilUsesHelmValues: null]]] - - createStack(scmManagerMock).install() - def actual = parseActualYaml() - - assertThat(actual['key']['some']).isEqualTo('thing') - assertThat(actual['key']['one']).isEqualTo(1) - assertThat(actual['prometheus']['prometheusSpec']['scrapeConfigSelectorNilUsesHelmValues']).isEqualTo(null) - } - - @Test - void 'ServiceMonitor selectors'() { - config.application.namePrefix = "test1-" - config.features.argocd.active = true - config.features.secrets.active = true - config.features.ingress.active = false - LinkedHashSet namespaceList = ["test1-argocd", - "test1-monitoring", - "test1-example-apps-staging", - "test1-example-apps-production", - "test1-secrets"] - config.application.namespaces.dedicatedNamespaces = namespaceList - createStack(scmManagerMock).install() - def actual = parseActualYaml() - - assertThat(actual['prometheus']['prometheusSpec']['serviceMonitorNamespaceSelector']).isEqualTo(new YamlSlurper().parseText(''' -matchExpressions: - - key: kubernetes.io/metadata.name - operator: In - values: - - test1-argocd - - test1-monitoring - - test1-example-apps-staging - - test1-example-apps-production - - test1-secrets -''')) - } - - private Monitoring createStack(ScmManagerMock scmManagerMock) { - // We use the real FileSystemUtils and not a mock to make sure file editing works as expected - when(gitHandler.getResourcesScm()).thenReturn(scmManagerMock) - def configuration = config - TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, new FileSystemUtils()) { - @Override - GitRepo getRepo(String repoTarget, GitProvider scm) { - def repo = super.getRepo(repoTarget, scmManagerMock) - clusterResourcesRepoDir = new File(repo.getAbsoluteLocalRepoTmpDir()) - - // Create dummy dashboards so cleanupUnusedDashboards can delete them - def dashboardDir = new File(clusterResourcesRepoDir, "apps/monitoring/misc/dashboard") - dashboardDir.mkdirs() - - new File(dashboardDir, "traefik-dashboard.yaml").text = "dummy" - new File(dashboardDir, "traefik-dashboard-requests-handling.yaml").text = "dummy" - new File(dashboardDir, "jenkins-dashboard.yaml").text = "dummy" - new File(dashboardDir, "scmm-dashboard.yaml").text = "dummy" - - return repo - } - - } - - new Monitoring(configuration, new FileSystemUtils() { - @Override - Path writeTempFile(Map mapValues) { - def ret = super.writeTempFile(mapValues) - temporaryYamlFilePrometheus = Path.of(ret.toString().replace(".ftl", "")) - return ret - } - }, deploymentStrategy, k8sClient, airGappedUtils, repoProvider, gitHandler) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFilePrometheus) as Map - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/RegistryTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/RegistryTest.groovy deleted file mode 100644 index c3ba2e6e1..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/RegistryTest.groovy +++ /dev/null @@ -1,83 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.CommandExecutorForTest -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.K8sClientForTest -import groovy.yaml.YamlSlurper -import org.junit.jupiter.api.Test - -import java.nio.file.Path - -import static com.cloudogu.gitops.config.Config.* -import static org.assertj.core.api.Assertions.assertThat - -class RegistryTest { - - K8sClientForTest k8sClient - CommandExecutorForTest helmCommands - HelmClient helmClient - Path temporaryYamlFile - - @Test - void 'is disabled when external registry is configured'() { - createRegistry().install() - - assertThat(helmCommands.actualCommands).isEmpty() - } - - @Test - void 'is installed'() { - createRegistry(new RegistrySchema(active: true)).install() - - assertThat(parseActualYaml()['service']['nodePort']).isEqualTo(DEFAULT_REGISTRY_PORT) - assertThat(parseActualYaml()['service']['type']).isEqualTo('NodePort') - assertThat(helmCommands.actualCommands[0].trim()).startsWith('helm repo add registry') - assertThat(helmCommands.actualCommands[1].trim()).startsWith('helm upgrade -i docker-registry registry/docker-registry --create-namespace') - assertThat(helmCommands.actualCommands[1].trim()).contains('--version') - assertThat(helmCommands.actualCommands[1].trim()).contains("--values ${temporaryYamlFile}") - assertThat(helmCommands.actualCommands[1].trim()).contains('--namespace foo-registry') - } - - @Test - void 'inject custom value into chart'() { - def registryConfig = new RegistrySchema(active: true, - helm: new HelmConfigWithValues(chart: 'test', - values: [service : [type: 'NodePortTest'], - customValue: 'testinjectionValue'])) - - createRegistry(registryConfig).install() - assertThat(parseActualYaml()['service'] as String).contains('NodePortTest') - assertThat(parseActualYaml()['customValue'] as String).contains('testinjectionValue') - } - - private Registry createRegistry(RegistrySchema registryConfig = new RegistrySchema()) { - def config = new Config(application: new ApplicationSchema(namePrefix: 'foo-'), - registry: registryConfig) - k8sClient = new K8sClientForTest() - helmCommands = new CommandExecutorForTest() - helmClient = new HelmClient(helmCommands) - - FileSystemUtils fileUtil = new FileSystemUtils() { - @Override - Path writeTempFile(Map mergeMap) { - def ret = super.writeTempFile(mergeMap) - temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")) - // Path after template invocation - return ret - } - } - AirGappedUtils airGappedUtils = new AirGappedUtils(config,null,fileUtil,helmClient, null) - // We use the real FileSystemUtils and not a mock to make sure file editing works as expected - new Registry(config,fileUtil, k8sClient, airGappedUtils, new HelmStrategy(config, helmClient)) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFile) as Map - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/VaultTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/VaultTest.groovy deleted file mode 100644 index 0f393beec..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/VaultTest.groovy +++ /dev/null @@ -1,236 +0,0 @@ -package com.cloudogu.gitops.tools - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.CommandExecutorForTest -import com.cloudogu.gitops.utils.FileSystemUtils -import groovy.yaml.YamlSlurper -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.mockito.ArgumentCaptor - -import java.nio.file.Files -import java.nio.file.Path - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.* - -@EnableKubernetesMockClient(crud = true) -class VaultTest { - - Config config = new Config(application: new Config.ApplicationSchema(namePrefix: 'foo-',), - features: new Config.FeaturesSchema(secrets: new Config.SecretsSchema(active: true,))) - - CommandExecutorForTest helmCommands = new CommandExecutorForTest() - FileSystemUtils fileSystemUtils = new FileSystemUtils() - DeploymentStrategy deploymentStrategy = mock(DeploymentStrategy) - AirGappedUtils airGappedUtils = mock(AirGappedUtils) - GitHandler gitHandler = new GitHandlerForTests(config, new ScmManagerMock()) - Path temporaryYamlFile - - K8sClient k8sClient - KubernetesClient client - - @BeforeEach - void init() { - k8sClient = new K8sClient() - k8sClient.client = client - } - - @Test - void 'is disabled via active flag'() { - config.features.secrets.active = false - createVault().install() - assertThat(helmCommands.actualCommands).isEmpty() - } - - @Test - void 'uses ingress if enabled'() { - config.features.secrets.vault.url = 'http://vault.local' - createVault().install() - - def ingressYaml = parseActualYaml()['server']['ingress'] - assertThat(ingressYaml['enabled']).isEqualTo(true) - assertThat((ingressYaml['hosts'] as List)[0]['host']).isEqualTo('vault.local') - } - - @Test - void 'uses ingress if enabled and image set'() { - config.features.secrets.vault.url = 'http://vault.local' - // Also set image to make sure ingress and image work at the same time under the server block - //config.features.secrets.vault.helm.image = 'localhost:5000/hashicorp/vault:1.12.0' - createVault().install() - - def ingressYaml = parseActualYaml()['server']['ingress'] - assertThat(ingressYaml['enabled']).isEqualTo(true) - } - - @Test - void 'does not use ingress by default'() { - createVault().install() - - assertThat(parseActualYaml()).doesNotContainKey('server') - } - - @Test - void 'Dev mode can be enabled via config'() { - config.features.secrets.vault.mode = 'dev' - config.application.username = 'abc' - config.application.password = '123' - config.features.argocd.active = true - - def vault = createVault() - - vault.install() - - def actualYaml = parseActualYaml() - assertThat(actualYaml['server']['dev']['enabled']).isEqualTo(true) - - assertThat(actualYaml['server']['dev']['devRootToken']).isNotEqualTo('root') - assertThat(actualYaml['server']['dev']['devRootToken']).isNotEqualTo(config.application.password) - - List actualPostStart = (List) actualYaml['server']['postStart'] - assertThat(actualPostStart[0]).isEqualTo('/bin/sh') - assertThat(actualPostStart[1]).isEqualTo('-c') - - assertThat(actualPostStart[2]).isEqualTo('USERNAME=abc PASSWORD=123 ARGOCD=true /var/opt/scripts/dev-post-start.sh 2>&1 | tee /tmp/dev-post-start.log') - - List actualVolumes = actualYaml['server']['volumes'] as List - List actualVolumeMounts = actualYaml['server']['volumeMounts'] as List - assertThat(actualVolumes[0]['name']).isEqualTo(actualVolumeMounts[0]['name']) - assertThat(actualVolumes[0]['configMap']['defaultMode']).isEqualTo(Integer.valueOf(0774)) - - assertThat(actualVolumeMounts[0]['readOnly']).is(true) - assertThat(actualPostStart[2] as String).contains(actualVolumeMounts[0]['mountPath'] as String + "/dev-post-start.sh") - - assertThat(actualYaml['server'] as Map).doesNotContainKey('resources') - } - - @Test - void 'Dev mode can be enabled via config with argoCD disabled'() { - config.features.secrets.vault.mode = 'dev' - config.application.username = 'abc' - config.application.password = '123' - createVault().install() - - def actualYaml = parseActualYaml() - List actualPostStart = (List) actualYaml['server']['postStart'] - assertThat(actualPostStart[2]).isEqualTo('USERNAME=abc PASSWORD=123 ARGOCD=false /var/opt/scripts/dev-post-start.sh 2>&1 | tee /tmp/dev-post-start.log') - } - - @Test - void 'Prod mode can be enabled'() { - config.features.secrets.vault.mode = 'prod' - createVault().install() - - assertThat(parseActualYaml()).doesNotContainKey('server') - } - - @Test - void 'custom image is used'() { - config.features.secrets.vault.helm.image = 'localhost:5000/hashicorp/vault:1.12.0' - createVault().install() - - def actualYaml = parseActualYaml() - assertThat(actualYaml['server']['image']['repository']).isEqualTo('localhost:5000/hashicorp/vault') - assertThat(actualYaml['server']['image']['tag']).isEqualTo('1.12.0') - } - - @Test - void 'helm release is installed'() { - config.features.secrets.vault.helm = new Config.SecretsSchema.VaultSchema.VaultHelmSchema(chart: 'vault', - repoURL: 'https://vault-reg', - version: '42.23.0') - createVault().install() - - verify(deploymentStrategy).deployFeature('https://vault-reg', - 'vault', - 'vault', - '42.23.0', - 'foo-secrets', - 'vault', - temporaryYamlFile, - RepoType.HELM) - - assertThat(parseActualYaml()).doesNotContainKey('global') - } - - @Test - void 'helm release is installed in air-gapped mode'() { - config.application.mirrorRepos = true - config.features.secrets.vault.helm = new Config.SecretsSchema.VaultSchema.VaultHelmSchema(chart: 'vault', - repoURL: 'https://vault-reg', - version: '42.23.0') - - when(airGappedUtils.mirrorHelmRepoToGit(any(Config.HelmConfig))).thenReturn('a/b') - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - config.application.localHelmChartFolder = rootChartsFolder.toString() - - Path SourceChart = rootChartsFolder.resolve('vault') - Files.createDirectories(SourceChart) - - Map ChartYaml = [version: '1.2.3'] - fileSystemUtils.writeYaml(ChartYaml, SourceChart.resolve('Chart.yaml').toFile()) - - createVault().install() - - def helmConfig = ArgumentCaptor.forClass(Config.HelmConfig) - verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()) - assertThat(helmConfig.value.chart).isEqualTo('vault') - assertThat(helmConfig.value.repoURL).isEqualTo('https://vault-reg') - assertThat(helmConfig.value.version).isEqualTo('42.23.0') - verify(deploymentStrategy).deployFeature('http://scmm.scm-manager.svc.cluster.local/scm/repo/a/b', - 'vault', '.', '1.2.3', 'foo-secrets', - 'vault', temporaryYamlFile, RepoType.GIT) - } - - @Test - void 'Sets pod resource limits and requests'() { - config.application.podResources = true - - createVault().install() - - def actualYaml = parseActualYaml() - assertThat(actualYaml['server']['resources'] as Map).containsKeys('limits', 'requests') - } - - @Test - void 'deploys image pull secrets for proxy registry'() { - config.registry.createImagePullSecrets = true - config.registry.proxyUrl = 'proxy-url' - config.registry.proxyUsername = 'proxy-user' - config.registry.proxyPassword = 'proxy-pw' - - createVault().install() - - assertThat(parseActualYaml()['global']['imagePullSecrets']).isEqualTo([[name: 'proxy-registry']]) - } - - private Vault createVault() { - // We use the real FileSystemUtils and not a mock to make sure file editing works as expected - - new Vault(config, new FileSystemUtils() { - @Override - Path writeTempFile(Map mapValues) { - def ret = super.writeTempFile(mapValues) - temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")) - return ret - } - }, k8sClient, deploymentStrategy, airGappedUtils, gitHandler) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFile) as Map - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/common/ToolTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/common/ToolTest.groovy deleted file mode 100644 index 49a19c014..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/common/ToolTest.groovy +++ /dev/null @@ -1,79 +0,0 @@ -package com.cloudogu.gitops.tools.common - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test - -@EnableKubernetesMockClient(crud = true) -class ToolTest { - Config config = new Config(application: new Config.ApplicationSchema(namePrefix: "foo-")) - - K8sClient k8sClient - KubernetesClient client - - @BeforeEach - void init() { - k8sClient = new K8sClient() - k8sClient.client = client - } - - @Test - void 'Image pull secrets are create automatically'() { - config.registry.createImagePullSecrets = true - config.registry.proxyUrl = 'proxy-url' - config.registry.proxyUsername = 'proxy-user' - config.registry.proxyPassword = 'proxy-pw' - config.registry.url = 'url' - config.registry.readOnlyUsername = 'ROuser' - config.registry.readOnlyPassword = 'ROpw' - config.registry.username = 'user' - config.registry.password = 'pw' - - createFeatureWithImage().install() - } - - protected ToolWithImageForTest createFeatureWithImage() { - Tool feature = new ToolWithImageForTest() - feature.config = config - feature.k8sClient = k8sClient - feature.namespace = 'foo-my-ns' - feature - } - - @Test - void 'Image pull secrets: Falls back to using readOnly credentials and URL '() { - config.registry.createImagePullSecrets = true - config.registry.url = 'url' - config.registry.readOnlyUsername = 'ROuser' - config.registry.readOnlyPassword = 'ROpw' - config.registry.username = 'user' - config.registry.password = 'pw' - - createFeatureWithImage().install() - } - - @Test - void 'Image pull secrets: Falls back to using credentials and URL '() { - config.registry.createImagePullSecrets = true - config.registry.url = 'url' - config.registry.username = 'user' - config.registry.password = 'pw' - - createFeatureWithImage().install() - } - - class ToolWithImageForTest extends Tool implements ToolWithImage { - - String namespace - Config config - K8sClient k8sClient - - @Override - boolean isEnabled() { - return true - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/core/JenkinsTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/core/JenkinsTest.groovy deleted file mode 100644 index 832c25ef2..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/core/JenkinsTest.groovy +++ /dev/null @@ -1,366 +0,0 @@ -package com.cloudogu.gitops.tools.core - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.config.scm.ScmTenantSchema -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.jenkins.GlobalPropertyManager -import com.cloudogu.gitops.infrastructure.jenkins.JobManager -import com.cloudogu.gitops.infrastructure.jenkins.PrometheusConfigurator -import com.cloudogu.gitops.infrastructure.jenkins.UserManager -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.utils.AirGappedUtils -import com.cloudogu.gitops.utils.CommandExecutorForTest -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.NetworkingUtils -import groovy.yaml.YamlSlurper -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.mockito.ArgumentCaptor -import org.mockito.Mock - -import java.nio.file.Path - -import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.* -import static org.mockito.Mockito.* - -class JenkinsTest { - Config config = new Config(scm: new ScmTenantSchema(scmManager: new ScmTenantSchema.ScmManagerTenantConfig(urlForJenkins: "testUrlJenkins")), - jenkins: new Config.JenkinsSchema(active: true)) - - String expectedNodeName = 'something' - - CommandExecutorForTest commandExecutor = new CommandExecutorForTest() - GlobalPropertyManager globalPropertyManager = mock(GlobalPropertyManager) - JobManager jobManger = mock(JobManager) - UserManager userManager = mock(UserManager) - PrometheusConfigurator prometheusConfigurator = mock(PrometheusConfigurator) - HelmStrategy deploymentStrategy = mock(HelmStrategy) - Path temporaryYamlFile - NetworkingUtils networkingUtils = mock(NetworkingUtils.class) - K8sClient k8sClient = mock(K8sClient) - - @Mock - ScmManagerMock scmManagerMock = new ScmManagerMock() - GitHandler gitHandler = new GitHandlerForTests(config, scmManagerMock) - - @BeforeEach - void setup() { - // waitForInternalNodeIp -> waitForNode() - when(k8sClient.waitForNode()).thenReturn("node/${expectedNodeName}".toString()) - when(k8sClient.run(anyString(), anyString(), anyString(), anyMap(), any())).thenReturn('') - } - - @Test - void 'Installs Jenkins'() { - def jenkins = createJenkins() - - config.jenkins.url = 'http://jenkins' - config.jenkins.helm.chart = 'jen-chart' - config.jenkins.helm.repoURL = 'https://jen-repo' - config.jenkins.helm.version = '4.8.1' - config.jenkins.username = 'jenusr' - config.jenkins.password = 'jenpw' - config.jenkins.internalBashImage = 'bash:42' - config.jenkins.internalDockerClientVersion = '23' - - when(k8sClient.run(anyString(), anyString(), anyString(), anyMap(), any(String[].class))).thenReturn(''' -root:x:0: -daemon:x:1: -docker:x:42:me -me:x:1000:''') - - jenkins.install() - - verify(deploymentStrategy).deployFeature('https://jen-repo', 'jenkins', - 'jen-chart', '4.8.1', 'jenkins', - 'jenkins', temporaryYamlFile, RepoType.HELM) - verify(k8sClient).label('node', expectedNodeName, new Tuple2('node', 'jenkins')) - verify(k8sClient).labelRemove('node', '--all', '', 'node') - verify(k8sClient).createSecret('generic', 'jenkins-credentials', 'jenkins', - new Tuple2('jenkins-admin-user', 'jenusr'), - new Tuple2('jenkins-admin-password', 'jenpw')) - - assertThat(parseActualYaml()['dockerClientVersion'].toString()).isEqualTo('23') - - assertThat(parseActualYaml()['controller']['image']['tag']).isEqualTo('4.8.1') - - assertThat(parseActualYaml()['controller']['jenkinsUrl']).isEqualTo('http://jenkins') - assertThat(parseActualYaml()['controller']['serviceType']).isEqualTo('NodePort') - - assertThat(parseActualYaml()['controller']['ingress']).isNull() - - List customInitContainers = parseActualYaml()['controller']['customInitContainers'] as List - assertThat(customInitContainers[0]['image']).isEqualTo('bash:42') - - assertThat(parseActualYaml()['agent']['runAsUser']).isEqualTo(1000) - assertThat(parseActualYaml()['agent']['runAsGroup']).isEqualTo(42) - - ArgumentCaptor nameCaptor = ArgumentCaptor.forClass(String.class); - ArgumentCaptor overridesCaptor = ArgumentCaptor.forClass(Map.class); - verify(k8sClient).run(nameCaptor.capture(), anyString(), eq(jenkins.namespace), overridesCaptor.capture(), any(String[].class)) - assertThat(nameCaptor.value).startsWith('tmp-docker-gid-grepper-') - List containers = overridesCaptor.value['spec']['containers'] as List - assertThat(containers[0]['image'].toString()).isEqualTo('bash:42') - } - - @Test - void 'Installs Jenkins without dockerGid'() { - when(k8sClient.run(anyString(), anyString(), anyString(), anyMap(), any())).thenReturn(''' -root:x:0: -daemon:x:1: -me:x:1000:''') - createJenkins().install() - - assertThat(parseActualYaml()['agent']['runAsUser']).isEqualTo('0') - assertThat(parseActualYaml()['agent']['runAsGroup']).isEqualTo('133') - } - - @Test - void 'Installs only if internal'() { - config.jenkins.internal = false - - createJenkins().install() - verify(deploymentStrategy, never()).deployFeature(anyString(), anyString(), anyString(), anyString(), - anyString(), anyString(), any(Path)) - - assertThat(temporaryYamlFile).isNull() - } - - @Test - void 'Additional helm values are merged with default values'() { - config.jenkins.helm.values = [controller: [nodePort: 42]] - - createJenkins().install() - - assertThat(parseActualYaml()['controller']['nodePort']).isEqualTo(42) - } - - @Test - void 'Enables ingress when baseUrl is set'() { - config.jenkins.ingress = 'jenkins.localhost' - config.application.baseUrl = 'someBaseUrl' - - createJenkins().install() - - assertThat(parseActualYaml()['controller']['ingress']['enabled']).isEqualTo(true) - assertThat(parseActualYaml()['controller']['ingress']['hostName']).isEqualTo('jenkins.localhost') - } - - @Test - void 'Maps config properly'() { - config.application.trace = true - config.features.argocd.active = true - config.scm.scmManager.url = 'http://scmm.scm-manager.svc.cluster.local/scm' - config.scm.scmManager.username = 'scmm-usr' - config.scm.scmManager.password = 'scmm-pw' - config.application.namePrefix = 'my-prefix-' - config.application.namePrefixForEnvVars = 'MY_PREFIX_' - config.registry.url = 'reg-url' - config.registry.path = 'reg-path' - config.registry.username = 'reg-usr' - config.registry.password = 'reg-pw' - config.registry.proxyUrl = 'reg-proxy-url' - config.registry.proxyPath = 'reg-proxy-path' - config.registry.proxyUsername = 'reg-proxy-usr' - config.registry.proxyPassword = 'reg-proxy-pw' - config.jenkins.internal = false - config.jenkins.helm.version = '4.8.1' - config.jenkins.username = 'jenusr' - config.jenkins.password = 'jenpw' - config.jenkins.url = 'http://jenkins' - config.jenkins.metricsUsername = 'metrics-usr' - config.jenkins.metricsPassword = 'metrics-pw' - config.jenkins.skipPlugins = true - config.jenkins.skipRestart = true - - createJenkins().install() - - def env = getEnvAsMap() - assertThat(commandExecutor.actualCommands[0]).isEqualTo("${System.getProperty('user.dir')}/scripts/jenkins/init-jenkins.sh" as String) - - assertThat(env['TRACE']).isEqualTo('true') - assertThat(env['INTERNAL_JENKINS']).isEqualTo('false') - assertThat(env['JENKINS_HELM_CHART_VERSION']).isEqualTo('4.8.1') - assertThat(env['JENKINS_URL']).isEqualTo('http://jenkins') - assertThat(env['JENKINS_USERNAME']).isEqualTo('jenusr') - assertThat(env['JENKINS_PASSWORD']).isEqualTo('jenpw') - assertThat(env['JENKINS_USERNAME']).isEqualTo('jenusr') - assertThat(env['NAME_PREFIX']).isEqualTo('my-prefix-') - assertThat(env['INSECURE']).isEqualTo('false') - - assertThat(env['SCM_URL']).isEqualTo('http://scmm.scm-manager.svc.cluster.local/scm') - assertThat(env['SCM_PASSWORD']).isEqualTo(scmManagerMock.credentials.password) - assertThat(env['INSTALL_ARGOCD']).isEqualTo('true') - - assertThat(env['SKIP_PLUGINS']).isEqualTo('true') - assertThat(env['SKIP_RESTART']).isEqualTo('true') - - verify(globalPropertyManager).setGlobalProperty('MY_PREFIX_SCM_URL', 'http://scmm.scm-manager.svc.cluster.local/scm') - verify(globalPropertyManager).setGlobalProperty('MY_PREFIX_K8S_VERSION', Config.K8S_VERSION) - - verify(globalPropertyManager).setGlobalProperty('MY_PREFIX_REGISTRY_URL', 'reg-url') - verify(globalPropertyManager).setGlobalProperty('MY_PREFIX_REGISTRY_PATH', 'reg-path') - verify(globalPropertyManager, never()).setGlobalProperty(eq('MY_PREFIX_REGISTRY_PROXY_URL'), anyString()) - verify(globalPropertyManager, never()).setGlobalProperty(eq('MY_PREFIX_REGISTRY_PROXY_PATH'), anyString()) - verify(globalPropertyManager, never()).setGlobalProperty(eq('MAVEN_CENTRAL_MIRROR'), anyString()) - - verify(userManager).createUser('metrics-usr', 'metrics-pw') - verify(userManager).grantPermission('metrics-usr', UserManager.Permissions.METRICS_VIEW) - } - - @Test - void 'Does not configure prometheus when external Jenkins'() { - config.features.monitoring.active = true - config.jenkins.internal = false - - createJenkins().install() - - verify(prometheusConfigurator, never()).enableAuthentication() - } - - @Test - void 'Does not configure prometheus when monitoring off'() { - config.features.monitoring.active = false - config.jenkins.internal = true - - createJenkins().install() - - verify(prometheusConfigurator, never()).enableAuthentication() - } - - @Test - void 'Configures prometheus'() { - config.features.monitoring.active = true - config.jenkins.internal = true - - createJenkins().install() - - verify(prometheusConfigurator).enableAuthentication() - } - - @Test - void "URL: Use k8s service name if running as k8s pod"() { - config.jenkins.internal = true - config.application.runningInsideK8s = true - - createJenkins().install() - assertThat(config.jenkins.url).isEqualTo("http://jenkins.jenkins.svc.cluster.local:80") - } - - @Test - void "URL: Use local ip and nodePort when outside of k8s"() { - config.jenkins.internal = true - config.application.runningInsideK8s = false - - when(networkingUtils.findClusterBindAddress()).thenReturn('192.168.16.2') - when(k8sClient.waitForNodePort(anyString(), anyString())).thenReturn('42') - - createJenkins().install() - assertThat(config.jenkins.url).endsWith('192.168.16.2:42') - } - - @Test - void 'Handles two registries'() { - config.registry.twoRegistries = true - config.application.namePrefix = 'my-prefix-' - config.application.namePrefixForEnvVars = 'MY_PREFIX_' - - config.registry.url = 'reg-url' - config.registry.path = 'reg-path' - config.registry.username = 'reg-usr' - config.registry.password = 'reg-pw' - config.registry.proxyUrl = 'reg-proxy-url' - config.registry.proxyPath = 'reg-proxy-path' - config.registry.proxyUsername = 'reg-proxy-usr' - config.registry.proxyPassword = 'reg-proxy-pw' - - createJenkins().install() - - verify(globalPropertyManager).setGlobalProperty('MY_PREFIX_REGISTRY_PROXY_URL', 'reg-proxy-url') - verify(globalPropertyManager).setGlobalProperty('MY_PREFIX_REGISTRY_PROXY_PATH', 'reg-proxy-path') - - verify(globalPropertyManager).setGlobalProperty(eq('MY_PREFIX_REGISTRY_URL'), anyString()) - verify(globalPropertyManager).setGlobalProperty(eq('MY_PREFIX_REGISTRY_PATH'), anyString()) - - } - - @Test - void 'Does not create create job credentials when argo cd is deactivated'() { - config.application.namePrefixForEnvVars = 'MY_PREFIX_' - when(userManager.isUsingCasSecurityRealm()).thenReturn(true) - - createJenkins().install() - - verify(userManager, never()).createUser(anyString(), anyString()) - } - - @Test - void 'Global property is set for additional envs'() { - - config.jenkins.additionalEnvs = [ADDITIONAL_DOCKER_RUN_ARGS: '-u0:0'] - - createJenkins().install() - verify(globalPropertyManager).setGlobalProperty(eq('ADDITIONAL_DOCKER_RUN_ARGS'), eq('-u0:0')) - } - - @Test - void 'Does not create create user if CAS security realm is used'() { - config.features.argocd.active = false - - createJenkins().install() - verify(jobManger, never()).createCredential(anyString(), anyString(), anyString(), anyString(), anyString()) - verify(jobManger, never()).startJob(anyString()) - } - - @Test - void 'Properly handles null values'() { - config.application.baseUrl = null - createJenkins().install() - - def env = getEnvAsMap() - assertThat(env['BASE_URL']).isNotEqualTo('null') - } - - @Test - void 'Sets maven mirror '() { - config.registry.url = 'some value' - config.jenkins.mavenCentralMirror = 'http://test' - config.application.namePrefixForEnvVars = 'MY_PREFIX_' - - createJenkins().install() - - verify(globalPropertyManager).setGlobalProperty(eq('MY_PREFIX_MAVEN_CENTRAL_MIRROR'), eq("http://test")) - } - - protected Map getEnvAsMap() { - commandExecutor.environment.collectEntries { it.split('=') } - } - - private Jenkins createJenkins() { - when(networkingUtils.createUrl(anyString(), anyString(), anyString())).thenCallRealMethod() - when(networkingUtils.createUrl(anyString(), anyString())).thenCallRealMethod() - - FileSystemUtils fileSystemUtils = new FileSystemUtils() { - @Override - Path writeTempFile(Map mergeMap) { - def ret = super.writeTempFile(mergeMap) - temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")) - // Path after template invocation - return ret - } - } - AirGappedUtils airGappedUtils = new AirGappedUtils(config,null,fileSystemUtils,null, gitHandler) - - new Jenkins(config, commandExecutor, fileSystemUtils, globalPropertyManager,jobManger, userManager, prometheusConfigurator, deploymentStrategy, k8sClient, networkingUtils, airGappedUtils, gitHandler ) - } - - private Map parseActualYaml() { - def ys = new YamlSlurper() - return ys.parse(temporaryYamlFile) as Map - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/core/ScmManagerSetupTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/core/ScmManagerSetupTest.groovy deleted file mode 100644 index d91f10a44..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/core/ScmManagerSetupTest.groovy +++ /dev/null @@ -1,78 +0,0 @@ -package com.cloudogu.gitops.tools.core - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManager -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.PluginApi -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApi -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient -import org.junit.jupiter.api.Test -import retrofit2.Call -import retrofit2.Response - -import static org.mockito.ArgumentMatchers.any -import static org.mockito.ArgumentMatchers.eq -import static org.mockito.Mockito.* - -class ScmManagerSetupTest { - - ScmManager scmManager = mock(ScmManager.class) - - HelmStrategy helmStrategy = mock(HelmStrategy.class) - ScmManagerApiClient apiClient = mock(ScmManagerApiClient.class) - - PluginApi pluginApi = mock(PluginApi.class) - ScmManagerApi generalApi = mock(ScmManagerApi.class) - - Config config = Config.fromMap([application: [namePrefix: 'test',], - scm : [scmManager: [internal : true, - url : "", - namespace : "scm-manager", - username : "admin", - password : "admin", - helm : [chart : "scm-manager", - repoURL: "https://packages.scm-manager.org/repository/helm-v2-releases/", - version: "3.11.2", - values : [:]], - urlForJenkins : "http://scmm.scm-manager.svc.cluster.local/scm", - ingress : "scmm.master.localhost", - skipRestart : false, - skipPlugins : false, - gitOpsUsername: ""]]]) - - @Test - void 'Helm chart is installed correctly'() { - when(scmManager.getConfig()).thenReturn(config) - when(scmManager.getHelmStrategy()).thenReturn(helmStrategy) - when(scmManager.getScmmConfig()).thenReturn(config.scm.scmManager) - ScmManagerSetup scmManagerSetup = new ScmManagerSetup(scmManager) - scmManagerSetup.setupHelm() - verify(helmStrategy).deployFeature(eq("https://packages.scm-manager.org/repository/helm-v2-releases/"), - eq("scm-manager"), - any(), - eq("3.11.2"), - eq("scm-manager"), - eq("scmm"), - any()) - } - - @Test - void 'ScmManager Plugins are installed correctly'() { - when(scmManager.getConfig()).thenReturn(config) - when(scmManager.getHelmStrategy()).thenReturn(helmStrategy) - when(scmManager.getScmmConfig()).thenReturn(config.scm.scmManager) - when(scmManager.getApiClient()).thenReturn(apiClient) - - Call apiCall = mock(Call.class) - - when(pluginApi.install(any(), any())).thenReturn(apiCall) - when(generalApi.checkScmmAvailable()).thenReturn(apiCall) - when(apiClient.pluginApi()).thenReturn(pluginApi) - when(apiClient.generalApi()).thenReturn(generalApi) - when(apiCall.execute()).thenReturn(Response.success(null)) - ScmManagerSetup scmManagerSetup = new ScmManagerSetup(scmManager) - scmManagerSetup.installScmmPlugins() - verify(pluginApi, atLeast(10)).install(any(), any()) - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetupTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetupTest.groovy deleted file mode 100644 index b8745c86b..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetupTest.groovy +++ /dev/null @@ -1,200 +0,0 @@ -package com.cloudogu.gitops.tools.core.argocd - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.TestGitProvider -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.utils.FileSystemUtils -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test - -import java.nio.file.Path - -import static org.assertj.core.api.Assertions.assertThat -import static org.junit.jupiter.api.Assertions.assertThrows - -class ArgoCDRepoSetupTest { - - Config config - GitProvider tenantProvider - GitProvider centralProvider - - @BeforeEach - void setUp() { - config = Config.fromMap(application: [namePrefix: '', - netpols : true, - namespaces: [dedicatedNamespaces: ["argocd", "monitoring", "secrets"], - tenantNamespaces : ["example-apps-staging", "example-apps-production"]]], - scm: [scmManager: [internal: true], - gitlab : [url: '']], - multiTenant: [scmManager : [url: ''], - gitlab : [url: ''], - useDedicatedInstance : false, - centralArgocdNamespace: 'argocd'], - features: [argocd : [operator : false, - active : true, - namespace: 'argocd'], - certManager: [active: false], - ingress : [active: true], - monitoring : [active: true, helm: [chart: 'kube-prometheus-stack', version: '42.0.3']], - mail : [active: false], - secrets : [active: true],]) - - def providers = TestGitProvider.buildProviders(config) - tenantProvider = providers.tenant as GitProvider - centralProvider = providers.central as GitProvider - } - - private ArgoCDRepoSetup createSetup(FileSystemUtils fs) { - def repoFactory = new TestGitRepoFactory(config, new FileSystemUtils()) - repoFactory.defaultProvider = tenantProvider - - def gitHandler = new GitHandlerForTests(config, tenantProvider, centralProvider) - return ArgoCDRepoSetup.create(config, fs, repoFactory, gitHandler) - } - - @Test - void 'create() single instance creates only cluster-resources and no tenantBootstrap'() { - config.multiTenant.useDedicatedInstance = false - - def setup = createSetup(new FileSystemUtils()) - - assertThat(setup.tenantBootstrap).isNull() - assertThat(setup.clusterResources).isNotNull() - assertThat(setup.allRepos).hasSize(1) - assertThat(setup.clusterResources.repo.repoTarget).isEqualTo('argocd/cluster-resources') - } - - @Test - void 'create() dedicated instance creates tenantBootstrap and clusterResources'() { - config.multiTenant.useDedicatedInstance = true - - def setup = createSetup(new FileSystemUtils()) - - assertThat(setup.tenantBootstrap).isNotNull() - assertThat(setup.clusterResources).isNotNull() - assertThat(setup.allRepos).hasSize(2) - } - - @Test - void 'tenantRepoLayout throws in single instance mode'() { - config.multiTenant.useDedicatedInstance = false - - def setup = createSetup(new FileSystemUtils()) - - assertThrows(IllegalStateException) { - setup.tenantRepoLayout() - } - } - - @Test - void 'prepareClusterResourcesRepo deletes helmDir when operator is enabled'() { - config.features.argocd.operator = true - config.multiTenant.useDedicatedInstance = false - config.application.netpols = true - def setup = createSetup(new FileSystemUtils()) - - setup.initLocalRepos() - setup.prepareClusterResourcesRepo() - - def clusterRepoLayout = setup.clusterRepoLayout() - assertThat(Path.of(clusterRepoLayout.helmDir())).doesNotExist() - } - - @Test - void 'prepareClusterResourcesRepo deletes operatorDir when operator is disabled'() { - config.features.argocd.operator = false - config.multiTenant.useDedicatedInstance = false - config.application.netpols = true - - def setup = createSetup(new FileSystemUtils()) - - setup.initLocalRepos() - setup.prepareClusterResourcesRepo() - - def clusterRepoLayout = setup.clusterRepoLayout() - assertThat(Path.of(clusterRepoLayout.operatorDir())).doesNotExist() - assertThat(Path.of(clusterRepoLayout.helmDir())).exists() - - } - - @Test - void 'prepareClusterResourcesRepo in dedicated mode deletes multiTenant folder'() { - config.features.argocd.operator = false - config.multiTenant.useDedicatedInstance = true - config.application.netpols = true - - def setup = createSetup(new FileSystemUtils()) - - setup.initLocalRepos() - setup.prepareClusterResourcesRepo() - - def clusterRepoLayout = setup.clusterRepoLayout() - - assertThat(Path.of(clusterRepoLayout.applicationsDir())).exists() - assertThat(Path.of(clusterRepoLayout.projectsDir())).exists() - assertThat(Path.of(clusterRepoLayout.multiTenantDir())).doesNotExist() - } - - @Test - void 'prepareClusterResourcesRepo in single instance deletes multiTenant folder'() { - config.features.argocd.operator = false - config.multiTenant.useDedicatedInstance = false - config.application.netpols = true - - def setup = createSetup(new FileSystemUtils()) - - setup.initLocalRepos() - setup.prepareClusterResourcesRepo() - - def clusterRepoLayout = setup.clusterRepoLayout() - assertThat(Path.of(clusterRepoLayout.multiTenantDir())).doesNotExist() - } - - @Test - void 'prepareClusterResourcesRepo deletes netpol file when netpols disabled'() { - config.application.netpols = false - - def setup = createSetup(new FileSystemUtils()) - - setup.initLocalRepos() - setup.prepareClusterResourcesRepo() - - def clusterRepoLayout = setup.clusterRepoLayout() - assertThat(Path.of(clusterRepoLayout.netpolFile())).doesNotExist() - } - - @Test - void 'create() sets subDirsToCopy based on enabled features'() { - config.features.ingress.active = true - config.features.monitoring.active = false - config.features.secrets.active = false - config.jenkins.active = false - config.features.mail.active = false - config.features.certManager.active = false - - def setup = createSetup(new FileSystemUtils()) - def dirs = setup.clusterResources.subDirsToCopy as Set - - assertThat(dirs).contains(RepoLayout.argocdSubdirRel()) - assertThat(dirs).contains(RepoLayout.ingressSubdirRel()) - - assertThat(dirs).doesNotContain(RepoLayout.monitoringSubdirRel()) - assertThat(dirs).doesNotContain(RepoLayout.secretsSubdirRel()) - assertThat(dirs).doesNotContain(RepoLayout.vaultSubdirRel()) - assertThat(dirs).doesNotContain(RepoLayout.jenkinsSubdirRel()) - assertThat(dirs).doesNotContain(RepoLayout.certManagerSubdirRel()) - } - - @Test - void 'create() includes secrets + vault subdirs when secrets feature active'() { - config.features.secrets.active = true - - def setup = createSetup(new FileSystemUtils()) - def dirs = setup.clusterResources.subDirsToCopy as Set - - assertThat(dirs).contains(RepoLayout.secretsSubdirRel()) - assertThat(dirs).contains(RepoLayout.vaultSubdirRel()) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDTest.groovy b/src/test/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDTest.groovy deleted file mode 100644 index 12ed201f5..000000000 --- a/src/test/groovy/com/cloudogu/gitops/tools/core/argocd/ArgoCDTest.groovy +++ /dev/null @@ -1,1602 +0,0 @@ -package com.cloudogu.gitops.tools.core.argocd - -import static com.github.stefanbirkner.systemlambda.SystemLambda.withEnvironmentVariable -import static org.assertj.core.api.Assertions.assertThat -import static org.assertj.core.api.AssertionsForClassTypes.assertThatCode -import static org.mockito.ArgumentMatchers.any -import static org.mockito.Mockito.* - -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.providers.GitProvider -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.TestGitProvider -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.utils.CommandExecutor -import com.cloudogu.gitops.utils.CommandExecutorForTest -import com.cloudogu.gitops.utils.FileSystemUtils -import com.cloudogu.gitops.utils.K8sClientForTest - -import java.nio.file.Files -import java.nio.file.Path -import java.util.stream.Collectors -import groovy.io.FileType -import groovy.json.JsonSlurper -import groovy.yaml.YamlSlurper - -import io.fabric8.kubernetes.api.model.NamespaceBuilder -import io.fabric8.kubernetes.api.model.Secret -import io.fabric8.kubernetes.api.model.SecretBuilder -import io.fabric8.kubernetes.api.model.apiextensions.v1.CustomResourceDefinition -import io.fabric8.kubernetes.api.model.apiextensions.v1.CustomResourceDefinitionBuilder -import io.fabric8.kubernetes.client.KubernetesClient -import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient -import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test -import org.mockito.Spy -import org.springframework.security.crypto.bcrypt.BCrypt - -@EnableKubernetesMockClient(crud = true) -class ArgoCDTest { - Map buildImages = [kubectl : 'kubectl-value', - helm : 'helm-value', - kubeval : 'kubeval-value', - helmKubeval: 'helmKubeval-value', - yamllint : 'yamllint-value'] - - Config config = Config.fromMap(application: [openshift : false, - insecure : false, - password : '123', - username : 'something', - namePrefix : '', - namePrefixForEnvVars: '', - gitName : 'Cloudogu', - gitEmail : 'hello@cloudogu.com', - namespaces : [dedicatedNamespaces: ["argocd", "monitoring", "traefik", "secrets"], - tenantNamespaces : ["example-apps-staging", "example-apps-production"]]], - scm: [scmManager: [internal: true], - gitlab : [url: '']], - multiTenant: [scmManager : [url: ''], - gitlab : [url: ''], - useDedicatedInstance: false], - content: [repos : [[url : 'https://github.com/cloudogu/gitops-build-lib', - target : '3rd-party-dependencies/gitops-build-lib', - overwriteMode: 'RESET'], - [url : 'https://github.com/cloudogu/ces-build-lib', - target : '3rd-party-dependencies/ces-build-lib', - overwriteMode: 'RESET'], - [url : 'https://github.com/cloudogu/spring-boot-helm-chart', - target : '3rd-party-dependencies/spring-boot-helm-chart', - overwriteMode: 'RESET'], - [url : 'https://github.com/cloudogu/spring-petclinic', - target : 'argocd/petclinic-plain', - ref : 'feature/gitops_ready', - targetRef : 'main', - overwriteMode : 'UPGRADE', - createJenkinsJob: true], - [url : 'https://github.com/cloudogu/spring-petclinic', - target : 'argocd/petclinic-helm', - ref : 'feature/gitops_ready', - targetRef : 'main', - overwriteMode : 'UPGRADE', - createJenkinsJob: true], - [url : 'https://github.com/cloudogu/gitops-playground', - path : 'example-apps-via-content-loader/', - ref : 'main', - templating : true, - type : 'FOLDER_BASED', - overwriteMode: 'UPGRADE']], - namespaces: ["example-apps-production", - "example-apps-staging"], - variables : [petclinic: [baseDomain: 'petclinic.localhost'], - images : [kubectl : 'alpine/kubectl:1.35.0', - helm : 'ghcr.io/cloudogu/helm:4.2.1-1', - kubeval : 'ghcr.io/cloudogu/helm:4.2.1-1', - helmKubeval: 'ghcr.io/cloudogu/helm:4.2.1-1', - yamllint : 'cytopia/yamllint:1.25-0.7', - petclinic : 'eclipse-temurin:17-jre-alpine', - maven : '']]], - features: [argocd : [operator : false, - active : true, - configOnly : true, - emailFrom : 'argocd@example.org', - emailToUser : 'app-team@example.org', - emailToAdmin : 'infra@example.org', - resourceInclusionsCluster: ''], - monitoring: [active: true, - helm : [chart : 'kube-prometheus-stack', - version: '42.0.3']], - ingress : [active: true], - secrets : [active: true, - - ]]) - - @Spy - CommandExecutor test = new CommandExecutor() - KubernetesClient client - KubernetesMockServer server - K8sClient k8sClient - - CommandExecutorForTest helmCommands = new CommandExecutorForTest() - // GitRepo argocdRepo - String actualHelmValuesFile - GitRepo clusterResourcesRepo - List petClinicRepos = [] - ArgoCD argocd - RepoLayout clusterResourcesRepoLayout - - @BeforeEach - void setupKubernetesClient() { - k8sClient = spy( new K8sClientForTest()) - k8sClient.client = client - k8sClient.SLEEPTIME = 1 - k8sClient.DEFAULT_RETRIES = 1 - - // no need to wait in tests, we stub! - doNothing().when(k8sClient).waitForResourcePhase( - any(String), - any(String), - any(String), - any(String) - ) - } - - @Test - void 'Installs argoCD'() { - // Simulate argocd Namespace does not exist - - def argocd = createArgoCD() - argocd.install() - this.clusterResourcesRepo = (argocd as ArgoCDForTest).clusterResourcesRepo - - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - assertThat(client.namespaces().withName('argocd').get()).isNotNull() - - // check values.yaml - List filesWithInternalSCMM = findFilesContaining(new File(clusterResourcesRepoLayout.rootDir()), - clusterResourcesRepo.gitProvider.url) - assertThat(filesWithInternalSCMM).isNotEmpty() - assertThat(parseActualYaml(actualHelmValuesFile)['argo-cd']['server']['service']['type']) - .isEqualTo('ClusterIP') - assertThat(parseActualYaml(actualHelmValuesFile)['argo-cd']['notifications']['argocdUrl']).isNull() - - assertThat(parseActualYaml(actualHelmValuesFile)['argo-cd']['crds']).isNull() - assertThat(parseActualYaml(actualHelmValuesFile)['global']).isNull() - - Secret repoCredentialsSecret = client.secrets() - .inNamespace('argocd') - .withName('argocd-repo-creds-scm') - .get() - - assertThat(repoCredentialsSecret).isNotNull() - assertThat(repoCredentialsSecret.metadata.labels['argocd.argoproj.io/secret-type']).isEqualTo('repo-creds') - - // Check dependency build and helm install (Chart liegt jetzt unter apps/argocd/argocd) - assertThat(helmCommands.actualCommands[0].trim()) - .isEqualTo('helm repo add argo https://argoproj.github.io/argo-helm') - assertThat(helmCommands.actualCommands[1].trim()) - .isEqualTo("helm dependency build ${clusterResourcesRepoLayout.helmDir()}".toString()) - assertThat(helmCommands.actualCommands[2].trim()) - .isEqualTo("helm upgrade -i argocd ${clusterResourcesRepoLayout.helmDir()} --create-namespace --namespace argocd".toString()) - - Secret argocdSecret = client.secrets() - .inNamespace('argocd') - .withName('argocd-secret') - .get() - - assertThat(argocdSecret).isNotNull() - - String patchedPasswordHash = decodedSecretValue(argocdSecret, 'admin.password') - - assertThat(BCrypt.checkpw(config.application.password as String, patchedPasswordHash)) - .as("Password hash mismatch") - .isTrue() - - assertThat(client.secrets() - .inNamespace('argocd') - .withLabels([owner: 'helm', name: 'argocd']) - .list() - .items).isEmpty() - - // Operator disabled -> operator Ordner sollte fehlen - assertThat(Path.of(clusterResourcesRepoLayout.operatorConfigFile()).toFile()).doesNotExist() - assertThat(Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile()).doesNotExist() - - // Projects (jetzt unter argocd/projects) - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of(clusterResourcesRepoLayout.projectsDir(), 'cluster-resources.yaml')) - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('https://prometheus-community.github.io/helm-charts') - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('http://scmm-scm-manager.default.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack') - - // Applications (jetzt unter argocd/applications) - def argocdYaml = new YamlSlurper().parse(Path.of(clusterResourcesRepoLayout.applicationsDir(), 'argocd.yaml')) - assertThat(argocdYaml['spec']['source']['directory']).isNull() - - // Neuer Pfad: Chart liegt unter argocd/argocd (nicht mehr nur argocd/) - assertThat(argocdYaml['spec']['source']['path'] as String) - .isIn('apps/argocd/argocd', 'apps/argocd/argocd/') - } - - @Test - void 'Installs Argo CD with custom values'() { - config.features.argocd.values = ['argo-cd': [key: 'value']] - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - def valuesYaml = parseActualYaml(actualHelmValuesFile) - assertThat(valuesYaml['argo-cd']['key']).isEqualTo('value') - } - - @Test - void 'When monitoring disabled: Does not push path monitoring to cluster resources'() { - config.features.monitoring.active = false - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - assertThat(new File(clusterResourcesRepoLayout.monitoringDir())).doesNotExist() - } - - @Test - void 'When monitoring enabled: Does push path monitoring to cluster resources'() { - config.features.monitoring.active = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - assertThat(new File(clusterResourcesRepoLayout.monitoringDir())).exists() - assertValidDashboards(clusterResourcesRepoLayout.monitoringDir()) - } - - void assertValidDashboards(String monitoringPath) { - Files.walk(Path.of(monitoringPath)) - .filter { it.toString() ==~ /.*-dashboard\.yaml/ }.each { Path path -> - def dashboardConfigMap = null - - assertThatCode { - dashboardConfigMap = parseActualYaml(path.toString()) - }.as("Invalid YAML in ${path.fileName}").doesNotThrowAnyException() - - assertThat(dashboardConfigMap.data as Map).hasSize(1) - .as('Expected only on dashboard json within map') - assertThatCode { - def dashboardJsonString = (dashboardConfigMap.data as Map).entrySet().first().value as String - new JsonSlurper().parseText(dashboardJsonString) - }.as("Invalid JSON in ${path.fileName}").doesNotThrowAnyException() - } - } - - @Test - void 'When mailServer disabled: Does not include mail configurations into cluster resources'() { - config.features.mail.active = false - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - def valuesYaml = parseActualYaml(actualHelmValuesFile) - assertThat(valuesYaml['argo-cd']['notifications']['enabled']).isEqualTo(false) - assertThat(valuesYaml['argo-cd']['notifications']['notifiers']).isNull() - } - - @Test - void 'When mailServer enabled: Includes mail configurations into cluster resources'() { - config.features.mail.active = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - def valuesYaml = parseActualYaml(actualHelmValuesFile) - - assertThat(valuesYaml['argo-cd']['notifications']['enabled']).isEqualTo(true) - assertThat(valuesYaml['argo-cd']['notifications']['notifiers']).isNotNull() - } - - @Test - void 'When emailaddress is set: Include given email addresses into configurations'() { - config.features.mail.active = true - config.features.argocd.emailFrom = 'argocd@example.com' - config.features.argocd.emailToUser = 'app-team@example.com' - config.features.argocd.emailToAdmin = 'argocd@example.com' - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - def valuesYaml = parseActualYaml(actualHelmValuesFile) - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml") - def argocdYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.applicationsDir(), 'argocd.yaml') - def defaultYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), 'default.yaml') - - assertThat(new YamlSlurper().parseText(valuesYaml['argo-cd']['notifications']['notifiers']['service.email'] as String)['from']).isEqualTo("argocd@example.com") - assertThat(clusterRessourcesYaml['metadata']['annotations']['notifications.argoproj.io/subscribe.email']).isEqualTo('argocd@example.com') - assertThat(argocdYaml['metadata']['annotations']['notifications.argoproj.io/subscribe.on-sync-status-unknown.email']).isEqualTo('argocd@example.com') - assertThat(defaultYaml['metadata']['annotations']['notifications.argoproj.io/subscribe.email']).isEqualTo('argocd@example.com') - } - - @Test - void 'When emailaddress is NOT set: Use default email addresses in configurations'() { - config.features.mail.active = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - def valuesYaml = parseActualYaml(actualHelmValuesFile) - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml") - def argocdYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.applicationsDir(), 'argocd.yaml') - def defaultYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), 'default.yaml') - - assertThat(new YamlSlurper().parseText(valuesYaml['argo-cd']['notifications']['notifiers']['service.email'] as String)['from']).isEqualTo("argocd@example.org") - assertThat(clusterRessourcesYaml['metadata']['annotations']['notifications.argoproj.io/subscribe.email']).isEqualTo('infra@example.org') - assertThat(argocdYaml['metadata']['annotations']['notifications.argoproj.io/subscribe.on-sync-status-unknown.email']).isEqualTo('infra@example.org') - assertThat(defaultYaml['metadata']['annotations']['notifications.argoproj.io/subscribe.email']).isEqualTo('infra@example.org') - } - - @Test - void 'When external Mailserver is set'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpPort = 1010110 - config.features.mail.smtpUser = 'argo@example.com' - config.features.mail.smtpPassword = '1101:ABCabc&/+*~' - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - def serviceEmail = new YamlSlurper().parseText(parseActualYaml(actualHelmValuesFile)['argo-cd']['notifications']['notifiers']['service.email'] as String) - - assertThat(serviceEmail['host']).isEqualTo(config.features.mail.smtpAddress) - assertThat(serviceEmail['port']).isEqualTo(config.features.mail.smtpPort) - // username and password are both linked to the k8s secret. Secrets will be created at runtime, in this test - assertThat(serviceEmail['username']).isEqualTo('$email-username') - assertThat(serviceEmail['password']).isEqualTo('$email-password') - - Secret mailSecret = client.secrets() - .inNamespace('argocd') - .withName('argocd-notifications-secret') - .get() - - assertThat(mailSecret).isNotNull() - assertThat(decodedSecretValue(mailSecret, 'email-username')).isEqualTo(config.features.mail.smtpUser) - assertThat(decodedSecretValue(mailSecret, 'email-password')).isEqualTo(config.features.mail.smtpPassword) - } - - @Test - void 'When external emailservers username is set, check if kubernetes secret will be created'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpUser = 'argo@example.com' - - createArgoCD().install() - - Secret mailSecret = client.secrets() - .inNamespace('argocd') - .withName('argocd-notifications-secret') - .get() - - assertThat(mailSecret).isNotNull() - assertThat(decodedSecretValue(mailSecret, 'email-username')).isEqualTo(config.features.mail.smtpUser) - } - - @Test - void 'When external emailservers password is set, check if kubernetes secret will be created'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - config.features.mail.smtpPassword = '1101:ABCabc&/+*~' - - createArgoCD().install() - - Secret mailSecret = client.secrets() - .inNamespace('argocd') - .withName('argocd-notifications-secret') - .get() - - assertThat(mailSecret).isNotNull() - assertThat(decodedSecretValue(mailSecret, 'email-password')).isEqualTo(config.features.mail.smtpPassword) - } - - @Test - void 'When external Mailserver is set without port, user, password'() { - config.features.mail.active = true - config.features.mail.smtpAddress = 'smtp.example.com' - - def argocd = createArgoCD() - argocd.install() - - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - def serviceEmail = new YamlSlurper().parseText(parseActualYaml(actualHelmValuesFile)['argo-cd']['notifications']['notifiers']['service.email'] as String) - - assertThat(client.secrets().inNamespace('argocd').withName('argocd-notifications-secret').get()).isNull() - - assertThat(serviceEmail['host']).isEqualTo("smtp.example.com") - assertThat(serviceEmail as Map).doesNotContainKey('port') - assertThat(serviceEmail as Map).doesNotContainKey('username') - assertThat(serviceEmail as Map).doesNotContainKey('password') - } - - @Test - void 'When external Mailserver is NOT set'() { - config.features.mail.active = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - def valuesYaml = parseActualYaml(actualHelmValuesFile) - - assertThat(new YamlSlurper().parseText(valuesYaml['argo-cd']['notifications']['notifiers']['service.email'] as String)['port']).isEqualTo(1025) - assertThat(new YamlSlurper().parseText(valuesYaml['argo-cd']['notifications']['notifiers']['service.email'] as String)) doesNotHaveToString('username') - assertThat(new YamlSlurper().parseText(valuesYaml['argo-cd']['notifications']['notifiers']['service.email'] as String)) doesNotHaveToString('password') - } - - @Test - void 'When vault disabled: Does not push path "secrets" to cluster resources'() { - config.features.secrets.active = false - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - assertThat(new File(clusterResourcesRepoLayout.vaultDir())).doesNotExist() - } - - @Test - void 'Prepares repos for air-gapped mode'() { - config.features.monitoring.active = false - config.application.mirrorRepos = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml") - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack') - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('https://prometheus-community.github.io/helm-charts') - } - - @Test - void 'Pushes repos with empty name-prefix'() { - def argocd = createArgoCD() - argocd.install() - this.clusterResourcesRepo = (argocd as ArgoCDForTest).clusterResourcesRepo - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - assertArgoCdYamlPrefixes(clusterResourcesRepo.gitProvider.url, '', clusterResourcesRepoLayout) - } - - @Test - void 'Creates Jenkinsfiles for two registries'() { - config.registry.twoRegistries = true - createArgoCD().install() - - assertJenkinsfileRegistryCredentials() - } - - @Test - void 'Pushes repos with name-prefix'() { - config.application.namePrefix = 'abc-' - - def argocd = createArgoCD() - argocd.install() - this.clusterResourcesRepo = (argocd as ArgoCDForTest).clusterResourcesRepo - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - assertArgoCdYamlPrefixes(clusterResourcesRepo.gitProvider.url, config.application.namePrefix, clusterResourcesRepoLayout) - } - - @Test - void 'SecurityContext null in Openshift'() { - config.application.openshift = true - createArgoCD().install() - - for (def petclinicRepo : petClinicRepos) { - if (petclinicRepo.repoTarget.contains('argocd/petclinic-plain')) { - assertThat(new File(petclinicRepo.absoluteLocalRepoTmpDir, '/k8s/staging/deployment.yaml').text).contains('runAsUser: null') - assertThat(new File(petclinicRepo.absoluteLocalRepoTmpDir, '/k8s/staging/deployment.yaml').text).contains('runAsGroup: null') - } - if (petclinicRepo.repoTarget.contains('argocd/petclinic-helm')) { - assertThat(new File(petclinicRepo.absoluteLocalRepoTmpDir, '/k8s/values-shared.yaml').text).contains('runAsUser: null') - assertThat(new File(petclinicRepo.absoluteLocalRepoTmpDir, '/k8s/values-shared.yaml').text).contains('runAsGroup: null') - } - } - } - - @Test - void 'Skips CRDs for argo cd'() { - config.application.skipCrds = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - assertThat(parseActualYaml(actualHelmValuesFile)['argo-cd']['crds']['install']).isEqualTo(false) - } - - @Test - void 'Write maven mirror into jenkinsfiles'() { - config.jenkins.mavenCentralMirror = 'http://test' - createArgoCD().install() - - for (def petclinicRepo : petClinicRepos) { - assertThat(new File(petclinicRepo.absoluteLocalRepoTmpDir, 'Jenkinsfile').text).contains('mvn.useMirrors([name: \'maven-central-mirror\', mirrorOf: \'central\', url: env.MAVEN_CENTRAL_MIRROR])') - } - } - - @Test - void 'ArgoCD with active network policies'() { - config.application.netpols = true - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - this.actualHelmValuesFile = "${clusterResourcesRepoLayout.helmDir()}/values.yaml" - - assertThat(parseActualYaml(actualHelmValuesFile)['argo-cd']['global']['networkPolicy']['create']).isEqualTo(true) - assertThat(new File(clusterResourcesRepoLayout.argocdRoot(), '/argocd/values.yaml').text.contains("namespace: monitoring")) - assertThat(new File(clusterResourcesRepoLayout.argocdRoot(), '/argocd/templates/allow-namespaces.yaml').text.contains("namespace: monitoring")) - assertThat(new File(clusterResourcesRepoLayout.argocdRoot(), '/argocd/templates/allow-namespaces.yaml').text.contains("namespace: default")) - } - - private void assertArgoCdYamlPrefixes(String scmmUrl, String expectedPrefix, RepoLayout repoLayout) { - - assertAllYamlFiles(new File(repoLayout.argocdRoot()), 'projects', 3) { Path file -> - def yaml = parseActualYaml(file.toString()) - List sourceRepos = yaml['spec']['sourceRepos'] as List - // Some projects might not have sourceRepos - if (sourceRepos) { - sourceRepos.each { - if (it.startsWith(scmmUrl)) { - assertThat(it) - .as("$file sourceRepos have name prefix") - .startsWith("${scmmUrl}/repo/${expectedPrefix}argocd") - } - } - } - - String metadataNamespace = yaml['metadata']['namespace'] as String - if (metadataNamespace) { - assertThat(metadataNamespace) - .as("$file metadata.namespace has name prefix") - .isEqualTo("${expectedPrefix}argocd".toString()) - } - - List sourceNamespaces = yaml['spec']['sourceNamespaces'] as List - if (sourceNamespaces) { - sourceNamespaces.each { - if (it != '*') { - assertThat(it) - .as("$file spec.sourceNamespace has name prefix") - .startsWith("${expectedPrefix}") - } - } - } - } - - assertAllYamlFiles(new File(repoLayout.argocdRoot()), 'applications', 3) { Path file -> - def yaml = parseActualYaml(file.toString()) - assertThat(yaml['spec']['source']['repoURL'] as String) - .as("$file repoURL have name prefix") - .startsWith("${scmmUrl}/repo/${expectedPrefix}argocd") - - assertThat(yaml['metadata']['namespace']) - .as("$file metadata.namspace has name prefix") - .isEqualTo("${expectedPrefix}argocd".toString()) - - assertThat(yaml['spec']['destination']['namespace']) - .as("$file spec.destination.namspace has name prefix") - .isEqualTo("${expectedPrefix}argocd".toString()) - } - - //checks all other folder for prefixed yaml files except "apps/argocd" - assertAllYamlFiles(new File(repoLayout.rootDir()), 'apps', 9, - ['/apps/argocd/']) { Path it -> - - def yaml = parseActualYaml(it.toString()) - List yamlDocuments = yaml instanceof List ? yaml : [yaml] - for (def document in yamlDocuments) { - if (document && document['kind'] != 'Namespace') { - def metadataNamespace = document['metadata']['namespace'] as String - assertThat(metadataNamespace) - .as("$it metadata.namespace has name prefix") - .startsWith("${expectedPrefix}") - } - } - } - } - - private static void assertAllYamlFiles(File rootDir, - String childDir, - Integer numberOfFiles, - List excludeContains = [], - Closure cl) { - def rootPath = Path.of(rootDir.absolutePath, childDir) - - def yamlFiles = Files.walk(rootPath) - .filter { Files.isRegularFile(it) } - .filter { Path p -> - def s = p.toString().replace('\\', '/') - (s.endsWith('.yaml') || s.endsWith('.yml')) && !excludeContains.any { ex -> s.contains(ex) } - } - .collect(Collectors.toList()) - - yamlFiles.each(cl) - - assertThat(yamlFiles.size()).isEqualTo(numberOfFiles) - } - - private static List findFilesContaining(File folder, String stringToSearch) { - List result = [] - folder.eachFileRecurse(FileType.FILES) { - if (it.text.contains(stringToSearch)) { - result += it - } - } - return result - } - - ArgoCD createArgoCD() { - prepareKubernetesObjectsForArgoCd() - def argoCD = ArgoCDForTest.newWithAutoProviders(config, k8sClient, helmCommands) - return argoCD - } - private void prepareKubernetesObjectsForArgoCd() { - String namespace = "${config.application.namePrefix ?: ''}${config.features.argocd.namespace ?: 'argocd'}" - - createNamespaceIfMissing(namespace) - createNamespaceIfMissing(config.multiTenant.centralArgocdNamespace ?: 'argocd') - - createArgoCdCrds() - - config.application.namespaces.getActiveNamespaces().each { String activeNamespace -> - createNamespaceIfMissing(activeNamespace) - } - - createSecretIfMissing('argocd-secret', namespace) - createSecretIfMissing('argocd-cluster', namespace) - createSecretIfMissing('argocd-default-cluster-config', namespace, - [namespaces: Base64.encoder.encodeToString('testnamespace1,testnamespace2'.bytes)]) - - if (config.multiTenant.useDedicatedInstance) { - createSecretIfMissing('argocd-default-cluster-config', config.multiTenant.centralArgocdNamespace ?: 'argocd', - [namespaces: Base64.encoder.encodeToString('testnamespace1,testnamespace2'.bytes)]) - } - } - - private void createArgoCdCrds() { - createNamespacedCrd('appprojects.argoproj.io', 'argoproj.io', 'v1alpha1', 'AppProject', 'appprojects', 'appproject') - createNamespacedCrd('applications.argoproj.io', 'argoproj.io', 'v1alpha1', 'Application', 'applications', 'application') - createNamespacedCrd('argocds.argoproj.io', 'argoproj.io', 'v1beta1', 'ArgoCD', 'argocds', 'argocd') - } - - private void createNamespacedCrd(String name, - String group, - String version, - String kind, - String plural, - String singular) { - if (client.apiextensions().v1().customResourceDefinitions().withName(name).get()) { - return - } - - CustomResourceDefinition crd = new CustomResourceDefinitionBuilder() - .withNewMetadata() - .withName(name) - .endMetadata() - .withNewSpec() - .withGroup(group) - .withScope('Namespaced') - .withNewNames() - .withKind(kind) - .withPlural(plural) - .withSingular(singular) - .endNames() - .addNewVersion() - .withName(version) - .withServed(true) - .withStorage(true) - .withNewSchema() - .withNewOpenAPIV3Schema() - .withType('object') - .withXKubernetesPreserveUnknownFields(true) - .endOpenAPIV3Schema() - .endSchema() - .endVersion() - .endSpec() - .build() - - client.apiextensions() - .v1() - .customResourceDefinitions() - .resource(crd) - .create() - } - - private void createNamespaceIfMissing(String name) { - if (!name) { - throw new IllegalArgumentException() - } - - if (!client.namespaces().withName(name).get()) { - client.namespaces().resource(new NamespaceBuilder() - .withNewMetadata() - .withName(name) - .endMetadata() - .build()) - .create() - } - } - - private String decodedSecretValue(Secret secret, String key) { - if (secret.stringData?.containsKey(key)) { - return secret.stringData[key] - } - - if (secret.data?.containsKey(key)) { - return new String(Base64.decoder.decode(secret.data[key])) - } - - return null - } - - private void createSecretIfMissing(String name, String namespace, Map data = [:]) { - if (!namespace) { - throw new IllegalArgumentException() - } - - createNamespaceIfMissing(namespace) - - if (!client.secrets().inNamespace(namespace).withName(name).get()) { - Secret secret = new SecretBuilder() - .withNewMetadata() - .withName(name) - .withNamespace(namespace) - .endMetadata() - .withType('Opaque') - .withData(data) - .build() - - client.secrets() - .inNamespace(namespace) - .resource(secret) - .create() - } - } - - void assertJenkinsfileRegistryCredentials() { - List defaultRegistryExpectedLines = ['String pathPrefix = !dockerRegistryPath?.trim() ? "" : "${dockerRegistryPath}/"', - 'imageName = "${dockerRegistryBaseUrl}/${pathPrefix}${application}:${imageTag}"'] - List twoRegistriesExpectedLines = ['String proxyPathPrefix = !dockerRegistryProxyPath?.trim() ? "" : "${dockerRegistryProxyPath}/"', - 'docker.withRegistry("https://${dockerRegistryProxyBaseUrl}/${proxyPathPrefix}", dockerRegistryProxyCredentials) {',] - - for (def petclinicRepo : petClinicRepos) { - String jenkinsfile = new File(petclinicRepo.absoluteLocalRepoTmpDir, 'Jenkinsfile').text - - defaultRegistryExpectedLines.each { expectedEnvVar -> assertThat(jenkinsfile).contains(expectedEnvVar) - } - - if (config.registry['twoRegistries']) { - twoRegistriesExpectedLines.each { expectedEnvVar -> assertThat(jenkinsfile).contains(expectedEnvVar) - } - } else { - twoRegistriesExpectedLines.each { expectedEnvVar -> assertThat(jenkinsfile).doesNotContain(expectedEnvVar) - } - } - } - } - - @Test - void 'Prepares ArgoCD repo with Operator configuration file'() { - def argocd = setupOperatorTest() - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def rbacConfigPath = Path.of(clusterResourcesRepoLayout.operatorRbacDir()) - - assertThat(argocdConfigPath.toFile()).exists() - assertThat(rbacConfigPath.toFile()).exists() - - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - assertThat(yaml['apiVersion']).isEqualTo('argoproj.io/v1beta1') - assertThat(yaml['kind']).isEqualTo('ArgoCD') - } - - @Test - void 'No files for operator when operator is false'() { - def argocd = createArgoCD() - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def rbacConfigPath = Path.of(clusterResourcesRepoLayout.operatorRbacDir()) - - assertThat(argocdConfigPath.toFile()).doesNotExist() - assertThat(rbacConfigPath.toFile()).doesNotExist() - } - - @Test - void 'Deploys with operator without OpenShift configuration'() { - def argocd = setupOperatorTest(openshift: false) - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - - assertThat(argocdConfigPath.toFile()).exists() - - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - assertThat(yaml['spec']['rbac']).isNull() - assertThat(yaml['spec']['sso']).isNull() - - def argocdYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.applicationsDir(), 'argocd.yaml') - assertThat(argocdYaml['spec']['source']['directory']['recurse'] as Boolean).isTrue() - assertThat(argocdYaml['spec']['source']['path']).isEqualTo('apps/argocd/operator/') - // Here we should assert all <#if argocd.isOperator> in YAML ️ - } - - @Test - void 'RBACs with operator using RbacDefinition outputs'() { - config.application.namePrefix = "testPrefix-" - - LinkedHashSet expectedNamespaces = ["testPrefix-monitoring", - "testPrefix-secrets", - "testPrefix-traefik", - "testPrefix-example-apps-staging", - "testPrefix-example-apps-production"] - // have to prepare activeNamespaces for unit-test, Application.groovy is setting this in integration way - config.application.namespaces.dedicatedNamespaces = new LinkedHashSet(["monitoring", - "secrets", - "traefik", - "example-apps-staging", - "example-apps-production"]) - - def argocd = setupOperatorTest(openshift: false) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - File rbacPath = Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile() - - expectedNamespaces.each { String ns -> - File roleFile = new File(rbacPath, "role-argocd-${ns}.yaml") - File bindingFile = new File(rbacPath, "rolebinding-argocd-${ns}.yaml") - - assertThat(roleFile).exists() - assertThat(bindingFile).exists() - - Map roleYaml = new YamlSlurper().parse(roleFile) as Map - Map bindingYaml = new YamlSlurper().parse(bindingFile) as Map - - assertThat(roleYaml["kind"]).isEqualTo("Role") - assertThat(roleYaml["metadata"]["name"]).isEqualTo("argocd") - assertThat(roleYaml["metadata"]["namespace"]).isEqualTo(ns) - - assertThat(bindingYaml["kind"]).isEqualTo("RoleBinding") - assertThat(bindingYaml["metadata"]["name"]).isEqualTo("argocd") - assertThat(bindingYaml["metadata"]["namespace"]).isEqualTo(ns) - - List> subjects = bindingYaml["subjects"] as List> - assertThat(subjects).isNotEmpty() - assertThat(subjects*.kind).containsOnly("ServiceAccount") - assertThat(subjects*.namespace).containsOnly("testPrefix-argocd") - assertThat(subjects*.name).containsExactlyInAnyOrder("argocd-argocd-server", - "argocd-argocd-application-controller", - "argocd-applicationset-controller") - - Map roleRef = bindingYaml["roleRef"] as Map - assertThat(roleRef).isNotNull() - assertThat(roleRef["name"]).isEqualTo("argocd") - assertThat(roleRef["kind"]).isEqualTo("Role") - } - } - - @Test - void 'Deploys with operator with OpenShift configuration'() { - def argocd = setupOperatorTest(openshift: true) - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - assertThat(argocdConfigPath.toFile()).exists() - - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - assertThat(yaml['spec']['sso']).isNotNull() - assertThat(yaml['spec']['sso']['dex']['openShiftOAuth']).isEqualTo(true) - assertThat(yaml['spec']['sso']['provider']).isEqualTo('dex') - assertThat(yaml['spec']['rbac']).isNotNull() - assertThat(yaml['spec']['server']['route']['enabled']).isEqualTo(true) - } - - @Test - void 'check if external_secrets_io and monitoring_coreos_com is set'() { - - config.features.monitoring.active = true - config.features.secrets.active = true - - String expectedMonitoring = 'monitoring.coreos.com' - String expectedExternalSecret = 'external-secrets.io' - - def argocd = setupOperatorTest(openshift: true) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def yaml = parseActualYaml(Path.of(clusterResourcesRepoLayout.operatorConfigFile()).toString()) - - def resourceInclusionsString = yaml['spec']['resourceInclusions'] as String - - assertThat(resourceInclusionsString.contains(expectedMonitoring)).isTrue() - assertThat(resourceInclusionsString.contains(expectedExternalSecret)).isTrue() - } - - @Test - void 'check if external_secrets_io and monitoring_coreos_com is not set'() { - - config.features.monitoring.active = false - config.features.secrets.active = false - - String expectedMonitoring = 'monitoring.coreos.com' - String expectedExternalSecret = 'external-secrets.io' - - def argocd = setupOperatorTest(openshift: true) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def yaml = parseActualYaml(Path.of(clusterResourcesRepoLayout.operatorConfigFile()).toString()) - - def resourceInclusionsString = yaml['spec']['resourceInclusions'] as String - - assertThat(resourceInclusionsString.contains(expectedMonitoring)).isFalse() - assertThat(resourceInclusionsString.contains(expectedExternalSecret)).isFalse() - } - - @Test - void 'Correctly sets resourceInclusions from config'() { - def argocd = setupOperatorTest() - - // Set the config to a custom resourceInclusionsCluster value - config.features.argocd.resourceInclusionsCluster = 'https://192.168.0.1:6443' - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - - def expectedClusterUrl = 'https://192.168.0.1:6443' - - // Retrieve and parse the resourceInclusions string into structured YAML - def resourceInclusionsString = yaml['spec']['resourceInclusions'] as String - def parsedResourceInclusions = new YamlSlurper().parseText(resourceInclusionsString) - - // Iterate over the parsed resource inclusions and check the 'clusters' field - parsedResourceInclusions.each { resource -> - assertThat(resource as Map).containsKey('clusters') - assertThat(resource['clusters'] as List).contains(expectedClusterUrl) - } - } - - @Test - void 'resourceInclusionsCluster from config file trumps ENVs'() { - def argocd = setupOperatorTest() - - // Set the config to a custom internalKubernetesApiUrl value - config.application.internalKubernetesApiUrl = 'https://192.168.0.1:6443' - - // Set environment variables for Kubernetes API server - withEnvironmentVariable("KUBERNETES_SERVICE_HOST", "100.125.0.1") - .and("KUBERNETES_SERVICE_PORT", "443") - .execute { - argocd.install() - } - - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - def expectedClusterUrlFromConfig = "https://192.168.0.1:6443" - - // Retrieve and parse the resourceInclusions string into structured YAML - def resourceInclusionsString = yaml['spec']['resourceInclusions'] as String - def parsedResourceInclusions = new YamlSlurper().parseText(resourceInclusionsString) - - // Ensure that the clusters field uses the config value, not the env variables - parsedResourceInclusions.each { resource -> - assertThat(resource as Map).containsKey('clusters') - assertThat(resource['clusters'] as List).contains(expectedClusterUrlFromConfig) - // Make sure the environment variable value does not appear - assertThat(resource['clusters'] as List).doesNotContain("https://100.125.0.1:443") - } - } - - @Test - void 'Sets env variables in ArgoCD components when provided'() { - def argocd = setupOperatorTest() - - // Set environment variables for ArgoCD - config.features.argocd.env = [[name: "ENV_VAR_1", value: "value1"], - [name: "ENV_VAR_2", value: "value2"]] as List - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - - def expectedEnv = [[name: "ENV_VAR_1", value: "value1"], - [name: "ENV_VAR_2", value: "value2"]] - - // Check that the env variables are added to the relevant components - assertThat(yaml['spec']['applicationSet']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['notifications']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['controller']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['repo']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['server']['env']).isEqualTo(expectedEnv) - } - - @Test - void 'Does not set env variables when none are provided'() { - def argocd = setupOperatorTest() - - // Ensure env is an empty list (default) - config.features.argocd.env = [] - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - - // Check that the env variables are not present - assertThat(yaml['spec']['applicationSet'] as Map).doesNotContainKey('env') - assertThat(yaml['spec']['notifications'] as Map).doesNotContainKey('env') - assertThat(yaml['spec']['controller'] as Map).doesNotContainKey('env') - assertThat(yaml['spec']['redis'] as Map).doesNotContainKey('env') - assertThat(yaml['spec']['repo'] as Map).doesNotContainKey('env') - assertThat(yaml['spec']['server'] as Map).doesNotContainKey('env') - } - - @Test - void 'Sets single env variable in ArgoCD components when provided'() { - def argocd = setupOperatorTest() - - // Set a single environment variable for ArgoCD - config.features.argocd.env = [[name: "ENV_VAR_SINGLE", value: "singleValue"]] as List - - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()) - def yaml = parseActualYaml(argocdConfigPath.toFile().toString()) - - def expectedEnv = [[name: "ENV_VAR_SINGLE", value: "singleValue"]] - - // Check that the single env variable is added to the relevant components - assertThat(yaml['spec']['applicationSet']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['notifications']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['controller']['env']).isEqualTo(expectedEnv) - assertThat(yaml['spec']['server']['env']).isEqualTo(expectedEnv) - } - - @Test - void 'Creates all necessary namespaces'() { - def argoCD = createArgoCD() - - argoCD.install() - - config.application.namespaces.getActiveNamespaces().each { namespace -> - assertThat(client.namespaces().withName(namespace).get()).isNotNull() - } - } - - @Test - void 'Operator config sets server insecure to true when insecure is set'() { - config.application.insecure = true - def argocd = setupOperatorTest() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def yaml = parseActualYaml(Path.of(clusterResourcesRepoLayout.operatorConfigFile()).toString()) - assertThat(yaml['spec']['server']['insecure']).isEqualTo(true) - } - - @Test - void 'Operator config sets custom values'() { - config.features.argocd.values = [key: 'value'] - config.features.argocd.values = [spec: [key: 'value']] - def argocd = setupOperatorTest() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def yaml = parseActualYaml(Path.of(clusterResourcesRepoLayout.operatorConfigFile()).toString()) - assertThat(yaml['spec']['key']).isEqualTo('value') - } - - @Test - void 'Operator config sets server_insecure to false when insecure is not set'() { - def argocd = setupOperatorTest() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def yaml = parseActualYaml(Path.of(clusterResourcesRepoLayout.operatorConfigFile()).toString()) - assertThat(yaml['spec']['server']['insecure']).isEqualTo(false) - } - - @Test - void 'Generates correct ingress yaml with expected host when insecure is true and not on OpenShift'() { - config.application.insecure = true - config.features.argocd.url = "http://argocd.localhost" - def argocd = setupOperatorTest(openshift: false) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml") - assertThat(ingressFile) - .as("Ingress file should be generated for insecure mode on non-OpenShift") - .exists() - - def ingressYaml = parseActualYaml(ingressFile.toString()) - - def rules = ingressYaml['spec']['rules'] as List - def host = rules[0]['host'] - assertThat(host) - .as("Ingress host should match configured ArgoCD hostname") - .isEqualTo(new URL(config.features.argocd.url).host) - } - - @Test - void 'Does not generate ingress yaml when insecure is false'() { - config.application.insecure = false - def argocd = setupOperatorTest(openshift: false) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml") - assertThat(ingressFile) - .as("Ingress file should not be generated when insecure is false") - .doesNotExist() - } - - @Test - void 'Does not generate ingress yaml when running on OpenShift'() { - config.application.insecure = true - def argocd = setupOperatorTest(openshift: true) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml") - assertThat(ingressFile) - .as("Ingress file should not be generated on OpenShift") - .doesNotExist() - } - - @Test - void 'Does not generate ingress yaml when insecure is false and OpenShift is true'() { - config.application.insecure = false - def argocd = setupOperatorTest(openshift: true) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml") - assertThat(ingressFile) - .as("Ingress file should not be generated when both flags are false") - .doesNotExist() - } - - @Test - void 'Central Bootstrapping for Tenant Applications'() { - setupDedicatedInstanceMode() - - assertThat(clusterResourcesRepoLayout).isNotNull() - - def ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml") - assertThat(ingressFile) - .as("Ingress file should not be generated when insecure is false") - .doesNotExist() - } - - @Test - void 'GOP DedicatedInstances Central templating works correctly'() { - setupDedicatedInstanceMode() - - assertThat(clusterResourcesRepoLayout).isNotNull() - - //Central Applications - assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/argocd.yaml")).exists() - assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/bootstrap.yaml")).exists() - assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/projects.yaml")).exists() - assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/example-apps.yaml")).doesNotExist() - - def argocdYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.argocdRoot(), "/applications/argocd.yaml") - def bootstrapYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.argocdRoot(), "/applications/bootstrap.yaml") - def projectsYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.argocdRoot(), "/applications/projects.yaml") - - assertThat(argocdYaml['metadata']['name']).isEqualTo('testPrefix-argocd') - assertThat(argocdYaml['metadata']['namespace']).isEqualTo('argocd') - assertThat(argocdYaml['spec']['project']).isEqualTo('testPrefix') - assertThat(argocdYaml['spec']['source']['path']).isEqualTo('apps/argocd/operator/') - - assertThat(bootstrapYaml['metadata']['name']).isEqualTo('testPrefix-bootstrap') - assertThat(bootstrapYaml['metadata']['namespace']).isEqualTo('argocd') - assertThat(bootstrapYaml['spec']['project']).isEqualTo('testPrefix') - assertThat(bootstrapYaml['spec']['source']['repoURL']).isEqualTo("scmm.testhost/scm/repo/testPrefix-argocd/cluster-resources.git") - - assertThat(projectsYaml['metadata']['name']).isEqualTo('testPrefix-projects') - assertThat(projectsYaml['metadata']['namespace']).isEqualTo('argocd') - assertThat(projectsYaml['spec']['project']).isEqualTo('testPrefix') - - //Central Project - assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/projects/tenant.yaml")).exists() - - def tenantProject = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.argocdRoot(), "/projects/tenant.yaml") - - assertThat(tenantProject['metadata']['name']).isEqualTo('testPrefix') - assertThat(tenantProject['metadata']['namespace']).isEqualTo('argocd') - def sourceRepos = (List) tenantProject['spec']['sourceRepos'] - assertThat(sourceRepos[0]).isEqualTo('scmm.testhost/scm/repo/testPrefix-argocd/cluster-resources.git') - } - - - @Test - void 'Append namespaces to Argocd argocd-default-cluster-config secrets'() { - config.application.namespaces.dedicatedNamespaces = new LinkedHashSet(['dedi-test1', 'dedi-test2', 'dedi-test3']) - config.application.namespaces.tenantNamespaces = new LinkedHashSet(['tenant-test1', 'tenant-test2', 'tenant-test3']) - - setupDedicatedInstanceMode() - - Secret defaultClusterConfig = client.secrets() - .inNamespace('argocd') - .withName('argocd-default-cluster-config') - .get() - - assertThat(defaultClusterConfig).isNotNull() - - String namespaces = decodedSecretValue(defaultClusterConfig, 'namespaces') - assertThat(namespaces).contains('testnamespace1') - assertThat(namespaces).contains('testnamespace2') - assertThat(namespaces).contains('testPrefix-dedi-test1') - assertThat(namespaces).contains('testPrefix-dedi-test2') - assertThat(namespaces).contains('testPrefix-dedi-test3') - assertThat(namespaces).contains('testPrefix-tenant-test1') - assertThat(namespaces).contains('testPrefix-tenant-test2') - assertThat(namespaces).contains('testPrefix-tenant-test3') - } - - @Test - void 'multiTenant folder gets deleted correctly if not in dedicated mode'() { - config.multiTenant.useDedicatedInstance = false - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), 'multiTenant/')).doesNotExist() - assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), 'applications/')).exists() - assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), 'projects/')).exists() - } - - @Test - void 'deleting unused folder in dedicated mode'() { - setupDedicatedInstanceMode() - - assertThat(clusterResourcesRepoLayout).isNotNull() - assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), 'multiTenant/')).doesNotExist() - assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), 'applications/')).exists() - assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), 'projects/')).exists() - } - - @Test - void 'RBACs generated correctly'() { - config.application.namespaces.tenantNamespaces = new LinkedHashSet(['testprefix-tenant-test1', 'testprefix-tenant-test2', 'testprefix-tenant-test3']) - setupDedicatedInstanceMode() - - File rbacFolder = new File(clusterResourcesRepoLayout.operatorRbacDir()) - File rbacTenantFolder = new File(clusterResourcesRepoLayout.operatorRbacDir() + "/tenant") - assertThat(rbacFolder).exists() - assertThat(rbacTenantFolder).exists() - - assertThat(rbacFolder.listFiles().count { it.isFile() }).isEqualTo(14) - assertThat(rbacTenantFolder.listFiles().count { it.isFile() }).isEqualTo(6) - - rbacFolder.eachFile { file -> - if (file.name.startsWith("role-") && file.name.contains('dedi')) { - def rbacFile = new YamlSlurper().parse(Path.of file.path) - assertThat(rbacFile['metadata']['namespace']).isIn(config.application.namespaces.getActiveNamespaces()) - } - if (file.name.startsWith("rolebinding-") && file.name.contains('dedi')) { - def rbacFile = new YamlSlurper().parse(Path.of file.path) - assertThat(rbacFile['subjects']['namespace']).isEqualTo(["argocd", "argocd", "argocd"]) - } - } - - rbacTenantFolder.eachFile { file -> - if (file.name.startsWith("role-")) { - def rbacFile = new YamlSlurper().parse(Path.of file.path) - assertThat(rbacFile['metadata']['namespace']).isIn(config.application.namespaces.tenantNamespaces) - } - - if (file.name.startsWith("rolebinding-")) { - def rbacFile = new YamlSlurper().parse(Path.of file.path) - assertThat(rbacFile['subjects']['namespace']).isEqualTo(["testPrefix-argocd", "testPrefix-argocd", "testPrefix-argocd"]) - } - } - - } - - @Test - void 'Operator RBAC includes node access rules when not on OpenShift'() { - config.application.namePrefix = "testprefix-" - - def argocd = setupOperatorTest(openshift: false) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - print config.toMap() - - File rbacDir = Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile() - File roleFile = new File(rbacDir, "role-argocd-testprefix-monitoring.yaml") - - Map yaml = new YamlSlurper().parse(roleFile) as Map - List> rules = yaml["rules"] as List> - - assertThat(rules).anyMatch { rule -> - List resources = rule["resources"] as List - resources.contains("nodes") && resources.contains("nodes/metrics") - } - } - - @Test - void 'Operator RBAC does not include node access rules when on OpenShift'() { - config.application.namePrefix = "testprefix-" - - def argocd = setupOperatorTest(openshift: true) - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - File rbacDir = Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile() - File roleFile = new File(rbacDir, "role-argocd-testprefix-monitoring.yaml") - println roleFile - - Map yaml = new YamlSlurper().parse(roleFile) as Map - List> rules = yaml["rules"] as List> - - assertThat(rules).noneMatch { rule -> - List resources = rule["resources"] as List - resources.contains("nodes") && resources.contains("nodes/metrics") - } - } - - @Test - void 'If not using mirror, ensure source repos in cluster-resources got right URL'() { - config.application.mirrorRepos = false - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), '/cluster-resources.yaml') - clusterRessourcesYaml['spec']['sourceRepos'] - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('https://charts.external-secrets.io', - 'https://codecentric.github.io/helm-charts', - 'https://prometheus-community.github.io/helm-charts', - 'https://traefik.github.io/charts', - 'https://helm.releases.hashicorp.com', - 'https://charts.jetstack.io') - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager') - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/kube-prometheus-stack.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/traefik.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/external-secrets.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/vault.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/cert-manager.git') - } - - @Test - void 'If using mirror, ensure source repos in cluster-resources got right URL'() { - config.application.mirrorRepos = true - - def argocd = createArgoCD() - argocd.install() - - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), '/cluster-resources.yaml') - clusterRessourcesYaml['spec']['sourceRepos'] - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack', - 'http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik', - 'http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets', - 'http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault', - 'http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager' - - ) - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/kube-prometheus-stack.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/traefik.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/external-secrets.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/vault.git', - 'http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/cert-manager.git') - } - - @Test - void 'If using mirror with GitLab, ensure source repos in cluster-resources got right URL'() { - config.application.mirrorRepos = true - config.scm.scmProviderType = 'GITLAB' - config.scm.gitlab.url = 'https://testGitLab.com/testgroup' - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), '/cluster-resources.yaml') - clusterRessourcesYaml['spec']['sourceRepos'] - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('https://testGitLab.com/testgroup/3rd-party-dependencies/kube-prometheus-stack.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/traefik.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/external-secrets.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/vault.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/cert-manager.git') - } - - @Test - void 'If using mirror with GitLab with prefix, ensure source repos in cluster-resources got right URL'() { - config.application.mirrorRepos = true - config.scm.scmProviderType = 'GITLAB' - config.scm.gitlab.url = "https://testGitLab.com/testgroup" - config.application.namePrefix = 'test1-' - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), '/cluster-resources.yaml') - clusterRessourcesYaml['spec']['sourceRepos'] - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('https://testGitLab.com/testgroup/3rd-party-dependencies/kube-prometheus-stack.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/traefik.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/external-secrets.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/vault.git', - 'https://testGitLab.com/testgroup/3rd-party-dependencies/cert-manager.git') - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager') - } - - @Test - void 'If using mirror with name-prefix, ensure source repos in cluster-resources got right URL'() { - config.application.mirrorRepos = true - config.application.namePrefix = 'test1-' - - def argocd = createArgoCD() - argocd.install() - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - def clusterRessourcesYaml = new YamlSlurper().parse(Path.of clusterResourcesRepoLayout.projectsDir(), '/cluster-resources.yaml') - clusterRessourcesYaml['spec']['sourceRepos'] - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).contains('http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager') - - assertThat(clusterRessourcesYaml['spec']['sourceRepos'] as List).doesNotContain('http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/kube-prometheus-stack.git', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/traefik.git', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/external-secrets.git', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/vault.git', - 'http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/cert-manager.git') - } - - void setupDedicatedInstanceMode() { - config.application.namePrefix = 'testPrefix-' - config.multiTenant.scmManager.url = 'scmm.testhost/scm' - config.multiTenant.scmManager.username = 'testUserName' - config.multiTenant.scmManager.password = 'testPassword' - config.multiTenant.useDedicatedInstance = true - this.argocd = setupOperatorTest() - - doReturn("Applied").when(k8sClient).applyYaml(any(String)) - - argocd.install() - this.clusterResourcesRepo = (argocd as ArgoCDForTest).clusterResourcesRepo - clusterResourcesRepoLayout = (argocd as ArgoCDForTest).getClusterRepoLayout() - - } - - protected ArgoCD setupOperatorTest(Map options = [:]) { - config.features.argocd.operator = true - config.features.argocd.resourceInclusionsCluster = 'https://192.168.0.1:6443' - config.application.openshift = options.openshift ?: false - - return createArgoCD() - } - - - private static void mockPrefixActiveNamespaces(Config config) { - def prefix = config.application.namePrefix ?: "" - - config.application.namespaces.with { - dedicatedNamespaces = new LinkedHashSet<>(dedicatedNamespaces.collect { (prefix + it).toString() }) - tenantNamespaces = new LinkedHashSet<>(tenantNamespaces.collect { (prefix + it).toString() }) - } - } - - static class ArgoCDForTest extends ArgoCD { - final Config cfg - final GitProvider tenantProvider - final GitProvider centralProvider - GitRepo clusterResourcesRepo - - static ArgoCDForTest newWithAutoProviders(Config cfg, - K8sClient k8sClient, - CommandExecutorForTest helmCommands) { - def provider = TestGitProvider.buildProviders(cfg) - return new ArgoCDForTest(cfg, - k8sClient, - helmCommands, - provider.tenant as GitProvider, - provider.central as GitProvider) - } - - ArgoCDForTest(Config cfg, - K8sClient k8sClient, - CommandExecutorForTest helmCommands, - GitProvider tenantProvider, - GitProvider centralProvider) { - super(cfg, - k8sClient, - new HelmClient(helmCommands), - new FileSystemUtils(), - new TestGitRepoFactory(cfg, new FileSystemUtils()), - new GitHandlerForTests(cfg, tenantProvider, centralProvider)) - this.cfg = cfg - this.tenantProvider = tenantProvider - this.centralProvider = centralProvider - mockPrefixActiveNamespaces(cfg) - } - - GitRepo getClusterResourcesRepo() { - return getRepoSetup().clusterResources?.repo - } - - RepoLayout getClusterRepoLayout() { - return getRepoSetup().clusterRepoLayout() - } - - } - - private Map parseActualYaml(String pathToYamlFile) { - File yamlFile = new File(pathToYamlFile) - def ys = new YamlSlurper() - return ys.parse(yamlFile) as Map - } - -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/AirGappedUtilsTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/AirGappedUtilsTest.groovy deleted file mode 100644 index 41a6384c6..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/AirGappedUtilsTest.groovy +++ /dev/null @@ -1,184 +0,0 @@ -package com.cloudogu.gitops.utils - -import com.cloudogu.gitops.application.orchestration.GitHandler -import com.cloudogu.gitops.config.Config -import com.cloudogu.gitops.infrastructure.git.GitRepo -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.Permission -import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository -import com.cloudogu.gitops.infrastructure.helm.HelmClient -import com.cloudogu.gitops.testhelper.git.GitHandlerForTests -import com.cloudogu.gitops.testhelper.git.ScmManagerMock -import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory -import com.cloudogu.gitops.testhelper.git.TestScmManagerApiClient -import groovy.yaml.YamlSlurper -import org.eclipse.jgit.api.Git -import org.eclipse.jgit.lib.Ref -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test - -import java.nio.file.Files -import java.nio.file.Path - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.ArgumentMatchers.* -import static org.mockito.Mockito.* - -class AirGappedUtilsTest { - - Config config = Config.fromMap([application: [localHelmChartFolder: '', - gitName : 'Cloudogu', - gitEmail : 'hello@cloudogu.com'], - scm : [scmManager: [url: '']]]) - - Config.HelmConfig helmConfig = new Config.HelmConfig([chart : 'kube-prometheus-stack', - repoURL: 'https://kube-prometheus-stack-repo-url', - version: '58.2.1']) - - Path rootChartsFolder = Files.createTempDirectory(this.class.getSimpleName()) - TestGitRepoFactory gitRepoFactory = new TestGitRepoFactory(config, new FileSystemUtils()) - FileSystemUtils fileSystemUtils = new FileSystemUtils() - TestScmManagerApiClient scmmApiClient = new TestScmManagerApiClient(config) - HelmClient helmClient = mock(HelmClient) - GitHandler gitHandler = new GitHandlerForTests(config, new ScmManagerMock()) - - @BeforeEach - void setUp() { - def response = scmmApiClient.mockSuccessfulResponse(201) - when(scmmApiClient.repositoryApi.create(any(Repository), anyBoolean())).thenReturn(response) - when(scmmApiClient.repositoryApi.createPermission(anyString(), anyString(), any(Permission))).thenReturn(response) - - } - - @Test - void 'Prepares repos for air-gapped use'() { - setupForAirgappedUse() - - def actualRepoNamespaceAndName = createAirGappedUtils().mirrorHelmRepoToGit(helmConfig) - - assertThat(actualRepoNamespaceAndName).isEqualTo("${GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES}/kube-prometheus-stack".toString()) - assertAirGapped() - } - - @Test - void 'Fails when unable to resolve version of dependencies'() { - setupForAirgappedUse([:]) - def exception = shouldFail(RuntimeException) { - createAirGappedUtils().mirrorHelmRepoToGit(helmConfig) - } - - assertThat(exception.message).isEqualTo('Unable to determine proper version for dependency grafana (version: 7.3.*) ' + - 'from repo 3rd-party-dependencies/kube-prometheus-stack') - } - - @Test - void 'Also works for charts without dependencies'() { - setupForAirgappedUse(null, []) - createAirGappedUtils().mirrorHelmRepoToGit(helmConfig) - - GitRepo prometheusRepo = gitRepoFactory.repos['3rd-party-dependencies/kube-prometheus-stack'] - def actualPrometheusChartYaml = new YamlSlurper().parse(Path.of(prometheusRepo.absoluteLocalRepoTmpDir, 'Chart.yaml')) - - def dependencies = actualPrometheusChartYaml['dependencies'] - assertThat(dependencies).isNull() - } - - @Test - void 'Fails for invalid helm charts'() { - setupForAirgappedUse() - - def expectedException = new RuntimeException() - doThrow(expectedException).when(helmClient).template(anyString(), anyString()) - - def exception = shouldFail(RuntimeException) { - createAirGappedUtils().mirrorHelmRepoToGit(helmConfig) - } - - assertThat(exception.getMessage()).isEqualTo("Helm chart in folder ${rootChartsFolder}/kube-prometheus-stack seems invalid.".toString()) - assertThat(exception.getCause()).isSameAs(expectedException) - } - - protected void setupForAirgappedUse(Map chartLock = null, List dependencies = null) { - Path sourceChart = rootChartsFolder.resolve('kube-prometheus-stack') - Files.createDirectories(sourceChart) - Map prometheusChartYaml = [version : '1.2.3', - name : 'kube-prometheus-stack-chart', - dependencies: [[condition : 'crds.enabled', - name : 'crds', - repository: '', - version : '0.0.0'], - [condition : 'grafana.enabled', - name : 'grafana', - repository: 'https://grafana-repo-url', - version : '7.3.*',]]] - - if (dependencies != null) { - if (dependencies.isEmpty()) { - prometheusChartYaml.remove('dependencies') - } else { - prometheusChartYaml['dependencies'] = dependencies - } - } - - fileSystemUtils.writeYaml(prometheusChartYaml, sourceChart.resolve('Chart.yaml').toFile()) - - if (chartLock == null) { - chartLock = [ - dependencies: [ - [ - name : 'crds', - repository: "", - version : '0.0.0' - ], - [ - name : 'grafana', - repository: 'https://grafana.github.io/helm-charts', - version : '7.3.9' - ] - ] - ] - } - fileSystemUtils.writeYaml(chartLock, sourceChart.resolve('Chart.lock').toFile()) - - config.application.localHelmChartFolder = rootChartsFolder.toString() - } - - protected void assertAirGapped() { - GitRepo prometheusRepo = gitRepoFactory.repos['3rd-party-dependencies/kube-prometheus-stack'] - assertThat(prometheusRepo).isNotNull() - assertThat(Path.of(prometheusRepo.absoluteLocalRepoTmpDir, 'Chart.lock')).doesNotExist() - - def ys = new YamlSlurper() - def actualPrometheusChartYaml = ys.parse(Path.of(prometheusRepo.absoluteLocalRepoTmpDir, 'Chart.yaml')) - assertThat(actualPrometheusChartYaml['name']).isEqualTo('kube-prometheus-stack-chart') - - def dependencies = actualPrometheusChartYaml['dependencies'] as List - assertThat(dependencies).hasSize(2) - assertThat(dependencies[0]['name']).isEqualTo('crds') - assertThat(dependencies[0]['version']).isEqualTo('0.0.0') - assertThat(dependencies[0]['repository']).isEqualTo('') - assertThat(dependencies[1]['name']).isEqualTo('grafana') - assertThat(dependencies[1]['version']).isEqualTo('7.3.9') - assertThat(dependencies[1]['repository']).isEqualTo('') - - assertHelmRepoCommits(prometheusRepo, '1.2.3', 'Chart kube-prometheus-stack-chart, version: 1.2.3\n\n' + - 'Source: https://kube-prometheus-stack-repo-url\nDependencies localized to run in air-gapped environments') - - verify(prometheusRepo).createRepositoryAndSetPermission(eq("Mirror of Helm chart kube-prometheus-stack from https://kube-prometheus-stack-repo-url"), - eq(false)) - } - - void assertHelmRepoCommits(GitRepo repo, String expectedTag, String expectedCommitMessage) { - def commits = Git.open(new File(repo.absoluteLocalRepoTmpDir)).log().setMaxCount(1).all().call().collect() - assertThat(commits.size()).isEqualTo(1) - assertThat(commits[0].fullMessage).isEqualTo(expectedCommitMessage) - - List tags = Git.open(new File(repo.absoluteLocalRepoTmpDir)).tagList().call() - assertThat(tags.size()).isEqualTo(1) - assertThat(tags[0].name).isEqualTo("refs/tags/${expectedTag}".toString()) - } - - AirGappedUtils createAirGappedUtils() { - new AirGappedUtils(config, gitRepoFactory, fileSystemUtils, helmClient, gitHandler) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/AllowlistFreemarkerObjectWrapperTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/AllowlistFreemarkerObjectWrapperTest.groovy deleted file mode 100644 index 0bf113c44..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/AllowlistFreemarkerObjectWrapperTest.groovy +++ /dev/null @@ -1,76 +0,0 @@ -package com.cloudogu.gitops.utils - -import static org.junit.jupiter.api.Assertions.* - -import freemarker.template.Configuration -import org.junit.jupiter.api.Test - -class AllowlistFreemarkerObjectWrapperTest { - - @Test - void 'should allow access to whitelisted static models'() { - def wrapper = new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, ["com.cloudogu.gitops.utils.DockerImageParser"] as Set) - def staticModels = wrapper.getStaticModels() - - assertNotNull(staticModels.get("com.cloudogu.gitops.utils.DockerImageParser")) - assertNull(staticModels.get("java.lang.Integer")) - assertNull(staticModels.get("java.lang.String")) - } - - @Test - void 'should deny access to non-whitelisted static models'() { - def wrapper = new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, ["java.lang.String"] as Set) - def staticModels = wrapper.getStaticModels() - - assertNull(staticModels.get("java.lang.Integer")) - assertNotNull(staticModels.get("java.lang.String")) - assertNull(staticModels.get("com.cloudogu.gitops.utils.DockerImageParser")) - } - - @Test - void 'should return true for isEmpty when allowlist is empty'() { - def wrapper = new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, [] as Set) - def staticModels = wrapper.getStaticModels() - - assertTrue(staticModels.isEmpty()) - } - - @Test - void 'templating only works for whitelisted statics'() { - def templateText = ''' - <#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> - <#assign imageObject = DockerImageParser.parse('test:latest')> - <#assign staticsTests=statics['System']> - <#assign imageObject = staticsTests.exit()> - '''.stripIndent() - - def model = [statics: new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, ['com.cloudogu.gitops.utils.DockerImageParser'] as Set).getStaticModels()] as Map - // create a temporary file to simulate an actual file input - def tempInputFile = File.createTempFile("test", ".ftl.yaml") - tempInputFile.text = templateText - - def exception = assertThrows(freemarker.core.InvalidReferenceException) { - new TemplatingEngine().replaceTemplates(tempInputFile, model) - } - - assert exception.message.contains("System"): "Exception message should mention 'System'" - } - - @Test - void 'templating in ftl files works correctly with whitelisted static models'() { - def templateText = ''' -<#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> -<#assign imageObject = DockerImageParser.parse('test:latest')> -<#assign staticsTests=statics['java.lang.Math']> -<#assign number = staticsTests.round(3.14)> - '''.stripIndent() - - def model = [statics: new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, ['java.lang.Math', 'com.cloudogu.gitops.utils.DockerImageParser'] as Set).getStaticModels()] as Map - // create a temporary file to simulate an actual file input - def tempInputFile = File.createTempFile("test", ".ftl.yaml") - tempInputFile.text = templateText - - new TemplatingEngine().replaceTemplates(tempInputFile, model) - - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/CommandExecutorForTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/CommandExecutorForTest.groovy deleted file mode 100644 index cc30202df..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/CommandExecutorForTest.groovy +++ /dev/null @@ -1,66 +0,0 @@ -package com.cloudogu.gitops.utils - -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.Mockito.mock - -class CommandExecutorForTest extends CommandExecutor { - List actualCommands = [] - - Queue outputs = new LinkedList() - - void enqueueOutput(Output output) { - outputs.add(output) - } - - void enqueueOutputs(Queue outputsQueue) { - outputs.addAll(outputsQueue) - } - - // This is actually only set when an env is passed to CommandExecutor - List environment = [] - - @Override - protected Output getOutput(Process proc, String command, boolean failOnError) { - actualCommands += command - Output output = outputs.poll() ?: new Output('', '', 0) - - if (failOnError && output.exitCode > 0) { - throw new RuntimeException("Executing command failed: ${command}") - } - - return output - } - - @Override - protected Process doExecute(String command) { - return mock(Process) - } - - @Override - protected Process doExecute(String[] command) { - return mock(Process) - } - - @Override - protected Process doExecute(String command, List envp) { - environment = envp - return mock(Process) - } - - String assertExecuted(String commandStartsWith) { - def actualCommand = actualCommands.find { - it.startsWith(commandStartsWith) - } - assertThat(actualCommand).as("Expected command to have been executed, but was not:\n${commandStartsWith}.\n" + "Actual commands:\n${actualCommands.join('\n')}") - .isNotNull() - return actualCommand - } - - void assertNotExecuted(String commandStartsWith) { - def actualCommand = actualCommands.find { - it.startsWith(commandStartsWith) - } - assertThat(actualCommand).as("Expected command to have been executed, but was not: ${commandStartsWith}") - .isNull() - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/CommandExecutorTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/CommandExecutorTest.groovy deleted file mode 100644 index e993f83e3..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/CommandExecutorTest.groovy +++ /dev/null @@ -1,21 +0,0 @@ -package com.cloudogu.gitops.utils - -import static org.assertj.core.api.Assertions.assertThat - -import org.junit.jupiter.api.Test - -class CommandExecutorTest { - - CommandExecutorForTest commandExecutor = new CommandExecutorForTest() - - @Test - void aggregatesEnvironment() { - def additionalEnv = [someKey: 'someValue'] - commandExecutor.execute('command', additionalEnv) - - assertThat(commandExecutor.actualCommands[0] as String).isEqualTo('command') - assertThat(commandExecutor.environment.toString()).contains('someKey=someValue') - // Make sure there are other env vars present and not solely the one we passed - assertThat(commandExecutor.environment.size()).isGreaterThan(additionalEnv.size()) - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/DockerImageParserTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/DockerImageParserTest.groovy deleted file mode 100644 index 43da66f1c..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/DockerImageParserTest.groovy +++ /dev/null @@ -1,35 +0,0 @@ -package com.cloudogu.gitops.utils - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat - -import org.junit.jupiter.api.Test - -class DockerImageParserTest { - @Test - void 'parses simple image string'() { - def result = DockerImageParser.parse('grafana/grafana:latest') - - assertThat(result.registry).isEqualTo('') - assertThat(result.repository).isEqualTo('grafana/grafana') - assertThat(result.getRegistryAndRepositoryAsString()).isEqualTo('grafana/grafana') - assertThat(result.tag).isEqualTo('latest') - } - - @Test - void 'parses image string with port'() { - def result = DockerImageParser.parse('localhost:5000/grafana/grafana:latest') - - assertThat(result.registry).isEqualTo('localhost:5000') - assertThat(result.repository).isEqualTo('grafana/grafana') - assertThat(result.getRegistryAndRepositoryAsString()).isEqualTo('localhost:5000/grafana/grafana') - assertThat(result.tag).isEqualTo('latest') - } - - @Test - void 'throws when there is no colon'() { - shouldFail(RuntimeException) { - DockerImageParser.parse('grafana/grafana') - } - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/FileSystemUtilsTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/FileSystemUtilsTest.groovy deleted file mode 100644 index 32b1a7e65..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/FileSystemUtilsTest.groovy +++ /dev/null @@ -1,105 +0,0 @@ -package com.cloudogu.gitops.utils - -import static org.assertj.core.api.Assertions.assertThat - -import java.nio.file.Files -import java.nio.file.Path - -import org.junit.jupiter.api.Test - -class FileSystemUtilsTest { - - FileSystemUtils fileSystemUtils = new FileSystemUtils() - - @Test - void copiesToTempDir() { - def expectedText = 'someText' - - File someFile = File.createTempFile(getClass().getSimpleName(), '') - someFile.withWriter { - { - it.println expectedText - } - } - Path tmpFile = fileSystemUtils.copyToTempDir(someFile.absolutePath) - - assertThat(tmpFile.toAbsolutePath().toString()).isNotEqualTo(someFile.getAbsoluteFile()) - assertThat(tmpFile.toFile().getText().trim()).isEqualTo(expectedText) - } - - @Test - void 'makes read-only folders writable recursively'() { - // Create temporary directory with nested structure - Path parentDir = Files.createTempDirectory(this.class.getSimpleName()) - - // Create some regular files - File regularFile = new File(parentDir.toFile(), "regularFile.txt") - regularFile.createNewFile() - - // Create nested directory - File nestedDir = new File(parentDir.toFile(), "nestedDir") - nestedDir.mkdir() - - // Create read-only file in nested directory - File readOnlyFile = new File(nestedDir, "readOnlyFile.txt") - readOnlyFile.createNewFile() - readOnlyFile.setWritable(false) - - // Create another read-only file in parent directory - File anotherReadOnlyFile = new File(parentDir.toFile(), "anotherReadOnlyFile.txt") - anotherReadOnlyFile.createNewFile() - anotherReadOnlyFile.setWritable(false) - - // Verify files are indeed read-only - assertThat(readOnlyFile.canWrite()).isFalse() - assertThat(anotherReadOnlyFile.canWrite()).isFalse() - - FileSystemUtils.makeWritable(parentDir.toFile()) - - // Verify all files are now writable - assertThat(regularFile.canWrite()).isTrue() - assertThat(readOnlyFile.canWrite()).isTrue() - assertThat(anotherReadOnlyFile.canWrite()).isTrue() - - // Clean up - parentDir.toFile().deleteDir() - } - - @Test - void 'reads and writes yaml'() { - Path tmpFile = fileSystemUtils.createTempFile() - Map yaml = [foo: 'bar', nested: [a: 1, b: 2]] - - fileSystemUtils.writeYaml(yaml, tmpFile.toFile()) - Map result = fileSystemUtils.readYaml(tmpFile) - - assertThat(result).isEqualTo(yaml) - } - - @Test - void 'readYaml falls back to classpath'() { - // testMainConfig.yaml exists in src/test/resources, so it is on the classpath - Map result = fileSystemUtils.readYaml(Path.of('testMainConfig.yaml')) - - assertThat(result) - .extracting('registry.internalPort') - .isEqualTo(30000) - } - - @Test - void 'readYaml falls back to classpath and removes src main resources'() { - // application-minimal.yaml exists in src/main/resources - // We simulate a path that might be in a config file pointing to the source tree - Map result = fileSystemUtils.readYaml(Path.of('src/main/resources/application-minimal.yaml')) - - assertThat(result) - .extracting('application.yes') - .isEqualTo(true) - } - - @Test - void 'readYaml returns empty map if not found'() { - Map result = fileSystemUtils.readYaml(Path.of('non-existent.yaml')) - assertThat(result).isEmpty() - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/HelmClientTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/HelmClientTest.groovy deleted file mode 100644 index e69de29bb..000000000 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/K8sClientForTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/K8sClientForTest.groovy deleted file mode 100644 index 5a4e0fad1..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/K8sClientForTest.groovy +++ /dev/null @@ -1,13 +0,0 @@ -package com.cloudogu.gitops.utils - -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer - -class K8sClientForTest extends K8sClient { - - K8sClientForTest() { - super() - this.client = new KubernetesMockServer().createClient() - this.SLEEPTIME = 1 - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/K8sClientTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/K8sClientTest.groovy deleted file mode 100644 index e69de29bb..000000000 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/NetworkingUtilsTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/NetworkingUtilsTest.groovy deleted file mode 100644 index 141cb4b6f..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/NetworkingUtilsTest.groovy +++ /dev/null @@ -1,75 +0,0 @@ -package com.cloudogu.gitops.utils - -import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient -import org.junit.jupiter.api.Test - -import static groovy.test.GroovyAssert.shouldFail -import static org.assertj.core.api.Assertions.assertThat -import static org.mockito.Mockito.mock -import static org.mockito.Mockito.when - -class NetworkingUtilsTest { - - K8sClient k8sClient = mock(K8sClient) - CommandExecutorForTest commandExecutor = new CommandExecutorForTest() - NetworkingUtils networkingUtils = new NetworkingUtils(k8sClient, commandExecutor) - - @Test - void 'clusterBindAddress: returns bind address for external cluster'() { - def internalNodeIp = "1.2.3.4" - def localIp = "5.6.7.8" - when(k8sClient.waitForInternalNodeIp()).thenReturn(internalNodeIp) - commandExecutor.enqueueOutput(new CommandExecutor.Output('', - "1.0.0.0 via w.x.y.z dev someDevice src ${localIp} uid 1000", 0)) - - def actualBindAddress = networkingUtils.findClusterBindAddress() - - assertThat(actualBindAddress).isEqualTo(internalNodeIp) - } - - @Test - void 'clusterBindAddress: returns localhost when node IP and local IP are equal'() { - def internalNodeIp = networkingUtils.localAddress - assertThat(internalNodeIp).isNotEmpty() - - when(k8sClient.waitForInternalNodeIp()).thenReturn(internalNodeIp) - - def actualBindAddress = networkingUtils.findClusterBindAddress() - - assertThat(actualBindAddress).isEqualTo('localhost') - } - - @Test - void 'clusterBindAddress: fails when no potential bind address'() { - when(k8sClient.waitForInternalNodeIp()).thenReturn('') - commandExecutor.enqueueOutput(new CommandExecutor.Output('', - "1.0.0.0 via w.x.y.z dev someDevice src 1.2.3.4 uid 1000", 0)) - - def exception = shouldFail(RuntimeException) { - networkingUtils.findClusterBindAddress() - } - assertThat(exception.message).isEqualTo('Could not connect to kubernetes cluster: no cluster bind address') - } - - @Test - void 'get hosts'() { - assertThat(NetworkingUtils.getHost("https://example.com")).isEqualTo("example.com") - assertThat(NetworkingUtils.getHost("http://example.com")).isEqualTo("example.com") - assertThat(NetworkingUtils.getHost("")).isEqualTo("") - assertThat(NetworkingUtils.getHost("example.com")).isEqualTo("example.com") - - assertThat(NetworkingUtils.getHost("http://example.com/bla")).isEqualTo("example.com/bla") - assertThat(NetworkingUtils.getHost("http://example.com:9090/bla")).isEqualTo("example.com:9090/bla") - assertThat(NetworkingUtils.getHost("example.com/bla")).isEqualTo("example.com/bla") - assertThat(NetworkingUtils.getHost("example.com:9090/bla")).isEqualTo("example.com:9090/bla") - } - - @Test - void 'get protocols'() { - assertThat(NetworkingUtils.getProtocol("https://example.com")).isEqualTo("https"); - assertThat(NetworkingUtils.getProtocol("http://example.com")).isEqualTo("http"); - assertThat(NetworkingUtils.getProtocol("ftp://example.com")).isEqualTo(""); - assertThat(NetworkingUtils.getProtocol("example.com")).isEqualTo(""); - assertThat(NetworkingUtils.getProtocol("")).isEqualTo("") - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/TemplatingEngineTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/TemplatingEngineTest.groovy deleted file mode 100644 index ac5256c61..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/TemplatingEngineTest.groovy +++ /dev/null @@ -1,102 +0,0 @@ -package com.cloudogu.gitops.utils - -import static org.assertj.core.api.Assertions.assertThat - -import org.junit.jupiter.api.BeforeEach -import org.junit.jupiter.api.Test - -class TemplatingEngineTest { - - File tmpDir - - @BeforeEach - void before() { - tmpDir = File.createTempDir('gitops-playground-tests-templatingengine') - tmpDir.deleteOnExit() - } - - @Test - void 'replaces two templates in different folders'() { - def fooTemplate = new File(tmpDir.absolutePath, "foo.ftl.txt") - fooTemplate.text = """ - this is the template - I can embed \${string} - <#if display> - and use ifs - <#else> - and use elses - - """ - - def tmpDir2 = File.createTempDir('gitops-playground-tests-templatingengine') - tmpDir2.deleteOnExit() - def barTemplate = new File(tmpDir2.absolutePath, "bar.ftl.txt") - barTemplate.text = "Hello \${name}" - - def engine = new TemplatingEngine() - engine.replaceTemplate(barTemplate, [name: "Playground",]) - - assertThat(new File(tmpDir2.absolutePath, "bar.txt").text).isEqualTo("Hello Playground") - assertThat(barTemplate).doesNotExist() - } - - @Test - void 'keeps template file'() { - def barTemplate = new File(tmpDir.absolutePath, "bar.ftl.txt") - def barTarget = new File(tmpDir.absolutePath, "bar.txt") - barTemplate.text = "Hello \${name}" - - def engine = new TemplatingEngine() - engine.template(barTemplate, barTarget, [name: "Playground",]) - - assertThat(barTarget.text).isEqualTo("Hello Playground") - assertThat(barTemplate).exists() - } - - @Test - void 'Templates from file to string'() { - def fooTemplate = new File(tmpDir.absolutePath, "foo.ftl.txt") - fooTemplate.text = "Hello \${name}" - - def engine = new TemplatingEngine() - String result = engine.template(fooTemplate, [name: "Playground",]) - - assertThat(result).isEqualTo("Hello Playground") - } - - @Test - void 'Templates from string to string'() { - def fooTemplate = "Hello \${name}" - - def engine = new TemplatingEngine() - String result = engine.template(fooTemplate, [name: "Playground",]) - - assertThat(result).isEqualTo("Hello Playground") - } - - @Test - void 'Ignores templates without variables'() { - def fooTemplate = "Hello name" - - def engine = new TemplatingEngine() - String result = engine.template(fooTemplate, [:]) - - assertThat(result).isEqualTo("Hello name") - } - - @Test - void "replaces yaml templates"() { - def barTemplate = new File(tmpDir.absolutePath + File.separator + "subdirectory", "result.ftl.yaml") - barTemplate.getParentFile().mkdirs() - barTemplate.text = 'foo: ${prefix}suffix' - def barTarget = new File(tmpDir.absolutePath, "subdirectory/keep-this-way.yaml") - barTarget.text = 'thiswont: ${prefix}-be-replaced' - - def engine = new TemplatingEngine() - engine.replaceTemplates(tmpDir, [prefix: "myteam-"]) - - assertThat(new File("$tmpDir/subdirectory/result.yaml").text).isEqualTo("foo: myteam-suffix") - assertThat(new File("$tmpDir/subdirectory/keep-this-way.yaml").text).isEqualTo('thiswont: ${prefix}-be-replaced') - assertThat(new File("$tmpDir/subdirectory/result.ftl.yaml").exists()).isFalse() - } -} \ No newline at end of file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProviderTest.groovy b/src/test/groovy/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProviderTest.groovy deleted file mode 100644 index 93e80c124..000000000 --- a/src/test/groovy/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProviderTest.groovy +++ /dev/null @@ -1,42 +0,0 @@ -package com.cloudogu.gitops.utils.jgit.helpers - -import org.eclipse.jgit.transport.CredentialItem -import org.eclipse.jgit.transport.URIish -import org.junit.jupiter.api.Test - -import static org.assertj.core.api.Assertions.assertThat - -class InsecureCredentialProviderTest { - @Test - void 'ignores irrelevant items'() { - def provider = new InsecureCredentialProvider() - - assertThat(provider.supports(new CredentialItem.Username(), new CredentialItem.Password())).isFalse() - assertThat(provider.supports(new CredentialItem.InformationalMessage("This is not a relevant message"), - new CredentialItem.YesNoType("This prompt is irrelevant as well"))).isFalse() - } - - @Test - void 'confirms insecure https processing'() { - def provider = new InsecureCredentialProvider() - - def message = new CredentialItem.InformationalMessage("A secure connection to https://192.168.178.37/scm/repo/argocd/cluster-resources could not be established because the server's certificate could not be validated.\n" + - "SSL reported: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n" + - "Do you want to skip SSL verification for this server?") - def skipSingle = new CredentialItem.YesNoType("Skip SSL verification for this single git operation") - def skipRepository = new CredentialItem.YesNoType("Skip SSL verification for git operations for repository /tmp/groovy-generated-tmpdir-2746077697650757929/.git") - def skipAlways = new CredentialItem.YesNoType("Always skip SSL verification for this server from now on") - - assertThat(provider.supports(message, - skipSingle, - skipRepository, - skipAlways)).isTrue() - - assertThat(provider.get(new URIish("https://192.168.178.37/scm/repo/argocd/cluster-resources"), message, skipSingle, skipRepository, skipAlways)) - .isTrue() - - assertThat(skipSingle.value).isTrue() - assertThat(skipRepository.value).isTrue() - assertThat(skipAlways.value).isFalse() - } -} \ No newline at end of file diff --git a/src/test/java/com/cloudogu/gitops/application/ApplicationTest.java b/src/test/java/com/cloudogu/gitops/application/ApplicationTest.java new file mode 100644 index 000000000..743b6a139 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/ApplicationTest.java @@ -0,0 +1,276 @@ +package com.cloudogu.gitops.application; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.DeploymentOrchestrator; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryProvisioning; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import io.micronaut.context.ApplicationContext; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.argThat; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class ApplicationTest { + + private final Config config = new Config(); + + @Test + void validatesGitConfigurationBeforeBuildingDeploymentContext() { + ContextBuilder contextBuilder = mock(ContextBuilder.class); + K8sClient k8sClient = mock(K8sClient.class); + GitHandler gitHandler = mock(GitHandler.class); + RepositoryProvisioning repositoryProvisioning = mock(RepositoryProvisioning.class); + DeploymentOrchestrator deploymentOrchestrator = mock(DeploymentOrchestrator.class); + DeploymentContext context = buildContext(); + RepositoryWorkspace workspace = mock(RepositoryWorkspace.class); + + when(contextBuilder.build()).thenReturn(context); + when(deploymentOrchestrator.getTools()).thenReturn(List.of()); + when(repositoryProvisioning.provideWorkspace(context)).thenReturn(workspace); + + Application application = new Application( + config, + contextBuilder, + k8sClient, + new CredentialsResolver(k8sClient), + gitHandler, + repositoryProvisioning, + deploymentOrchestrator + ); + + application.start(); + + var order = inOrder(gitHandler, contextBuilder); + order.verify(gitHandler).validate(); + order.verify(contextBuilder).build(); + } + + @Test + void storesResolvedApplicationPasswordWithoutMutatingConfig() { + ContextBuilder contextBuilder = mock(ContextBuilder.class); + K8sClient k8sClient = mock(K8sClient.class); + GitHandler gitHandler = mock(GitHandler.class); + RepositoryProvisioning repositoryProvisioning = mock(RepositoryProvisioning.class); + DeploymentOrchestrator deploymentOrchestrator = mock(DeploymentOrchestrator.class); + DeploymentContext context = buildContext(); + RepositoryWorkspace workspace = mock(RepositoryWorkspace.class); + Credentials reference = new Credentials(); + reference.setSecretName("argocd-credentials"); + reference.setSecretNamespace("gop-job"); + config.getApplication().setCredentials(reference); + config.getApplication().setUsername("fallback-user"); + config.getApplication().setPassword("fallback-password"); + + when(contextBuilder.build()).thenReturn(context); + when(deploymentOrchestrator.getTools()).thenReturn(List.of()); + when(repositoryProvisioning.provideWorkspace(context)).thenReturn(workspace); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))) + .thenReturn(new Credentials("secret-user", "secret-password")); + + Application application = new Application( + config, + contextBuilder, + k8sClient, + new CredentialsResolver(k8sClient), + gitHandler, + repositoryProvisioning, + deploymentOrchestrator + ); + + application.start(); + + verify(k8sClient).createSecret( + eq("generic"), + eq("gop-configuration"), + eq("gop-job"), + argThat(tuple -> "gop-initial-password".equals(tuple.getFirst()) + && "secret-password".equals(tuple.getSecond())), + argThat(tuple -> "gop-config".equals(tuple.getFirst()) + && tuple.getSecond().toString().contains("secretName: \"argocd-credentials\"") + && !tuple.getSecond().toString().contains("secret-password")) + ); + assertThat(config.getApplication().getUsername()).isEqualTo("fallback-user"); + assertThat(config.getApplication().getPassword()).isEqualTo("fallback-password"); + assertThat(reference.getUsername()).isNull(); + assertThat(reference.getPassword()).isNull(); + } + + @Test + void featuresOrderingIsCorrect() { + Application application = ApplicationContext.run() + .registerSingleton(config) + .getBean(Application.class); + + List features = application.getTools().stream() + .map(tool -> tool.getClass().getSimpleName()) + .collect(Collectors.toList()); + + assertThat(features).isEqualTo(List.of( + "ScmManager", + "Registry", + "ArgoCD", + "Ingress", + "CertManager", + "Jenkins", + "Monitoring", + "ExternalSecretsOperator", + "Vault", + "ContentLoader" + )); + } + + @Test + void getActiveNamespacesCorrectly() { + config.getRegistry().setActive(true); + config.getJenkins().setActive(true); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getArgocd().setActive(true); + config.getFeatures().getIngress().setActive(true); + config.getApplication().setNamePrefix("test1-"); + config.getContent().setNamespaces(List.of( + "${config.application.namePrefix}example-apps-staging", + "${config.application.namePrefix}example-apps-production" + )); + + List namespaceList = new ArrayList<>(Arrays.asList( + "test1-argocd", + "test1-example-apps-staging", + "test1-example-apps-production", + "test1-" + config.getFeatures().getIngress().getIngressNamespace(), + "test1-monitoring", + "test1-registry", + "test1-jenkins" + )); + + Application application = ApplicationContext.run() + .registerSingleton(config) + .getBean(Application.class); + + application.setNamespaceListToConfig(buildContext()); + + assertThat(config.getApplication().getNamespaces().getActiveNamespaces()) + .containsExactlyInAnyOrderElementsOf(namespaceList); + } + + @Test + void getActiveNamespacesCorrectlyInOpenshift() { + config.getRegistry().setActive(true); + config.getJenkins().setActive(true); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getArgocd().setActive(true); + config.getFeatures().getIngress().setActive(true); + config.getApplication().setNamePrefix("test1-"); + config.getApplication().setOpenshift(true); + config.getContent().setNamespaces(List.of( + "${config.application.namePrefix}example-apps-staging", + "${config.application.namePrefix}example-apps-production" + )); + + List namespaceList = new ArrayList<>(Arrays.asList( + "test1-argocd", + "test1-example-apps-staging", + "test1-example-apps-production", + "test1-" + config.getFeatures().getIngress().getIngressNamespace(), + "test1-monitoring", + "test1-registry", + "test1-jenkins" + )); + + Application application = ApplicationContext.run() + .registerSingleton(config) + .getBean(Application.class); + + application.setNamespaceListToConfig(buildContext()); + + assertThat(config.getApplication().getNamespaces().getActiveNamespaces()) + .containsExactlyInAnyOrderElementsOf(namespaceList); + } + + @Test + void handlesContentNamespacesWithoutTemplate() { + config.getContent().setNamespaces(List.of( + "example-apps-staging", + "example-apps-production" + )); + + Application application = ApplicationContext.run() + .registerSingleton(config) + .getBean(Application.class); + + application.setNamespaceListToConfig(buildContext()); + + assertThat(config.getApplication().getNamespaces().getActiveNamespaces()).containsAll(List.of( + "example-apps-staging", + "example-apps-production" + )); + } + + @Test + void handlesEmptyContentNamespaces() { + Application application = ApplicationContext.run() + .registerSingleton(config) + .getBean(Application.class); + + application.setNamespaceListToConfig(buildContext()); + + // No exception == happy + } + + @Test + void getActiveNamespacesCorrectlyInOpenshiftIfJenkinsAndScmAreExternal() { + config.getRegistry().setActive(true); + config.getJenkins().setActive(true); + config.getJenkins().setInternal(false); + config.getScm().setScmManager(new ScmTenantSchema.ScmManagerTenantConfig()); + config.getScm().getScmManager().setInternal(false); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getArgocd().setActive(true); + config.getFeatures().getIngress().setActive(true); + config.getApplication().setNamePrefix("test1-"); + config.getApplication().setOpenshift(true); + config.getContent().setNamespaces(List.of( + "${config.application.namePrefix}example-apps-staging", + "${config.application.namePrefix}example-apps-production" + )); + + List namespaceList = new ArrayList<>(Arrays.asList( + "test1-argocd", + "test1-example-apps-staging", + "test1-example-apps-production", + "test1-" + config.getFeatures().getIngress().getIngressNamespace(), + "test1-monitoring", + "test1-registry" + )); + + Application application = ApplicationContext.run() + .registerSingleton(config) + .getBean(Application.class); + + application.setNamespaceListToConfig(buildContext()); + + assertThat(config.getApplication().getNamespaces().getActiveNamespaces()) + .containsExactlyInAnyOrderElementsOf(namespaceList); + } + + private DeploymentContext buildContext() { + return new ContextBuilder(config).build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/application/content/ContentLoaderTest.java b/src/test/java/com/cloudogu/gitops/application/content/ContentLoaderTest.java new file mode 100644 index 000000000..1af5e2015 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/content/ContentLoaderTest.java @@ -0,0 +1,1521 @@ +package com.cloudogu.gitops.application.content; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.testhelper.git.TestScmManagerApiClient; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.core.Jenkins; +import com.cloudogu.gitops.tools.core.JenkinsToolConfigMapper; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.io.FileUtils; +import org.eclipse.jgit.api.CloneCommand; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.errors.ConfigInvalidException; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider; +import org.eclipse.jgit.util.SystemReader; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.ArgumentCaptor; + +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Field; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.function.Consumer; + +import static com.cloudogu.gitops.config.Config.ContentRepoType; +import static com.cloudogu.gitops.config.Config.OverwriteMode; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@Slf4j +@EnableKubernetesMockClient(crud = true) +@SuppressWarnings("unchecked") +class ContentLoaderTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final List foldersToDelete = new ArrayList<>(); + + private final Config config = createConfig(); + private final K8sClient k8sClient = new K8sClient(); + private final CredentialsResolver credentialsResolver = new CredentialsResolver(k8sClient); + private final TestGitRepoFactory scmmRepoProvider = new TestGitRepoFactory(config, new FileSystemUtils()); + private final TestScmManagerApiClient scmmApiClient = new TestScmManagerApiClient(config); + private final JenkinsForTest jenkins = new JenkinsForTest(); + private final ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + private final GitHandler gitHandler = new GitHandlerForTests(scmManagerMock); + private final Deployer deployer = mock(Deployer.class); + private final RepositoryWorkspace repositoryWorkspace = mock(RepositoryWorkspace.class); + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + + KubernetesClient client; + + @TempDir + File tmpDir; + + private final List expectedTargetRepos = List.of( + repoCoordinate("common", "repo"), + repoCoordinate("ns1a", "repo1a1"), + repoCoordinate("ns1a", "repo1a2"), + repoCoordinate("ns1b", "repo1b1"), + repoCoordinate("ns1b", "repo1b2"), + repoCoordinate("ns2a", "repo2a1"), + repoCoordinate("ns2a", "repo2a2"), + repoCoordinate("ns2b", "repo2b1"), + repoCoordinate("ns2b", "repo2b2"), + repoCoordinate("copy", "repo1"), + repoCoordinate("copy", "repo2") + ); + + private final List contentRepos = List.of( + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("copy/repo1"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("copy/repo2"); + repo.setPath("subPath"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setPath("subPath"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo1")); + repo.setType(ContentRepoType.FOLDER_BASED); + repo.setTemplating(true); + }), + repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo2")); + repo.setType(ContentRepoType.FOLDER_BASED); + repo.setPath("subPath"); + }) + ); + + @AfterAll + static void cleanFolders() { + for (File folder : foldersToDelete) { + FileUtils.deleteQuietly(folder); + } + } + + @Disabled("TODO: Does not run on Jenkins: Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name or service not known") + @Test + void deploysImagePullSecrets() { + config.getRegistry().setCreateImagePullSecrets(true); + config.getContent().setNamespaces(List.of("example-apps-staging", "example-apps-production")); + + install(createContent(config), config); + + assertRegistrySecrets("reg-user", "reg-pw"); + } + + @Disabled("TODO: Does not run on Jenkins: Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name or service not known") + @Test + void deploysImagePullSecretsFromReadOnlyVars() { + config.getRegistry().setCreateImagePullSecrets(true); + config.getContent().setNamespaces(List.of("example-apps-staging", "example-apps-production")); + config.getRegistry().setReadOnlyUsername("other-user"); + config.getRegistry().setReadOnlyPassword("other-pw"); + + install(createContent(config), config); + + assertRegistrySecrets("other-user", "other-pw"); + } + + @Disabled("TODO: Does not run on Jenkins: Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name or service not known") + @Test + void deploysAdditionalImagePullSecretsForProxyRegistry() { + config.getRegistry().setCreateImagePullSecrets(true); + config.getContent().setNamespaces(List.of("example-apps-staging", "example-apps-production")); + config.getRegistry().setTwoRegistries(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-pw"); + + install(createContent(config), config); + + assertRegistrySecrets("reg-user", "reg-pw"); + } + + @Test + void resolvesRegistrySecretsForContentImagePullSecrets() { + Config contentConfig = createConfig(); + contentConfig.getRegistry().setCreateImagePullSecrets(true); + contentConfig.getRegistry().setTwoRegistries(true); + contentConfig.getRegistry().setProxyUrl("proxy-url"); + contentConfig.getRegistry().setReadOnlyCredentials( + new Credentials(null, null, "registry-read-only-credentials", "gop-job") + ); + contentConfig.getRegistry().setProxyCredentials( + new Credentials(null, null, "registry-proxy-credentials", "gop-job") + ); + contentConfig.getContent().setNamespaces(List.of("example-apps-staging")); + + K8sClient runtimeK8sClient = mock(K8sClient.class); + when(runtimeK8sClient.getCredentialsFromSecret(any(Credentials.class))).thenAnswer(invocation -> { + Credentials reference = invocation.getArgument(0); + if ("registry-read-only-credentials".equals(reference.getSecretName())) { + return new Credentials("runtime-read-only-user", "runtime-read-only-password"); + } + return new Credentials("runtime-proxy-user", "runtime-proxy-password"); + }); + ContentLoaderForTest contentLoader = new ContentLoaderForTest( + contentConfig, + runtimeK8sClient, + new CredentialsResolver(runtimeK8sClient), + scmmRepoProvider, + jenkins, + gitHandler, + fileSystemUtils, + deployer + ); + + contentLoader.createImagePullSecrets(); + + verify(runtimeK8sClient).createImagePullSecret( + "registry", + "example-apps-staging", + "reg-url", + "runtime-read-only-user", + "runtime-read-only-password" + ); + verify(runtimeK8sClient).createImagePullSecret( + "proxy-registry", + "example-apps-staging", + "proxy-url", + "runtime-proxy-user", + "runtime-proxy-password" + ); + assertThat(contentConfig.getRegistry().getReadOnlyPassword()).isEmpty(); + assertThat(contentConfig.getRegistry().getProxyPassword()).isEmpty(); + } + + @Test + void combinesContentReposSuccessfully() throws Exception { + config.getContent().setRepos(contentRepos); + + List repos = cloneContentRepos(createContent(config), config); + + for (ContentLoader.RepoCoordinate expected : expectedTargetRepos) { + assertThat(new File(findRoot(repos), expected.getNamespace() + "/" + expected.getRepoName() + "/file")) + .exists() + .isFile(); + } + + assertThat(Files.readString(new File(findRoot(repos), "common/repo/file").toPath())) + .contains("folderBasedRepo2"); + + assertThat(new File(findRoot(repos), "common/repo/folderBasedRepo1")).exists().isFile(); + assertThat(new File(findRoot(repos), "common/repo/folderBasedRepo2")).exists().isFile(); + assertThat(new File(findRoot(repos), "common/repo/copyRepo1")).exists().isFile(); + assertThat(new File(findRoot(repos), "common/repo/copyRepo2")).exists().isFile(); + + assertThat(new File(findRoot(repos), "common/repo/some.yaml")).exists(); + assertThat(Files.readString(new File(findRoot(repos), "common/repo/some.yaml").toPath())) + .contains("namePrefix: foo-"); + assertThat(new File(findRoot(repos), "common/repo/someOther.yaml.ftl")).exists(); + assertThat(Files.readString(new File(findRoot(repos), "common/repo/someOther.yaml.ftl").toPath())) + .contains("namePrefix: ${config.application.namePrefix}"); + } + + @Test + void supportsContentVariables() throws Exception { + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo1")); + repo.setType(ContentRepoType.FOLDER_BASED); + repo.setTemplating(true); + }))); + config.getContent().getVariables().put("someapp", Map.of("somevalue", "this is a custom variable")); + + List repos = cloneContentRepos(createContent(config), config); + + assertThat(new File(findRoot(repos), "common/repo/some.yaml")).exists(); + assertThat(Files.readString(new File(findRoot(repos), "common/repo/some.yaml").toPath())) + .contains("namePrefix: foo-"); + assertThat(Files.readString(new File(findRoot(repos), "common/repo/some.yaml").toPath())) + .contains("myvar: this is a custom variable"); + } + + @Test + void authenticatesContentRepos() throws Exception { + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setCredentials(new Credentials("user", "pw")); + }))); + + ContentLoaderForTest content = createContent(config); + cloneContentRepos(content, config); + + ArgumentCaptor captor = + ArgumentCaptor.forClass(UsernamePasswordCredentialsProvider.class); + verify(content.cloneSpy).setCredentialsProvider(captor.capture()); + + UsernamePasswordCredentialsProvider value = captor.getValue(); + assertThat(readPrivateField(value, "username")).isEqualTo("user"); + assertThat((char[]) readPrivateField(value, "password")).isEqualTo("pw".toCharArray()); + } + + @Test + @DisplayName("Authenticates content Repos with secret") + void authenticatesContentReposWithSecret() throws Exception { + k8sClient.setClient(client); + Secret secret = new SecretBuilder() + .withNewMetadata() + .withName("secret-test-name") + .withNamespace("default") + .endMetadata() + .withType("Opaque") + .withData(Map.of( + "username", "YWRtaW4=", + "password", "czNjcjN0" + )) + .build(); + + k8sClient.getClient().secrets() + .inNamespace("default") + .resource(secret) + .create(); + + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setCredentials(new Credentials(null, null, "secret-test-name", "default")); + }))); + + ContentLoaderForTest content = createContent(config); + cloneContentRepos(content, config); + + ArgumentCaptor captor = + ArgumentCaptor.forClass(UsernamePasswordCredentialsProvider.class); + verify(content.cloneSpy).setCredentialsProvider(captor.capture()); + + UsernamePasswordCredentialsProvider value = captor.getValue(); + assertThat(readPrivateField(value, "username")).isEqualTo("admin"); + assertThat((char[]) readPrivateField(value, "password")).isEqualTo("s3cr3t".toCharArray()); + } + + @Test + void checksOutCommitRefsTagsAndNonDefaultBranchesForContentRepos() throws Exception { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setRef("someTag"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/tag"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setRef("8bc1d1165468359b16d9771d4a9a3df26afc03e8"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/ref"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setRef("someBranch"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/branch"); + }) + )); + + List repos = cloneContentRepos(createContent(config), config); + + assertThat(new File(findRoot(repos), "common/tag/README.md")).exists().isFile(); + assertThat(Files.readString(new File(findRoot(repos), "common/tag/README.md").toPath())).contains("someTag"); + assertThat(new File(findRoot(repos), "common/ref/README.md")).exists().isFile(); + assertThat(Files.readString(new File(findRoot(repos), "common/ref/README.md").toPath())).contains("main"); + assertThat(new File(findRoot(repos), "common/branch/README.md")).exists().isFile(); + assertThat(Files.readString(new File(findRoot(repos), "common/branch/README.md").toPath())).contains( + "someBranch"); + } + + @Test + void checksOutDefaultBranchWhenNoRefSet() throws Exception { + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("", "git-repo-different-default-branch")); + repo.setTarget("common/default"); + repo.setType(ContentRepoType.COPY); + }))); + + List repos = cloneContentRepos(createContent(config), config); + + assertThat(new File(findRoot(repos), "common/default/README.md")).exists().isFile(); + assertThat(Files.readString(new File(findRoot(repos), "common/default/README.md").toPath())).contains( + "different"); + } + + @Test + void failsIfCommitRefDoesNotExist() { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setRef("someTag"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/tag"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setRef("does/not/exist"); + repo.setType(ContentRepoType.FOLDER_BASED); + repo.setTarget("does not matter"); + }) + )); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> cloneContentRepos(createContent(config), config) + ); + + assertThat(exception.getMessage()).startsWith("Reference 'does/not/exist' not found in content repository"); + } + + @Test + void respectsOrderOfFolderBasedRepositories() throws Exception { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.FOLDER_BASED); + }), + repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo2")); + repo.setRef("main"); + repo.setType(ContentRepoType.FOLDER_BASED); + repo.setPath("subPath"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setPath("subPath"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + }) + )); + + List repos = cloneContentRepos(createContent(config), config); + + assertThat(Files.readString(new File(findRoot(repos), "common/repo/file").toPath())).contains("copyRepo1"); + } + + @Test + void isAbleToCopyIntoMirroredRepo() throws IOException, GitAPIException { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("mirrorRepo1", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo1")); + repo.setType(ContentRepoType.FOLDER_BASED); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + repo.setPath("subPath"); + }) + )); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + + try (Git git = cloneRepo("common/repo", tmpDir)) { + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("copyRepo2"); + assertThat(new File(tmpDir, "mirrorRepo1")).exists().isFile(); + assertThat(new File(tmpDir, "copyRepo2")).exists().isFile(); + assertThat(new File(tmpDir, "folderBasedRepo1")).exists().isFile(); + + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertTag(git, "someTag"); + assertBranch(git, "someBranch"); + } + } + + @Test + void handlesMirrorAndCopyTogether() throws IOException, GitAPIException { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("folderBasedRepo1")); + repo.setType(ContentRepoType.FOLDER_BASED); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + repo.setPath("subPath"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("mirrorRepo1", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setOverwriteMode(OverwriteMode.RESET); + repo.setTarget("common/repo"); + }) + )); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + + try (Git git = cloneRepo("common/repo", tmpDir)) { + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("mirrorRepo1"); + assertThat(new File(tmpDir, "folderBasedRepo1")).doesNotExist(); + assertThat(new File(tmpDir, "copyRepo2")).doesNotExist(); + + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertTag(git, "someTag"); + assertBranch(git, "someBranch"); + } + } + + @Test + void handlesMultipleMirrorsOfTheSameRepoWithDifferentRefs() throws IOException, GitAPIException { + String repoToMirror = createContentRepo("mirrorRepo1", "git-repository-with-branches-tags"); + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(repoToMirror); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("main"); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(repoToMirror); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("someBranch"); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + }), + repository(repo -> { + repo.setUrl(repoToMirror); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("someTag"); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + repo.setPath("subPath"); + }) + )); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + + try (Git git = cloneRepo("common/repo", tmpDir)) { + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("copyRepo2"); + assertThat(new File(tmpDir, "mirrorRepo1")).exists().isFile(); + + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertTag(git, "someTag"); + assertBranch(git, "someBranch"); + } + } + + @Test + void handlesTargetRefs() throws IOException, GitAPIException { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("mirror/tag"); + repo.setRef("someTag"); + repo.setTargetRef("my-tag"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("mirror/branch"); + repo.setRef("someBranch"); + repo.setTargetRef("my-branch"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("copy/tag"); + repo.setRef("someTag"); + repo.setTargetRef("my-tag"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("copy/branch"); + repo.setRef("someBranch"); + repo.setTargetRef("my-branch"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("mirror/tag2branch"); + repo.setRef("someTag"); + repo.setTargetRef("refs/heads/my-branch"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("mirror/branch2tag"); + repo.setRef("someBranch"); + repo.setTargetRef("refs/tags/my-tag"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("copy/tag2branch"); + repo.setRef("someTag"); + repo.setTargetRef("refs/heads/my-branch"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("copy/branch2tag"); + repo.setRef("someBranch"); + repo.setTargetRef("refs/tags/my-tag"); + }) + )); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + + assertTagAndReadme("mirror/tag", "my-tag", "someTag"); + assertBranchAndReadme("mirror/branch", "my-branch", "someBranch"); + assertTagAndReadme("copy/tag", "my-tag", "someTag"); + assertBranchAndReadme("copy/branch", "my-branch", "someBranch"); + assertTagAndReadme("mirror/branch2tag", "my-tag", "someBranch"); + assertBranchAndReadme("mirror/tag2branch", "my-branch", "someTag"); + assertTagAndReadme("copy/branch2tag", "my-tag", "someBranch"); + assertBranchAndReadme("copy/tag2branch", "my-branch", "someTag"); + } + + @Test + void handlesMultipleMirrorsOfSameRepoWhereOneIsNotPushed() { + String repoToMirror = createContentRepo("copyRepo1", "git-repository-with-branches-tags"); + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(repoToMirror); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("main"); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(repoToMirror); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("someBranch"); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.UPGRADE); + repo.setPath("subPath"); + }) + )); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + } + + @Test + void isAbleToMirrorIntoRepoThatHasSameCommits() { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("mirrorRepo1", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("mirrorRepo1", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("common/repo"); + repo.setOverwriteMode(OverwriteMode.RESET); + }) + )); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + } + + @Test + void parsesRepoCoordinates() throws Exception { + config.getContent().setRepos(contentRepos); + + ContentLoaderForTest content = createContent(config); + List actualTargetRepos = cloneContentRepos(content, config); + List repos = actualTargetRepos; + + assertThat(actualTargetRepos).hasSameSizeAs(expectedTargetRepos); + + for (ContentLoader.RepoCoordinate expected : expectedTargetRepos) { + List actual = actualTargetRepos.stream() + .filter(candidate -> candidate.getNamespace().equals( + expected.getNamespace()) + && candidate.getRepoName().equals(expected.getRepoName())) + .toList(); + + assertThat(actual) + .withFailMessage( + "Could not find repo with namespace=%s and repo=%s in %s", + expected.getNamespace(), + expected.getRepoName(), + actualTargetRepos + ) + .hasSize(1); + + assertThat(actual.get(0).getClonedContentRepo().getAbsolutePath()) + .isEqualTo(new File( + findRoot(repos), + expected.getNamespace() + "/" + expected.getRepoName() + ).getAbsolutePath()); + } + } + + @Test + void createsAndPushesContentReposWholeFlow() throws IOException, GitAPIException { + List repos = new ArrayList<>(contentRepos); + repos.add(repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setTarget("common/mirror"); + })); + repos.add(repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("main"); + repo.setTarget("common/mirrorWithBranchRef"); + })); + repos.add(repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("someTag"); + repo.setTarget("common/mirrorWithTagRef"); + })); + config.getContent().setRepos(repos); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + + String expectedRepo = "copy/repo1"; + try (Git git = cloneRepo(expectedRepo, tmpDir)) { + String commitMsg = git.log().call().iterator().next().getFullMessage(); + assertThat(commitMsg).isEqualTo("Initialize content repo " + expectedRepo); + + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("copyRepo1"); + assertThat(new File(tmpDir, "copyRepo1")).exists().isFile(); + } + + expectedRepo = "common/mirror"; + try (Git git = cloneRepo(expectedRepo, createRandomSubDir())) { + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertTag(git, "someTag"); + assertBranch(git, "someBranch"); + } + + expectedRepo = "common/mirrorWithBranchRef"; + try (Git git = cloneRepo(expectedRepo, createRandomSubDir())) { + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertNoTags(git); + assertOnlyBranch(git, "main"); + } + + expectedRepo = "common/mirrorWithTagRef"; + try (Git git = cloneRepo(expectedRepo, createRandomSubDir())) { + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertTag(git, "someTag"); + assertOnlyBranch(git, "main"); + } + + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("8bc1d1165468359b16d9771d4a9a3df26afc03e8"); + repo.setTarget("common/mirrorWithCommitRef"); + }))); + + RuntimeException exception = assertThrows(RuntimeException.class, () -> install(createContent(config), config)); + assertThat(exception.getMessage()) + .startsWith( + "Mirroring commit references is not supported for content repos at the moment. content repository"); + assertThat(exception.getMessage()) + .endsWith("ref: 8bc1d1165468359b16d9771d4a9a3df26afc03e8"); + + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("", "git-repository-with-branches-tags")); + repo.setType(ContentRepoType.MIRROR); + repo.setRef("8bc1d11"); + repo.setTarget("common/mirrorWithShortCommitRef"); + }))); + + exception = assertThrows(RuntimeException.class, () -> install(createContent(config), config)); + assertThat(exception.getMessage()) + .startsWith( + "Mirroring commit references is not supported for content repos at the moment. content repository"); + assertThat(exception.getMessage()).endsWith("ref: 8bc1d11"); + } + + @Test + void resetCommonRepoToRepo() throws IOException, GitAPIException { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setPath("subPath"); + }) + )); + + String expectedRepo = "common/repo"; + GitRepo repo = scmmRepoProvider.create(expectedRepo, scmManagerMock); + scmManagerMock.initOnceRepo(repo.getRepoTarget()); + install(createContent(config), config); + + String url = repo.getGitRepositoryUrl(); + try (Git git = Git.cloneRepository().setURI(url).setBranch("main").setDirectory(tmpDir).call()) { + verify(repo).createRepositoryAndSetPermission(any(String.class), eq(false)); + + String commitMsg = git.log().call().iterator().next().getFullMessage(); + assertThat(commitMsg).isEqualTo("Initialize content repo " + expectedRepo); + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("copyRepo2"); + assertThat(new File(tmpDir, "copyRepo2")).exists().isFile(); + } + + config.getContent().setRepos(List.of(repository(contentRepo -> { + contentRepo.setUrl(createContentRepo("copyRepo1")); + contentRepo.setRef("main"); + contentRepo.setType(ContentRepoType.COPY); + contentRepo.setTarget("common/repo"); + contentRepo.setOverwriteMode(OverwriteMode.RESET); + }))); + + install(createContent(config), config); + scmManagerMock.clearInitOnce(); + + File folderAfterReset = Files.createTempDirectory("second-cloned-repo").toFile(); + folderAfterReset.deleteOnExit(); + try (Git git2 = Git.cloneRepository().setURI(url).setBranch("main").setDirectory(folderAfterReset).call()) { + assertThat(git2).isNotNull(); + assertThat(Files.readString(new File(folderAfterReset, "file").toPath())).contains("copyRepo1"); + assertThat(new File(folderAfterReset, "copyRepo2").exists()).isFalse(); + } + } + + @Test + void updateCommonRepoTest() throws IOException, GitAPIException { + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + }))); + + scmmApiClient.mockRepoApiBehaviour(); + install(createContent(config), config); + + String expectedRepo = "common/repo"; + GitRepo repo = scmmRepoProvider.create(expectedRepo, new ScmManagerProviderMock()); + String url = repo.getGitRepositoryUrl(); + + try (Git git = Git.cloneRepository().setURI(url).setBranch("main").setDirectory(tmpDir).call()) { + verify(repo).createRepositoryAndSetPermission(any(String.class), eq(false)); + + String commitMsg = git.log().call().iterator().next().getFullMessage(); + assertThat(commitMsg).isEqualTo("Initialize content repo " + expectedRepo); + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("copyRepo1"); + assertThat(new File(tmpDir, "copyRepo1")).exists().isFile(); + } + + config.getContent().setRepos(List.of(repository(contentRepo -> { + contentRepo.setUrl(createContentRepo("copyRepo2")); + contentRepo.setType(ContentRepoType.COPY); + contentRepo.setTarget("common/repo"); + contentRepo.setPath("subPath"); + contentRepo.setOverwriteMode(OverwriteMode.UPGRADE); + }))); + + install(createContent(config), config); + + File folderAfterReset = Files.createTempDirectory("second-cloned-repo").toFile(); + folderAfterReset.deleteOnExit(); + try (Git git2 = Git.cloneRepository().setURI(url).setBranch("main").setDirectory(folderAfterReset).call()) { + assertThat(git2).isNotNull(); + assertThat(Files.readString(new File(folderAfterReset, "file").toPath())).contains("copyRepo2"); + assertThat(new File(folderAfterReset, "copyRepo2").exists()).isTrue(); + } + } + + @Test + void initCommonRepoExpectUnchangedRepo() throws IOException, GitAPIException { + config.getContent().setRepos(List.of( + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + }), + repository(repo -> { + repo.setUrl(createContentRepo("copyRepo2")); + repo.setType(ContentRepoType.COPY); + repo.setTarget("common/repo"); + repo.setPath("subPath"); + }) + )); + + String expectedRepo = "common/repo"; + GitRepo repo = scmmRepoProvider.create(expectedRepo, scmManagerMock); + scmManagerMock.initOnceRepo(repo.getRepoTarget()); + install(createContent(config), config); + + String url = repo.getGitRepositoryUrl(); + try (Git git = Git.cloneRepository().setURI(url).setBranch("main").setDirectory(tmpDir).call()) { + verify(repo).createRepositoryAndSetPermission(any(String.class), eq(false)); + + String commitMsg = git.log().call().iterator().next().getFullMessage(); + assertThat(commitMsg).isEqualTo("Initialize content repo " + expectedRepo); + assertThat(Files.readString(new File(tmpDir, "file").toPath())).contains("copyRepo2"); + assertThat(new File(tmpDir, "copyRepo2")).exists().isFile(); + } + + config.getContent().setRepos(List.of(repository(contentRepo -> { + contentRepo.setUrl(createContentRepo("copyRepo1")); + contentRepo.setRef("main"); + contentRepo.setType(ContentRepoType.COPY); + contentRepo.setTarget("common/repo"); + contentRepo.setOverwriteMode(OverwriteMode.INIT); + }))); + + install(createContent(config), config); + scmManagerMock.clearInitOnce(); + + File folderAfterReset = Files.createTempDirectory("second-cloned-repo").toFile(); + folderAfterReset.deleteOnExit(); + try (Git git = Git.cloneRepository().setURI(url).setBranch("main").setDirectory(folderAfterReset).call()) { + assertThat(git).isNotNull(); + assertThat(Files.readString(new File(folderAfterReset, "file").toPath())).contains("copyRepo2"); + assertThat(new File(folderAfterReset, "copyRepo2").exists()).isTrue(); + } + } + + @Test + void ensureJenkinsJobWillBeCreated() { + config.getJenkins().setActive(true); + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setCreateJenkinsJob(true); + repo.setTarget("common/repo"); + }))); + scmmApiClient.mockRepoApiBehaviour(); + + install(createContent(config), config); + + assertThat(jenkins.createdJobs).containsExactly("common/common"); + } + + @Test + void ensureJenkinsJobCreationWillBeIgnored() { + config.getJenkins().setActive(true); + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setCreateJenkinsJob(false); + repo.setTarget("common/repo"); + }))); + scmmApiClient.mockRepoApiBehaviour(); + + install(createContent(config), config); + + assertThat(jenkins.createdJobs).isEmpty(); + } + + @Test + void ensureJenkinsJobWillNotBeCreatedIfJenkinsIsNotEnabled() { + config.getJenkins().setActive(false); + config.getContent().setRepos(List.of(repository(repo -> { + repo.setUrl(createContentRepo("copyRepo1")); + repo.setRef("main"); + repo.setType(ContentRepoType.COPY); + repo.setCreateJenkinsJob(true); + repo.setTarget("common/repo"); + }))); + scmmApiClient.mockRepoApiBehaviour(); + + install(createContent(config), config); + + assertThat(jenkins.createdJobs).isEmpty(); + } + + @Test + void deployHelmReleasesFromContentSkipsWhenHelmReleasesMissingOrEmpty() { + ContentLoaderForTest contentLoader = createContent(config); + install(contentLoader, config); + + assertThat(contentLoader.deployCalls).isEmpty(); + } + + @Test + void deployHelmReleasesFromContentCallsDeployHelmChartWithValuesPathAndHelmConfig() throws IOException { + Path valuesFile = Files.createTempFile("harbor-values-", ".yaml"); + Files.writeString( + valuesFile, """ + expose: + type: ingress + """ + ); + + Config cfg = Config.fromMap(Map.of( + "content", Map.of( + "helmReleases", List.of(Map.of( + "name", "harbor", + "repoURL", "https://helm.goharbor.io", + "chart", "harbor", + "version", "1.18.2", + "namespace", "my-prefix-harbor", + "releaseName", "harbor", + "valuesPath", valuesFile.toString() + )) + ) + )); + + ContentLoaderForTest contentLoader = createContent(cfg); + install(contentLoader, cfg); + + assertThat(contentLoader.deployCalls).hasSize(1); + DeployCall call = contentLoader.deployCalls.get(0); + + assertThat(call.featureName).isEqualTo("harbor"); + assertThat(call.releaseName).isEqualTo("harbor"); + assertThat(call.namespace).isEqualTo("my-prefix-harbor"); + assertThat(call.valuesPath).isNotBlank(); + assertThat(Path.of(call.valuesPath).toFile()).exists(); + assertThat(call.helmConfig.repoURL()).isEqualTo("https://helm.goharbor.io"); + assertThat(call.helmConfig.chart()).isEqualTo("harbor"); + assertThat(call.helmConfig.version()).isEqualTo("1.18.2"); + assertThat(call.config).isSameAs(cfg); + } + + @Test + void deployHelmReleasesFromContentReadsValuesFileAndInlineValuesOverrideFileValues(@TempDir Path tempDir) + throws IOException { + Path valuesFile = tempDir.resolve("harbor-values.yaml"); + Files.writeString( + valuesFile, """ + replicas: 1 + service: + type: ClusterIP + """ + ); + + Config cfg = Config.fromMap(Map.of( + "content", Map.of( + "helmReleases", List.of(Map.ofEntries( + Map.entry("name", "harbor"), + Map.entry("repoURL", "https://helm.goharbor.io"), + Map.entry("chart", "harbor"), + Map.entry("version", "1.18.2"), + Map.entry("namespace", "my-prefix-harbor"), + Map.entry("releaseName", "harbor"), + Map.entry("valuesPath", valuesFile.toString()), + Map.entry( + "values", Map.of( + "replicas", 2, + "service", Map.of("type", "NodePort") + ) + ) + )) + ) + )); + + ContentLoaderForTest contentLoader = createContent(cfg); + install(contentLoader, cfg); + + assertThat(contentLoader.deployCalls).hasSize(1); + DeployCall call = contentLoader.deployCalls.get(0); + Path mergedTemp = Path.of(call.valuesPath); + assertThat(mergedTemp).exists(); + + Map mergedYaml = readYaml(mergedTemp.toFile()); + assertThat(mergedYaml.get("replicas")).isEqualTo(2); + assertThat(((Map) mergedYaml.get("service")).get("type")).isEqualTo("NodePort"); + } + + @Test + void deployHelmReleasesFromContentUsesValuesFileWhenInlineValuesAreEmpty(@TempDir Path tempDir) + throws IOException { + Path valuesFile = tempDir.resolve("values.yaml"); + Files.writeString( + valuesFile, """ + replicas: 1 + """ + ); + + Config cfg = Config.fromMap(Map.of( + "content", Map.of( + "helmReleases", List.of(Map.of( + "name", "elasticsearch", + "repoURL", "https://helm.elastic.co", + "chart", "elasticsearch", + "version", "8.5.1", + "namespace", "my-prefix-elasticsearch", + "valuesPath", valuesFile.toString() + )) + ) + )); + + ContentLoaderForTest contentLoader = createContent(cfg); + install(contentLoader, cfg); + + assertThat(contentLoader.deployCalls).hasSize(1); + DeployCall call = contentLoader.deployCalls.get(0); + Path mergedTemp = Path.of(call.valuesPath); + assertThat(mergedTemp).exists(); + + Map mergedYaml = readYaml(mergedTemp.toFile()); + assertThat(mergedYaml.get("replicas")).isEqualTo(1); + } + + @Test + void deployHelmReleasesFromContentUsesInlineValuesWhenNoHelmValuesPathIsSet() throws IOException { + Config cfg = Config.fromMap(Map.of( + "content", Map.of( + "helmReleases", List.of(Map.of( + "name", "elasticsearch", + "repoURL", "https://helm.elastic.co", + "chart", "elasticsearch", + "version", "8.5.1", + "namespace", "my-prefix-elasticsearch", + "values", Map.of("replicas", 2) + )) + ) + )); + + ContentLoaderForTest contentLoader = createContent(cfg); + install(contentLoader, cfg); + + assertThat(contentLoader.deployCalls).hasSize(1); + DeployCall call = contentLoader.deployCalls.get(0); + Path mergedTemp = Path.of(call.valuesPath); + assertThat(mergedTemp).exists(); + + Map mergedYaml = readYaml(mergedTemp.toFile()); + assertThat(mergedYaml.get("replicas")).isEqualTo(2); + } + + @Test + void deployHelmReleasesFromContentDefaultsChartVersionToWildcardWhenMissing() { + Config cfg = Config.fromMap(Map.of( + "content", Map.of( + "helmReleases", List.of(Map.of( + "name", "harbor", + "repoURL", "https://helm.goharbor.io", + "chart", "harbor", + "version", " ", + "namespace", "my-prefix-harbor", + "releaseName", "harbor", + "values", Map.of("foo", "bar") + )) + ) + )); + + ContentLoaderForTest contentLoader = createContent(cfg); + install(contentLoader, cfg); + + assertThat(contentLoader.deployCalls).hasSize(1); + DeployCall call = contentLoader.deployCalls.get(0); + assertThat(call.helmConfig.version()).isEqualTo("*"); + } + + static String createContentRepo() { + return createContentRepo("", "git-repository"); + } + + static String createContentRepo(String initPath) { + return createContentRepo(initPath, "git-repository"); + } + + static String createContentRepo(String initPath, String baseBareRepo) { + try { + File bareRepoDir = Files.createTempDirectory("gitops-playground-test-content-repo").toFile(); + bareRepoDir.deleteOnExit(); + foldersToDelete.add(bareRepoDir); + + FileUtils.copyDirectory( + new File(System.getProperty("user.dir") + + "/src/test/resources/com/cloudogu/gitops/utils/data/" + baseBareRepo + "/"), + bareRepoDir + ); + String bareRepoUri = "file://" + bareRepoDir.getAbsolutePath(); + log.debug("Repo {}: bare repo {}", initPath, bareRepoUri); + + if (!initPath.isEmpty()) { + File tempRepo = Files.createTempDirectory("gitops-playground-temp-repo").toFile(); + tempRepo.deleteOnExit(); + foldersToDelete.add(tempRepo); + log.debug("Repo {}: cloned bare repo to {}", initPath, tempRepo); + + try (Git git = Git.cloneRepository() + .setURI(bareRepoUri) + .setBranch("main") + .setDirectory(tempRepo) + .call()) { + + FileUtils.copyDirectory( + new File(System.getProperty("user.dir") + + "/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/" + initPath), + tempRepo + ); + + git.add().addFilepattern(".").call(); + SystemReader.getInstance().getUserConfig().clear(); + git.commit().setMessage("Initialize with " + initPath).call(); + git.push().call(); + tempRepo.delete(); + } + } + + return bareRepoUri; + } catch (IOException | GitAPIException | ConfigInvalidException e) { + throw new IllegalStateException("Failed to create test content repository", e); + } + } + + private Map parseYaml(String path) throws IOException { + return readYaml(new File(path)); + } + + private void assertRegistrySecrets(String regUser, String regPw) { + } + + private ContentLoaderForTest createContent(Config contentConfig) { + return new ContentLoaderForTest( + contentConfig, + k8sClient, + credentialsResolver, + scmmRepoProvider, + jenkins, + gitHandler, + fileSystemUtils, + deployer + ); + } + + private boolean install(ContentLoaderForTest contentLoader, Config contentConfig) { + return contentLoader.execute(new ContextBuilder(contentConfig).build(), repositoryWorkspace); + } + + private List cloneContentRepos( + ContentLoaderForTest contentLoader, + Config contentConfig) throws Exception { + return contentLoader.cloneContentRepos(new ContextBuilder(contentConfig).build()); + } + + private static Map parseActualYaml(File pathToYamlFile) throws IOException { + return readYaml(pathToYamlFile); + } + + private static Map readYaml(File file) throws IOException { + return YAML_MAPPER.readValue(file, YAML_MAP_TYPE); + } + + private static String findRoot(List repos) { + return new File(repos.get(0).getClonedContentRepo().getParent()).getParent(); + } + + Git cloneRepo(String expectedRepo, File repoFolder) throws GitAPIException { + GitRepo repo = scmmRepoProvider.create(expectedRepo, new ScmManagerProviderMock()); + String url = repo.getGitRepositoryUrl(); + + Git git = Git.cloneRepository() + .setURI(url) + .setBranch("main") + .setDirectory(repoFolder) + .call(); + git.getRepository().getConfig().setBoolean("gc", null, "autoDetach", false); + return git; + } + + private File createRandomSubDir() { + return createRandomSubDir(""); + } + + private File createRandomSubDir(String prefix) { + String directoryName = (prefix.isEmpty() ? "" : prefix + "-") + System.currentTimeMillis(); + File randomDir = tmpDir.toPath().resolve(directoryName).toFile(); + randomDir.mkdirs(); + return randomDir; + } + + void assertTagAndReadme(String repo, String expectedTag, String expectedReadmeContent) + throws GitAPIException, IOException { + File repoFolder = createRandomSubDir(); + try (Git git = cloneRepo(repo, repoFolder)) { + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertTag(git, expectedTag); + + git.checkout().setName(expectedTag).call(); + assertThat(new File(repoFolder, "README.md")).exists().isFile(); + assertThat(Files.readString(new File(repoFolder, "README.md").toPath())).contains(expectedReadmeContent); + } + } + + void assertBranchAndReadme(String repo, String expectedBranch, String expectedReadmeContent) + throws GitAPIException, IOException { + File repoFolder = createRandomSubDir(); + try (Git git = cloneRepo(repo, repoFolder)) { + git.fetch().setRefSpecs("refs/*:refs/*").call(); + assertBranch(git, expectedBranch); + + git.checkout().setName(expectedBranch).call(); + assertThat(new File(repoFolder, "README.md")).exists().isFile(); + assertThat(Files.readString(new File(repoFolder, "README.md").toPath())).contains(expectedReadmeContent); + } + } + + private static void assertOnlyBranch(Git git, String branch) throws GitAPIException { + List branches = assertBranch(git, branch); + List otherBranches = branches.stream() + .filter(ref -> !ref.getName().contains(branch)) + .toList(); + + assertThat(otherBranches) + .withFailMessage( + "More than the expected branch main found. Available branches: %s", + otherBranches.stream().map(Ref::getName).toList() + ) + .hasSize(0); + } + + private static void assertNoTags(Git git) throws GitAPIException { + List tags = git.tagList().call(); + assertThat(tags) + .withFailMessage( + "No tags in mirrored repo with ref expected. Available tags: %s", + tags.stream().map(Ref::getName).toList() + ) + .hasSize(0); + } + + private static List assertBranch(Git git, String someBranch) throws GitAPIException { + List branches = git.branchList().call(); + assertThat(branches.stream() + .filter(ref -> ref.getName().equals("refs/heads/" + someBranch)) + .toList()) + .withFailMessage( + "Branch '%s' not found in git repository. Available branches: %s", + someBranch, + branches.stream().map(Ref::getName).toList() + ) + .hasSize(1); + return branches; + } + + private static void assertTag(Git git, String expectedTag) throws GitAPIException { + List tags = git.tagList().call(); + assertThat(tags.stream() + .filter(ref -> ref.getName().equals("refs/tags/" + expectedTag)) + .toList()) + .withFailMessage( + "Tag '%s' not found in git repository. Available tags: %s", + expectedTag, + tags.stream().map(Ref::getName).toList() + ) + .hasSize(1); + } + + private static Config createConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("foo-"); + + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setUrl(""); + config.getScm().setScmManager(scmManager); + + config.getRegistry().setUrl("reg-url"); + config.getRegistry().setPath("reg-path"); + config.getRegistry().setUsername("reg-user"); + config.getRegistry().setPassword("reg-pw"); + config.getRegistry().setCreateImagePullSecrets(false); + return config; + } + + private static ContentLoader.RepoCoordinate repoCoordinate(String namespace, String repoName) { + ContentLoader.RepoCoordinate coordinate = new ContentLoader.RepoCoordinate(); + coordinate.setNamespace(namespace); + coordinate.setRepoName(repoName); + return coordinate; + } + + private static Config.ContentSchema.ContentRepositorySchema repository( + Consumer configurator) { + Config.ContentSchema.ContentRepositorySchema repository = + new Config.ContentSchema.ContentRepositorySchema(); + configurator.accept(repository); + return repository; + } + + private static Object readPrivateField(Object target, String fieldName) throws ReflectiveOperationException { + Field field = target.getClass().getDeclaredField(fieldName); + field.setAccessible(true); + return field.get(target); + } + + class JenkinsForTest extends Jenkins { + + private final List createdJobs = new ArrayList<>(); + + JenkinsForTest() { + super( + null, + null, + null, + null, + null, + null, + null, + null, + null, + null, + null, + null, + new JenkinsToolConfigMapper(config), + null, + null, + null + ); + } + + @Override + public void createJenkinsjob(String namespace, String repoName) { + createdJobs.add(namespace + "/" + repoName); + } + } + + class ContentLoaderForTest extends ContentLoader { + + private final Config contentConfig; + final List deployCalls = new ArrayList<>(); + CloneCommand cloneSpy; + + ContentLoaderForTest( + Config config, + K8sClient k8sClient, + CredentialsResolver credentialsResolver, + GitRepoFactory repoProvider, + Jenkins jenkins, + GitHandler gitHandler, + FileSystemUtils fileSystemUtils, + Deployer deployer) { + super(config, k8sClient, credentialsResolver, repoProvider, jenkins, gitHandler, fileSystemUtils, deployer); + this.contentConfig = config; + } + + List cloneContentRepos(DeploymentContext context) throws Exception { + this.context = context; + return super.cloneContentRepos(); + } + + @Override + protected void deployHelmChart( + String featureName, + String releaseName, + String namespace, + HelmChartConfig helmConfig, + String helmValuesTemplatePath, + DeploymentContext context, + boolean initByHelm) { + DeployCall call = new DeployCall(); + call.featureName = featureName; + call.releaseName = releaseName; + call.namespace = namespace; + call.helmConfig = helmConfig; + call.valuesPath = helmValuesTemplatePath; + call.config = contentConfig; + call.initByHelm = initByHelm; + deployCalls.add(call); + } + + @Override + protected CloneCommand gitClone() { + return cloneSpy = spy(super.gitClone().setNoCheckout(true)); + } + } + + static class DeployCall { + String featureName; + String releaseName; + String namespace; + HelmChartConfig helmConfig; + String valuesPath; + Config config; + boolean initByHelm; + } +} diff --git a/src/test/java/com/cloudogu/gitops/application/context/ContextBuilderTest.java b/src/test/java/com/cloudogu/gitops/application/context/ContextBuilderTest.java new file mode 100644 index 000000000..b60e4c50a --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/context/ContextBuilderTest.java @@ -0,0 +1,49 @@ +package com.cloudogu.gitops.application.context; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class ContextBuilderTest { + + @Test + void buildsDefaultDeploymentContextFromConfig() { + Config config = new Config(); + + DeploymentContext context = new ContextBuilder(config).build(); + + assertThat(context.getTenantMode()).isEqualTo(DeploymentContext.TenantMode.SINGLE_TENANT); + assertThat(context.isSingleTenant()).isTrue(); + assertThat(context.isMultiTenant()).isFalse(); + assertThat(context.getScmManagerDeploymentMode()).isEqualTo(DeploymentContext.ScmManagerDeploymentMode.EXTERNAL); + assertThat(context.isInternalScmManager()).isFalse(); + assertThat(context.isExternalScmManager()).isTrue(); + assertThat(context.isAirgapped()).isFalse(); + assertThat(context.getClusterDistribution()).isEqualTo(DeploymentContext.ClusterDistribution.KUBERNETES); + assertThat(context.isOpenshift()).isFalse(); + } + + @Test + void buildsDerivedDeploymentContextValuesFromConfig() { + Config config = new Config(); + config.getMultiTenant().setUseDedicatedInstance(true); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setInternal(true); + config.getScm().setScmManager(scmManager); + config.getApplication().setMirrorRepos(true); + config.getApplication().setOpenshift(true); + + DeploymentContext context = new ContextBuilder(config).build(); + + assertThat(context.getTenantMode()).isEqualTo(DeploymentContext.TenantMode.MULTI_TENANT); + assertThat(context.isMultiTenant()).isTrue(); + assertThat(context.getScmManagerDeploymentMode()).isEqualTo(DeploymentContext.ScmManagerDeploymentMode.INTERNAL); + assertThat(context.isInternalScmManager()).isTrue(); + assertThat(context.isExternalScmManager()).isFalse(); + assertThat(context.isAirgapped()).isTrue(); + assertThat(context.getClusterDistribution()).isEqualTo(DeploymentContext.ClusterDistribution.OPENSHIFT); + assertThat(context.isOpenshift()).isTrue(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/application/credentials/CredentialsResolverTest.java b/src/test/java/com/cloudogu/gitops/application/credentials/CredentialsResolverTest.java new file mode 100644 index 000000000..46125d3d1 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/credentials/CredentialsResolverTest.java @@ -0,0 +1,135 @@ +package com.cloudogu.gitops.application.credentials; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +class CredentialsResolverTest { + + private final K8sClient k8sClient = mock(K8sClient.class); + private final CredentialsResolver resolver = new CredentialsResolver(k8sClient); + + @Test + void returnsFallbackCredentialsWithoutSecretReference() { + ResolvedCredentials resolved = resolver.resolve(null, "plain-user", "plain-password"); + + assertThat(resolved.username()).isEqualTo("plain-user"); + assertThat(resolved.password()).isEqualTo("plain-password"); + verifyNoInteractions(k8sClient); + } + + @Test + void returnsFallbackCredentialsForEmptySecretReference() { + ResolvedCredentials resolved = resolver.resolve(new Credentials(), "plain-user", "plain-password"); + + assertThat(resolved.username()).isEqualTo("plain-user"); + assertThat(resolved.password()).isEqualTo("plain-password"); + verifyNoInteractions(k8sClient); + } + + @Test + void resolvesCredentialsFromSecretWithoutMutatingReference() { + Credentials reference = secretReference(); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))) + .thenReturn(new Credentials("secret-user", "secret-password")); + + ResolvedCredentials resolved = resolver.resolve(reference, "plain-user", "plain-password"); + + assertThat(resolved.username()).isEqualTo("secret-user"); + assertThat(resolved.password()).isEqualTo("secret-password"); + assertThat(reference.getUsername()).isNull(); + assertThat(reference.getPassword()).isNull(); + + ArgumentCaptor captor = ArgumentCaptor.forClass(Credentials.class); + verify(k8sClient).getCredentialsFromSecret(captor.capture()); + Credentials effectiveReference = captor.getValue(); + assertThat(effectiveReference).isNotSameAs(reference); + assertThat(effectiveReference.getUsername()).isEqualTo("plain-user"); + assertThat(effectiveReference.getPassword()).isNull(); + assertThat(effectiveReference.getSecretName()).isEqualTo("tool-credentials"); + assertThat(effectiveReference.getSecretNamespace()).isEqualTo("gop-job"); + assertThat(effectiveReference.getUsernameKey()).isEqualTo("custom-user"); + assertThat(effectiveReference.getPasswordKey()).isEqualTo("custom-password"); + } + + @Test + void resolvesImmutableSecretReference() { + CredentialsReference reference = new CredentialsReference( + "tool-credentials", + "gop-job", + "custom-user", + "custom-password" + ); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))) + .thenReturn(new Credentials("secret-user", "secret-password")); + + ResolvedCredentials resolved = resolver.resolveReference(reference, "plain-user", "plain-password"); + + assertThat(resolved.username()).isEqualTo("secret-user"); + assertThat(resolved.password()).isEqualTo("secret-password"); + } + + @Test + void usesFallbackUsernameWhenSecretDoesNotProvideOne() { + Credentials reference = secretReference(); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))) + .thenAnswer(invocation -> { + Credentials effectiveReference = invocation.getArgument(0); + return new Credentials(effectiveReference.getUsername(), "secret-password"); + }); + + ResolvedCredentials resolved = resolver.resolve(reference, "oauth2.0", "plain-password"); + + assertThat(resolved.username()).isEqualTo("oauth2.0"); + assertThat(resolved.password()).isEqualTo("secret-password"); + } + + @Test + void rejectsSecretReferenceWithoutNamespace() { + Credentials reference = new Credentials(); + reference.setSecretName("tool-credentials"); + + assertThatThrownBy(() -> resolver.resolve(reference, "plain-user", "plain-password")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessage("Kubernetes Secret credentials require both secretName and secretNamespace"); + verifyNoInteractions(k8sClient); + } + + @Test + void rejectsSecretReferenceWithoutName() { + Credentials reference = new Credentials(); + reference.setSecretNamespace("gop-job"); + + assertThatThrownBy(() -> resolver.resolve(reference, "plain-user", "plain-password")) + .isInstanceOf(IllegalArgumentException.class) + .hasMessage("Kubernetes Secret credentials require both secretName and secretNamespace"); + verifyNoInteractions(k8sClient); + } + + @Test + void doesNotExposePasswordInToString() { + ResolvedCredentials resolved = new ResolvedCredentials("user", "do-not-log-me"); + + assertThat(resolved.toString()) + .contains("user", "") + .doesNotContain("do-not-log-me"); + } + + private static Credentials secretReference() { + Credentials reference = new Credentials(); + reference.setSecretName("tool-credentials"); + reference.setSecretNamespace("gop-job"); + reference.setUsernameKey("custom-user"); + reference.setPasswordKey("custom-password"); + return reference; + } +} diff --git a/src/test/java/com/cloudogu/gitops/application/orchestration/DeploymentOrchestratorTest.java b/src/test/java/com/cloudogu/gitops/application/orchestration/DeploymentOrchestratorTest.java new file mode 100644 index 000000000..e88cf7945 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/orchestration/DeploymentOrchestratorTest.java @@ -0,0 +1,41 @@ +package com.cloudogu.gitops.application.orchestration; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.tools.common.AbstractTool; +import org.junit.jupiter.api.Test; +import org.mockito.InOrder; + +import java.util.List; + +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class DeploymentOrchestratorTest { + + @Test + void deploysEnabledToolsInConfiguredOrderWithContextAndWorkspace() { + DeploymentContext context = new ContextBuilder(new Config()).build(); + RepositoryWorkspace workspace = new RepositoryWorkspace(mock(GitRepo.class)); + AbstractTool firstTool = mock(AbstractTool.class); + AbstractTool secondTool = mock(AbstractTool.class); + AbstractTool disabledTool = mock(AbstractTool.class); + + when(firstTool.isEnabled(context)).thenReturn(true); + when(secondTool.isEnabled(context)).thenReturn(true); + + new DeploymentOrchestrator(List.of(firstTool, disabledTool, secondTool)).deployTools(context, workspace); + + InOrder order = inOrder(firstTool, secondTool); + order.verify(firstTool).execute(context, workspace); + order.verify(secondTool).execute(context, workspace); + + verify(disabledTool, never()).execute(context, workspace); + } +} diff --git a/src/test/java/com/cloudogu/gitops/application/orchestration/GitHandlerTest.java b/src/test/java/com/cloudogu/gitops/application/orchestration/GitHandlerTest.java new file mode 100644 index 000000000..b08fb2b00 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/orchestration/GitHandlerTest.java @@ -0,0 +1,442 @@ +package com.cloudogu.gitops.application.orchestration; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.GitlabMock; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.utils.NetworkingUtils; +import org.junit.jupiter.api.Test; + +import java.net.URI; +import java.net.URISyntaxException; +import java.util.LinkedHashMap; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +class GitHandlerTest { + + private static Config config() { + return config(Map.of()); + } + + private static Config config(Map overrides) { + Map base = new LinkedHashMap<>(); + base.put("application", Map.of("namePrefix", "")); + base.put( + "scm", Map.of( + "scmProviderType", ScmProviderType.SCM_MANAGER, + "scmManager", Map.of("internal", true), + "gitlab", Map.of("url", "") + ) + ); + base.put( + "multiTenant", Map.of( + "scmManager", Map.of("url", ""), + "gitlab", Map.of("url", ""), + "useDedicatedInstance", false + ) + ); + + Map merged = deepMerge(base, overrides); + return Config.fromMap(merged); + } + + @SuppressWarnings("unchecked") + private static Map deepMerge(Map left, Map right) { + Map out = new LinkedHashMap<>(left); + + right.forEach((key, value) -> { + Object leftValue = left.get(key); + if (value instanceof Map valueMap && leftValue instanceof Map leftMap) { + out.put( + key, + deepMerge((Map) leftMap, (Map) valueMap) + ); + } else { + out.put(key, value); + } + }); + + return out; + } + + private static GitHandler handler(Config config) { + return handler(config, mock(K8sClient.class)); + } + + private static GitHandler handler(Config config, K8sClient k8sClient) { + return new GitHandler( + k8sClient, + mock(NetworkingUtils.class), + config, + new CredentialsResolver(k8sClient) + ); + } + + private static DeploymentContext context(Config config) { + return new ContextBuilder(config).build(); + } + + // ---------- validate() ------------------------------------------------------------ + + @Test + void validateScmManagerSelectedAndGitopsUsernameReceivesNamePrefix() { + Config config = config(Map.of( + "application", Map.of("namePrefix", "fv40-"), + "scm", Map.of( + "scmManager", Map.of( + "url", "https://scmm.example.com/scm", + "internal", true + ) + ) + )); + + GitHandler gitHandler = handler(config); + + gitHandler.validate(); + + assertEquals(ScmProviderType.SCM_MANAGER, config.getScm().getScmProviderType()); + assertEquals("fv40-gitops", config.getScm().getScmManager().getGitOpsUsername()); + } + + @Test + void validateGitLabAcceptsKubernetesSecretCredentials() { + Config config = config(Map.of( + "scm", Map.of( + "scmProviderType", ScmProviderType.GITLAB, + "gitlab", Map.of( + "url", "https://gitlab.example.com", + "parentGroupId", "123", + "credentials", Map.of( + "secretName", "gitlab-credentials", + "secretNamespace", "gop-job" + ) + ) + ) + )); + K8sClient k8sClient = mock(K8sClient.class); + GitHandler gitHandler = handler(config, k8sClient); + + assertDoesNotThrow(gitHandler::validate); + + assertEquals(ScmProviderType.GITLAB, config.getScm().getScmProviderType()); + assertNull(config.getScm().getScmManager()); + verifyNoInteractions(k8sClient); + } + + @Test + void validateGitLabChosenProviderSwitchedScmmNulledMissingPatOrParentGroupIdThrows() { + Config config = config(Map.of( + "scm", Map.of("gitlab", Map.of("url", "https://gitlab.example.com")) + )); + + GitHandler gitHandler = handler(config); + + RuntimeException exception = assertThrows(RuntimeException.class, gitHandler::validate); + + assertTrue(exception.getMessage().toLowerCase().contains("gitlab")); + assertEquals(ScmProviderType.GITLAB, config.getScm().getScmProviderType()); + assertNull(config.getScm().getScmManager()); + } + + // ---------- getResourcesScm() ----------------------------------------------------- + + @Test + void getResourcesScmCentralWinsOverTenant() { + GitHandler gitHandler = handler(config()); + + gitHandler.setTenant(mock(GitProvider.class, "tenant")); + gitHandler.setCentral(mock(GitProvider.class, "central")); + + assertSame(gitHandler.getCentral(), gitHandler.getResourcesScm()); + } + + @Test + void getResourcesScmTenantReturnedWhenCentralAbsentThrowsWhenNone() { + GitHandler gitHandler = handler(config()); + + gitHandler.setTenant(mock(GitProvider.class)); + + assertSame(gitHandler.getTenant(), gitHandler.getResourcesScm()); + + gitHandler.setTenant(null); + + IllegalStateException exception = assertThrows( + IllegalStateException.class, + gitHandler::getResourcesScm + ); + + assertTrue(exception.getMessage().contains("No SCM provider")); + } + + // ---------- prepareProviders(): SCM_MANAGER --------------------------------------- + + @Test + void prepareProvidersScmManagerTenantOnlyCreatesTenantProviderOnly() { + Config config = Config.fromMap(Map.of( + "scm", Map.of( + "scmManager", Map.of("internal", true), + "gitlab", Map.of("url", "") + ), + "multiTenant", Map.of("useDedicatedInstance", false) + )); + + ScmManagerProviderMock tenant = new ScmManagerProviderMock(); + GitHandlerForTests gitHandler = new GitHandlerForTests(tenant); + + gitHandler.prepareProviders(context(config)); + + assertEquals("scm-manager", config.getScm().getScmManager().getNamespace()); + + assertSame(tenant, gitHandler.getTenant()); + assertNull(gitHandler.getCentral()); + assertSame(tenant, gitHandler.getResourcesScm()); + } + + @Test + void prepareProvidersScmManagerTenantOnlyDoesNotCreateRepositories() { + Config config = Config.fromMap(Map.of( + "scm", Map.of( + "scmManager", Map.of("internal", true), + "gitlab", Map.of("url", "") + ), + "multiTenant", Map.of("useDedicatedInstance", false) + )); + + ScmManagerProviderMock tenant = new ScmManagerProviderMock(); + GitHandlerForTests gitHandler = new GitHandlerForTests(tenant); + + gitHandler.prepareProviders(context(config)); + + assertTrue(tenant.getCreatedRepos().isEmpty()); + } + + @Test + void prepareProvidersScmManagerDedicatedCreatesTenantAndCentralProviders() { + Config config = config(Map.of( + "application", Map.of("namePrefix", "fv40-"), + "scm", Map.of( + "scmProviderType", ScmProviderType.SCM_MANAGER, + "scmManager", Map.of("internal", true), + "gitlab", Map.of("url", "") + ), + "multiTenant", Map.of( + "useDedicatedInstance", true, + "scmManager", Map.of("url", ""), + "gitlab", Map.of("url", "") + ) + )); + + ScmManagerProviderMock tenant = new ScmManagerProviderMock(); + tenant.setNamePrefix("fv40-"); + ScmManagerProviderMock central = new ScmManagerProviderMock(); + central.setNamePrefix("fv40-"); + GitHandlerForTests gitHandler = new GitHandlerForTests(tenant, central); + + gitHandler.prepareProviders(context(config)); + + assertSame(tenant, gitHandler.getTenant()); + assertSame(central, gitHandler.getCentral()); + assertSame(central, gitHandler.getResourcesScm()); + } + + @Test + void prepareProvidersScmManagerDedicatedDoesNotCreateRepositories() { + Config config = config(Map.of( + "application", Map.of("namePrefix", "fv40-"), + "scm", Map.of( + "scmProviderType", ScmProviderType.SCM_MANAGER, + "scmManager", Map.of("internal", true), + "gitlab", Map.of("url", "") + ), + "multiTenant", Map.of( + "useDedicatedInstance", true, + "scmManager", Map.of("url", ""), + "gitlab", Map.of("url", "") + ) + )); + + ScmManagerProviderMock tenant = new ScmManagerProviderMock(); + tenant.setNamePrefix("fv40-"); + ScmManagerProviderMock central = new ScmManagerProviderMock(); + central.setNamePrefix("fv40-"); + GitHandlerForTests gitHandler = new GitHandlerForTests(tenant, central); + + gitHandler.prepareProviders(context(config)); + + assertTrue(tenant.getCreatedRepos().isEmpty()); + assertTrue(central.getCreatedRepos().isEmpty()); + } + + // ---------- prepareProviders(): GITLAB ------------------------------------------- + + @Test + void prepareProvidersGitlabDedicatedCreatesTenantAndCentralProviders() throws URISyntaxException { + Config config = config(Map.of( + "application", Map.of("namePrefix", "fv40-"), + "scm", Map.of( + "scmProviderType", ScmProviderType.GITLAB, + "gitlab", Map.of( + "url", "https://gitlab.example.com", + "password", "pat", + "parentGroupId", 123 + ), + "scmManager", Map.of("internal", true) + ), + "multiTenant", Map.of( + "useDedicatedInstance", true, + "gitlab", Map.of( + "url", "https://gitlab.example.com", + "password", "pat2", + "parentGroupId", 456 + ), + "scmManager", Map.of("url", "") + ) + )); + + GitlabMock tenant = new GitlabMock(); + tenant.setBase(new URI(config.getScm().getGitlab().getUrl())); + tenant.setNamePrefix("fv40-"); + + GitlabMock central = new GitlabMock(); + central.setBase(new URI(config.getMultiTenant().getGitlab().getUrl())); + central.setNamePrefix("fv40-"); + + GitHandlerForTests gitHandler = new GitHandlerForTests(tenant, central); + + gitHandler.prepareProviders(context(config)); + + assertSame(tenant, gitHandler.getTenant()); + assertSame(central, gitHandler.getCentral()); + assertSame(central, gitHandler.getResourcesScm()); + assertSame(tenant, gitHandler.getTenant()); + assertSame(central, gitHandler.getCentral()); + assertSame(central, gitHandler.getResourcesScm()); + } + + @Test + void prepareProvidersGitlabDedicatedDoesNotCreateRepositories() throws URISyntaxException { + Config config = config(Map.of( + "application", Map.of("namePrefix", "fv40-"), + "scm", Map.of( + "scmProviderType", ScmProviderType.GITLAB, + "gitlab", Map.of( + "url", "https://gitlab.example.com", + "password", "pat", + "parentGroupId", 123 + ), + "scmManager", Map.of("internal", true) + ), + "multiTenant", Map.of( + "useDedicatedInstance", true, + "gitlab", Map.of( + "url", "https://gitlab.example.com", + "password", "pat2", + "parentGroupId", 456 + ), + "scmManager", Map.of("url", "") + ) + )); + + GitlabMock tenant = new GitlabMock(); + tenant.setBase(new URI(config.getScm().getGitlab().getUrl())); + tenant.setNamePrefix("fv40-"); + + GitlabMock central = new GitlabMock(); + central.setBase(new URI(config.getMultiTenant().getGitlab().getUrl())); + central.setNamePrefix("fv40-"); + + GitHandlerForTests gitHandler = new GitHandlerForTests(tenant, central); + + gitHandler.prepareProviders(context(config)); + + assertTrue(tenant.getCreatedRepos().isEmpty()); + assertTrue(central.getCreatedRepos().isEmpty()); + } + + @Test + void prepareProvidersResolvesScmManagerCredentialsWithoutMutatingConfig() { + Config config = config(Map.of( + "scm", Map.of( + "scmProviderType", ScmProviderType.SCM_MANAGER, + "scmManager", Map.of( + "internal", true, + "username", "fallback-user", + "password", "fallback-password", + "credentials", Map.of( + "secretName", "scmm-credentials", + "secretNamespace", "gop-job" + ) + ) + ) + )); + K8sClient k8sClient = mock(K8sClient.class); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))) + .thenReturn(new Credentials("secret-user", "secret-password")); + GitHandler gitHandler = handler(config, k8sClient); + + gitHandler.validate(); + gitHandler.prepareProviders(context(config)); + + assertEquals("secret-user", gitHandler.getTenant().getCredentials().getUsername()); + assertEquals("secret-password", gitHandler.getTenant().getCredentials().getPassword()); + assertEquals("fallback-user", config.getScm().getScmManager().getUsername()); + assertEquals("fallback-password", config.getScm().getScmManager().getPassword()); + assertEquals("scmm-credentials", config.getScm().getScmManager().getCredentials().getSecretName()); + assertNull(config.getScm().getScmManager().getCredentials().getUsername()); + assertNull(config.getScm().getScmManager().getCredentials().getPassword()); + verify(k8sClient).getCredentialsFromSecret(any(Credentials.class)); + } + + @Test + void prepareProvidersUsesGitLabUsernameFallbackWhenSecretContainsOnlyToken() { + Config config = config(Map.of( + "scm", Map.of( + "scmProviderType", ScmProviderType.GITLAB, + "gitlab", Map.of( + "url", "https://gitlab.example.com", + "parentGroupId", "123", + "credentials", Map.of( + "secretName", "gitlab-credentials", + "secretNamespace", "gop-job" + ) + ) + ) + )); + K8sClient k8sClient = mock(K8sClient.class); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))) + .thenAnswer(invocation -> { + Credentials reference = invocation.getArgument(0); + return new Credentials(reference.getUsername(), "secret-token"); + }); + GitHandler gitHandler = handler(config, k8sClient); + + gitHandler.validate(); + gitHandler.prepareProviders(context(config)); + + assertEquals("oauth2.0", gitHandler.getTenant().getCredentials().getUsername()); + assertEquals("secret-token", gitHandler.getTenant().getCredentials().getPassword()); + assertNull(config.getScm().getGitlab().getPassword()); + assertEquals("gitlab-credentials", config.getScm().getGitlab().getCredentials().getSecretName()); + assertNull(config.getScm().getGitlab().getCredentials().getPassword()); + } + +} diff --git a/src/test/java/com/cloudogu/gitops/application/repository/RepositoryProvisioningTest.java b/src/test/java/com/cloudogu/gitops/application/repository/RepositoryProvisioningTest.java new file mode 100644 index 000000000..5b805c68a --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/repository/RepositoryProvisioningTest.java @@ -0,0 +1,347 @@ +package com.cloudogu.gitops.application.repository; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.utils.FileSystemUtils; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class RepositoryProvisioningTest { + + Config config; + + GitRepoFactory gitRepoFactory = mock(GitRepoFactory.class); + GitHandler gitHandler = mock(GitHandler.class); + + GitProvider tenantProvider = mock(GitProvider.class); + GitProvider centralProvider = mock(GitProvider.class); + + GitRepo clusterResourcesRepo; + GitRepo tenantBootstrapRepo; + + @BeforeEach + void setUp() throws GitAPIException, IOException { + config = Config.fromMap(Map.of( + "application", Map.of( + "namePrefix", "", + "mirrorRepos", false, + "openshift", false, + "insecure", false, + "gitName", "Cloudogu", + "gitEmail", "hello@cloudogu.com" + ), + "scm", Map.of( + "scmProviderType", ScmProviderType.SCM_MANAGER, + "scmManager", Map.of("internal", false), + "gitlab", Map.of("url", "") + ), + "multiTenant", Map.of( + "useDedicatedInstance", false, + "scmManager", Map.of("url", ""), + "gitlab", Map.of("url", "") + ) + )); + + doReturn(tenantProvider).when(gitHandler).getTenant(); + doReturn(tenantProvider).when(gitHandler).getResourcesScm(); + + clusterResourcesRepo = createGitRepoSpy("argocd/cluster-resources", tenantProvider); + tenantBootstrapRepo = createGitRepoSpy("argocd/cluster-resources", tenantProvider); + } + + @Test + void provideWorkspaceCreatesSingleInstanceWorkspaceWithClusterResourcesRepositoryOnly() { + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + RepositoryWorkspace workspace = provisioning.provideWorkspace(createDeploymentContext()); + + assertThat(workspace.getClusterResourcesRepository()).isSameAs(clusterResourcesRepo); + assertThat(workspace.hasTenantBootstrapRepository()).isFalse(); + + verify(gitRepoFactory).create(eq("argocd/cluster-resources"), eq(tenantProvider)); + verify(gitHandler).getResourcesScm(); + } + + @Test + void provideWorkspaceCreatesDedicatedWorkspaceWithCentralClusterResourcesAndTenantBootstrapRepository() + throws GitAPIException, IOException { + + config.getMultiTenant().setUseDedicatedInstance(true); + + doReturn(centralProvider).when(gitHandler).getResourcesScm(); + doReturn(tenantProvider).when(gitHandler).getTenant(); + + clusterResourcesRepo = createGitRepoSpy("argocd/cluster-resources", centralProvider); + tenantBootstrapRepo = createGitRepoSpy("argocd/cluster-resources", tenantProvider); + + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(centralProvider))) + .thenReturn(clusterResourcesRepo); + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(tenantBootstrapRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + RepositoryWorkspace workspace = provisioning.provideWorkspace(createDeploymentContext()); + + assertThat(workspace.getClusterResourcesRepository()).isSameAs(clusterResourcesRepo); + assertThat(workspace.getTenantBootstrapRepository()).isSameAs(tenantBootstrapRepo); + assertThat(workspace.hasTenantBootstrapRepository()).isTrue(); + + assertThat(new File(workspace.clusterResourcesRootDir()).getCanonicalPath()) + .isNotEqualTo(new File(workspace.tenantBootstrapRootDir()).getCanonicalPath()); + + verify(gitRepoFactory).create(eq("argocd/cluster-resources"), eq(centralProvider)); + verify(gitRepoFactory).create(eq("argocd/cluster-resources"), eq(tenantProvider)); + } + + @Test + void provideWorkspaceReturnsSameWorkspaceInstanceWhenCalledMultipleTimes() { + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + RepositoryWorkspace firstWorkspace = provisioning.provideWorkspace(createDeploymentContext()); + RepositoryWorkspace secondWorkspace = provisioning.provideWorkspace(createDeploymentContext()); + + assertThat(secondWorkspace).isSameAs(firstWorkspace); + + verify(gitRepoFactory, times(1)).create(eq("argocd/cluster-resources"), eq(tenantProvider)); + } + + @Test + void prepareOnlyPreparesLocalWorkspaceWhenInternalScmManagerMustBeDeployedFirst() throws GitAPIException { + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + config.getScm().getScmManager().setInternal(true); + + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + provisioning.prepare(createDeploymentContext()); + + verify(tenantProvider, never()).createRepository(anyString(), anyString(), anyBoolean()); + verify(clusterResourcesRepo, never()).cloneRepo(); + } + + @Test + void prepareEnsuresAndClonesRepositoriesWhenScmManagerIsExternal() throws GitAPIException { + config.getScm().getScmManager().setInternal(false); + + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + provisioning.prepare(createDeploymentContext()); + + verify(tenantProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for basic cluster-resources", + false + ); + verify(clusterResourcesRepo).cloneRepo(); + } + + @Test + void ensureRemoteRepositoriesExistCreatesClusterResourcesRepositoryInSingleInstanceMode() { + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + provisioning.provideWorkspace(createDeploymentContext()); + provisioning.ensureRemoteRepositoriesExist(); + + verify(tenantProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for basic cluster-resources", + false + ); + } + + @Test + void ensureRemoteRepositoriesExistCreatesBothRepositoriesInDedicatedMode() throws GitAPIException, IOException { + config.getMultiTenant().setUseDedicatedInstance(true); + + doReturn(centralProvider).when(gitHandler).getResourcesScm(); + doReturn(tenantProvider).when(gitHandler).getTenant(); + + clusterResourcesRepo = createGitRepoSpy("argocd/cluster-resources", centralProvider); + tenantBootstrapRepo = createGitRepoSpy("argocd/cluster-resources", tenantProvider); + + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(centralProvider))) + .thenReturn(clusterResourcesRepo); + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(tenantBootstrapRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + provisioning.provideWorkspace(createDeploymentContext()); + provisioning.ensureRemoteRepositoriesExist(); + + verify(centralProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for basic cluster-resources", + false + ); + + verify(tenantProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for tenant bootstrap resources", + false + ); + } + + @Test + void ensureRemoteRepositoriesExistIsIdempotent() { + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + provisioning.provideWorkspace(createDeploymentContext()); + + provisioning.ensureRemoteRepositoriesExist(); + provisioning.ensureRemoteRepositoriesExist(); + + verify(tenantProvider, times(1)).createRepository( + "argocd/cluster-resources", + "GitOps repo for basic cluster-resources", + false + ); + } + + @Test + void publishClusterResourcesRepositoryChangesUsesDefaultMessageWhenNoMessageIsProvided() throws GitAPIException { + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(clusterResourcesRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + provisioning.provideWorkspace(createDeploymentContext()); + + provisioning.publishClusterResourcesRepositoryChanges("argocd"); + + verify(clusterResourcesRepo).commitAndPush("Update argocd resources"); + } + + @Test + void publishFailsWhenWorkspaceHasNotBeenPrepared() { + RepositoryProvisioning provisioning = createProvisioning(); + + assertThatThrownBy(() -> provisioning.publishClusterResourcesRepositoryChanges("argocd")) + .isInstanceOf(IllegalStateException.class) + .hasMessage("Repository workspace must be prepared before repository changes can be published."); + } + + @Test + void dedicatedWorkspaceFailsWhenClusterResourcesAndTenantBootstrapUseSameLocalWorkspace() throws IOException { + config.getMultiTenant().setUseDedicatedInstance(true); + + String sameRootDir = createTempDir("shared-workspace"); + + GitRepo sharedClusterRepo = mock(GitRepo.class); + GitRepo sharedTenantRepo = mock(GitRepo.class); + + sharedClusterRepo.setGitProvider(centralProvider); + sharedTenantRepo.setGitProvider(tenantProvider); + + doReturn("argocd/cluster-resources").when(sharedClusterRepo).getRepoTarget(); + doReturn("argocd/cluster-resources").when(sharedTenantRepo).getRepoTarget(); + doReturn(sameRootDir).when(sharedClusterRepo).getAbsoluteLocalRepoTmpDir(); + doReturn(sameRootDir).when(sharedTenantRepo).getAbsoluteLocalRepoTmpDir(); + + doReturn(centralProvider).when(gitHandler).getResourcesScm(); + doReturn(tenantProvider).when(gitHandler).getTenant(); + + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(centralProvider))) + .thenReturn(sharedClusterRepo); + when(gitRepoFactory.create(eq("argocd/cluster-resources"), eq(tenantProvider))) + .thenReturn(sharedTenantRepo); + + RepositoryProvisioning provisioning = createProvisioning(); + + assertThatThrownBy(() -> provisioning.provideWorkspace(createDeploymentContext())) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("Dedicated Multi-Tenant mode requires separate local workspaces") + .hasMessageContaining(sameRootDir); + } + + @Test + void clusterResourcesRepoTargetReturnsUnprefixedTarget() { + config.getApplication().setNamePrefix("testPrefix-"); + + RepositoryProvisioning provisioning = createProvisioning(); + + assertThat(provisioning.clusterResourcesRepoTarget()).isEqualTo("argocd/cluster-resources"); + } + + private RepositoryProvisioning createProvisioning() { + return new RepositoryProvisioning(gitRepoFactory, gitHandler); + } + + private DeploymentContext createDeploymentContext() { + return new DeploymentContext( + Boolean.TRUE.equals(config.getMultiTenant().getUseDedicatedInstance()) + ? DeploymentContext.TenantMode.MULTI_TENANT + : DeploymentContext.TenantMode.SINGLE_TENANT, + Boolean.TRUE.equals(config.getScm().getScmManager().getInternal()) + ? DeploymentContext.ScmManagerDeploymentMode.INTERNAL + : DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + Boolean.TRUE.equals(config.getApplication().getMirrorRepos()), + Boolean.TRUE.equals(config.getApplication().getOpenshift()) + ? DeploymentContext.ClusterDistribution.OPENSHIFT + : DeploymentContext.ClusterDistribution.KUBERNETES + ); + } + + private GitRepo createGitRepoSpy(String repoTarget, GitProvider gitProvider) throws GitAPIException, IOException { + GitRepo gitRepo = spy(new GitRepo( + config, + gitProvider, + repoTarget, + new FileSystemUtils() + )); + + doNothing().when(gitRepo).cloneRepo(); + doNothing().when(gitRepo).initLocalRepoIfNeeded(); + doNothing().when(gitRepo).checkoutRemoteMainIfLocalMainMissing(); + doNothing().when(gitRepo).commitAndPush(anyString()); + + return gitRepo; + } + + private static String createTempDir(String prefix) throws IOException { + return Files.createTempDirectory(prefix).toFile().getCanonicalPath(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/application/repository/RepositoryWorkspaceTest.java b/src/test/java/com/cloudogu/gitops/application/repository/RepositoryWorkspaceTest.java new file mode 100644 index 000000000..18389caed --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/application/repository/RepositoryWorkspaceTest.java @@ -0,0 +1,322 @@ +package com.cloudogu.gitops.application.repository; + +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.Mockito.anyString; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; + +class RepositoryWorkspaceTest { + + GitRepo clusterResourcesRepository = mock(GitRepo.class); + GitRepo tenantBootstrapRepository = mock(GitRepo.class); + + String clusterResourcesRootDir; + String tenantBootstrapRootDir; + + @BeforeEach + void setUp() throws IOException { + clusterResourcesRootDir = createTempDir("cluster-resources"); + tenantBootstrapRootDir = createTempDir("tenant-bootstrap"); + + doReturn(clusterResourcesRootDir) + .when(clusterResourcesRepository) + .getAbsoluteLocalRepoTmpDir(); + + doReturn(tenantBootstrapRootDir) + .when(tenantBootstrapRepository) + .getAbsoluteLocalRepoTmpDir(); + } + + @Test + void hasTenantBootstrapRepositoryReturnsFalseInSingleInstanceMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + assertThat(workspace.hasTenantBootstrapRepository()).isFalse(); + } + + @Test + void hasTenantBootstrapRepositoryReturnsTrueInDedicatedMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + assertThat(workspace.hasTenantBootstrapRepository()).isTrue(); + } + + @Test + void tenantBootstrapRepositoryOrFailReturnsTenantBootstrapRepositoryWhenAvailable() { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + assertThat(workspace.tenantBootstrapRepositoryOrFail()).isSameAs(tenantBootstrapRepository); + } + + @Test + void tenantBootstrapRepositoryOrFailThrowsInSingleInstanceMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + assertThatThrownBy(workspace::tenantBootstrapRepositoryOrFail) + .isInstanceOf(IllegalStateException.class) + .hasMessage("Tenant bootstrap repository is not available in single-instance mode."); + } + + @Test + void createLocalDirectoriesCreatesClusterResourcesDirectoryStructureInSingleInstanceMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + workspace.createLocalDirectories(); + + assertThat(Path.of(clusterResourcesRootDir)).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps")).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps", "argocd")).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps", "argocd", "applications")).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps", "argocd", "projects")).exists(); + + assertThat(Path.of(tenantBootstrapRootDir, "apps")).doesNotExist(); + } + + @Test + void createLocalDirectoriesCreatesClusterResourcesAndTenantBootstrapDirectoryStructuresInDedicatedMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.createLocalDirectories(); + + assertThat(Path.of(clusterResourcesRootDir)).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps")).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps", "argocd")).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps", "argocd", "applications")).exists(); + assertThat(Path.of(clusterResourcesRootDir, "apps", "argocd", "projects")).exists(); + + assertThat(Path.of(tenantBootstrapRootDir)).exists(); + assertThat(Path.of(tenantBootstrapRootDir, "apps")).exists(); + assertThat(Path.of(tenantBootstrapRootDir, "apps", "argocd")).exists(); + assertThat(Path.of(tenantBootstrapRootDir, "apps", "argocd", "applications")).exists(); + assertThat(Path.of(tenantBootstrapRootDir, "apps", "argocd", "projects")).exists(); + } + + @Test + void cloneRepositoriesClonesOnlyClusterResourcesRepositoryInSingleInstanceMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + workspace.cloneRepositories(); + + verify(clusterResourcesRepository).cloneRepo(); + verifyNoInteractions(tenantBootstrapRepository); + } + + @Test + void cloneRepositoriesClonesClusterResourcesAndTenantBootstrapRepositoriesInDedicatedMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.cloneRepositories(); + + verify(clusterResourcesRepository).cloneRepo(); + verify(tenantBootstrapRepository).cloneRepo(); + } + + @Test + void initLocalRepositoriesIfNeededInitializesOnlyClusterResourcesRepositoryInSingleInstanceMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + workspace.initLocalRepositoriesIfNeeded(); + + verify(clusterResourcesRepository).initLocalRepoIfNeeded(); + verifyNoInteractions(tenantBootstrapRepository); + } + + @Test + void initLocalRepositoriesIfNeededInitializesClusterResourcesAndTenantBootstrapRepositoriesInDedicatedMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.initLocalRepositoriesIfNeeded(); + + verify(clusterResourcesRepository).initLocalRepoIfNeeded(); + verify(tenantBootstrapRepository).initLocalRepoIfNeeded(); + } + + @Test + void clusterResourcesPathMethodsReturnExpectedPaths() { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + assertThat(workspace.clusterResourcesRootDir()).isEqualTo(clusterResourcesRootDir); + assertThat(workspace.clusterResourcesAppsDir()).isEqualTo(Path.of(clusterResourcesRootDir, "apps").toString()); + assertThat(workspace.clusterResourcesArgoCdDir()).isEqualTo(Path.of( + clusterResourcesRootDir, + "apps", + "argocd" + ).toString()); + assertThat(workspace.clusterResourcesApplicationsDir()).isEqualTo(Path.of( + clusterResourcesRootDir, + "apps", + "argocd", + "applications" + ).toString()); + assertThat(workspace.clusterResourcesProjectsDir()).isEqualTo(Path.of( + clusterResourcesRootDir, + "apps", + "argocd", + "projects" + ).toString()); + } + + @Test + void tenantBootstrapPathMethodsReturnExpectedPathsInDedicatedMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + assertThat(workspace.tenantBootstrapRootDir()).isEqualTo(tenantBootstrapRootDir); + assertThat(workspace.tenantBootstrapAppsDir()).isEqualTo(Path.of(tenantBootstrapRootDir, "apps").toString()); + assertThat(workspace.tenantBootstrapArgoCdDir()).isEqualTo(Path.of( + tenantBootstrapRootDir, + "apps", + "argocd" + ).toString()); + assertThat(workspace.tenantBootstrapApplicationsDir()).isEqualTo(Path.of( + tenantBootstrapRootDir, + "apps", + "argocd", + "applications" + ).toString()); + assertThat(workspace.tenantBootstrapProjectsDir()).isEqualTo(Path.of( + tenantBootstrapRootDir, + "apps", + "argocd", + "projects" + ).toString()); + } + + @Test + void tenantBootstrapPathMethodsThrowInSingleInstanceMode() { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + assertThatThrownBy(workspace::tenantBootstrapRootDir) + .isInstanceOf(IllegalStateException.class) + .hasMessage("Tenant bootstrap repository is not available in single-instance mode."); + + assertThatThrownBy(workspace::tenantBootstrapAppsDir) + .isInstanceOf(IllegalStateException.class); + + assertThatThrownBy(workspace::tenantBootstrapArgoCdDir) + .isInstanceOf(IllegalStateException.class); + + assertThatThrownBy(workspace::tenantBootstrapApplicationsDir) + .isInstanceOf(IllegalStateException.class); + + assertThatThrownBy(workspace::tenantBootstrapProjectsDir) + .isInstanceOf(IllegalStateException.class); + } + + @Test + void commitAndPushClusterResourcesChangesCommitsOnlyClusterResourcesRepository() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.commitAndPushClusterResourcesChanges("Update cluster resources"); + + verify(clusterResourcesRepository).commitAndPush("Update cluster resources"); + verify(tenantBootstrapRepository, never()).commitAndPush(anyString()); + } + + @Test + void commitAndPushTenantBootstrapChangesCommitsTenantBootstrapRepositoryWhenAvailable() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.commitAndPushTenantBootstrapChanges("Update tenant bootstrap"); + + verify(tenantBootstrapRepository).commitAndPush("Update tenant bootstrap"); + verify(clusterResourcesRepository, never()).commitAndPush(anyString()); + } + + @Test + void commitAndPushTenantBootstrapChangesThrowsInSingleInstanceMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + assertThatThrownBy(() -> workspace.commitAndPushTenantBootstrapChanges("Update tenant bootstrap")) + .isInstanceOf(IllegalStateException.class) + .hasMessage("Tenant bootstrap repository is not available in single-instance mode."); + + verify(clusterResourcesRepository, never()).commitAndPush(anyString()); + } + + @Test + void commitAndPushClusterResourcesAndTenantBootstrapChangesCommitsOnlyClusterResourcesRepositoryInSingleInstanceMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + workspace.commitAndPushClusterResourcesAndTenantBootstrapChanges("Update resources"); + + verify(clusterResourcesRepository).commitAndPush("Update resources"); + verifyNoInteractions(tenantBootstrapRepository); + } + + @Test + void commitAndPushClusterResourcesAndTenantBootstrapChangesCommitsBothRepositoriesInDedicatedMode() throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.commitAndPushClusterResourcesAndTenantBootstrapChanges("Update resources"); + + verify(clusterResourcesRepository).commitAndPush("Update resources"); + verify(tenantBootstrapRepository).commitAndPush("Update resources"); + } + + @Test + void alignWithRemoteMainIfPresentChecksOutOnlyClusterResourcesRepositoryInSingleInstanceMode() throws GitAPIException, IOException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepository); + + workspace.alignWithRemoteMainIfPresent(); + + verify(clusterResourcesRepository).checkoutRemoteMainIfLocalMainMissing(); + verifyNoInteractions(tenantBootstrapRepository); + } + + @Test + void alignWithRemoteMainIfPresentChecksOutBothRepositoriesInDedicatedMode() throws GitAPIException, IOException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepository, + tenantBootstrapRepository + ); + + workspace.alignWithRemoteMainIfPresent(); + + verify(clusterResourcesRepository).checkoutRemoteMainIfLocalMainMissing(); + verify(tenantBootstrapRepository).checkoutRemoteMainIfLocalMainMissing(); + } + + private static String createTempDir(String prefix) throws IOException { + return Files.createTempDirectory(prefix).toFile().getCanonicalPath(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/cli/ApplicationConfiguratorTest.java b/src/test/java/com/cloudogu/gitops/cli/ApplicationConfiguratorTest.java new file mode 100644 index 000000000..6315c58b4 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/cli/ApplicationConfiguratorTest.java @@ -0,0 +1,835 @@ +package com.cloudogu.gitops.cli; + +import com.cloudogu.gitops.application.content.ContentLoader; +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryProvisioning; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.TestLogger; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.tools.common.CommonToolConfig; +import com.cloudogu.gitops.tools.core.Jenkins; +import com.cloudogu.gitops.tools.core.argocd.ArgoCD; +import com.cloudogu.gitops.tools.core.argocd.ArgoCDToolConfigMapper; +import com.cloudogu.gitops.tools.core.argocd.mode.DeploymentModeFactory; +import com.cloudogu.gitops.utils.FileSystemUtils; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mock; +import org.mockito.Mockito; + +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class ApplicationConfiguratorTest { + + static final String EXPECTED_REGISTRY_URL = "http://my-reg"; + static final int EXPECTED_REGISTRY_INTERNAL_PORT = 33333; + static final Config.VaultMode EXPECTED_VAULT_MODE = Config.VaultMode.DEV; + public static final String EXPECTED_JENKINS_URL = "http://my-jenkins"; + public static final String EXPECTED_SCMM_URL = "http://my-scmm"; + + private ApplicationConfigurator applicationConfigurator; + private FileSystemUtils fileSystemUtils; + private TestLogger testLogger; + private CommonToolConfig commonFeatureConfig; + private ContentLoader featureContent; + private ArgoCD featureArgoCd; + private RepositoryProvisioning repositoryProvisioning; + + @Mock + ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + + Config testConfig = Config.fromMap(Map.of( + "application", Map.of( + "localHelmChartFolder", "someValue", + "namePrefix", "" + ), + "registry", Map.of( + "url", EXPECTED_REGISTRY_URL, + "proxyUrl", "proxy-" + EXPECTED_REGISTRY_URL, + "proxyUsername", "proxy-user", + "proxyPassword", "proxy-pw", + "internalPort", EXPECTED_REGISTRY_INTERNAL_PORT + ), + "jenkins", Map.of("url", EXPECTED_JENKINS_URL), + "scm", Map.of("scmManager", Map.of("url", EXPECTED_SCMM_URL)), + "multiTenant", Map.of("scmManager", Map.of("url", "")), + "features", Map.of("secrets", Map.of("vault", Map.of("mode", EXPECTED_VAULT_MODE))) + )); + + @BeforeEach + void setup() { + fileSystemUtils = new FileSystemUtils(); + applicationConfigurator = configuratorWithEnvironment(Map.of()); + testLogger = new TestLogger(applicationConfigurator.getClass()); + commonFeatureConfig = new CommonToolConfig(); + + K8sClient k8sClient = Mockito.mock(K8sClient.class); + HelmClient helmClient = Mockito.mock(HelmClient.class); + GitRepoFactory gitRepoFactory = Mockito.mock(GitRepoFactory.class); + Deployer deployer = Mockito.mock(Deployer.class); + repositoryProvisioning = Mockito.mock(RepositoryProvisioning.class); + + GitHandler gitHandler = new GitHandlerForTests(scmManagerMock); + DeploymentContext context = new ContextBuilder(testConfig).build(); + + featureContent = Mockito.spy(new ContentLoader( + testConfig, + k8sClient, + new CredentialsResolver(k8sClient), + gitRepoFactory, + Mockito.mock(Jenkins.class), + gitHandler, + fileSystemUtils, + deployer + )); + featureContent.isEnabled(context); + + featureArgoCd = Mockito.spy(new ArgoCD( + k8sClient, + helmClient, + fileSystemUtils, + gitHandler, + new DeploymentModeFactory(), + new ArgoCDToolConfigMapper(testConfig), + new CredentialsResolver(k8sClient) + )); + featureArgoCd.isEnabled(context); + } + + @Test + void correctConfigWithNoProgramArguments() { + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getJenkins().getUrl()).isEqualTo(EXPECTED_JENKINS_URL); + assertThat(actualConfig.getJenkins().getInternal()).isEqualTo(false); + assertThat(actualConfig.getFeatures().getSecrets().getVault().getMode()).isEqualTo(EXPECTED_VAULT_MODE); + + // Dynamic value (depends on vault mode) + assertThat(actualConfig.getFeatures().getSecrets().getActive()).isEqualTo(true); + } + + @Test + void setsConfigApplicationRunningInsideK8s() { + applicationConfigurator = configuratorWithEnvironment(Map.of("KUBERNETES_SERVICE_HOST", "127.0.0.1")); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getApplication().getRunningInsideK8s()).isEqualTo(true); + } + + @Test + void setsJenkinsActiveIfExternalUrlIsSet() { + testConfig.getJenkins().setUrl("external"); + Config actualConfig = applicationConfigurator.initConfig(testConfig); + assertThat(actualConfig.getJenkins().getActive()).isEqualTo(true); + } + + @Test + void leavesJenkinsUrlForScmEmptyIfNotActive() { + testConfig.getJenkins().setUrl(""); + testConfig.getJenkins().setActive(false); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + assertThat(actualConfig.getJenkins().getUrlForScm()).isEmpty(); + } + + @Test + void failsIfMonitoringLocalIsNotSet() { + testConfig.getApplication().setMirrorRepos(true); + testConfig.getApplication().setLocalHelmChartFolder(""); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> commonFeatureConfig.validateConfig(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo( + "Missing config for localHelmChartFolder.\n" + + "Either run inside the official container image or setting env var LOCAL_HELM_CHART_FOLDER='charts' " + + "after running 'scripts/downloadHelmCharts.sh' from the repo" + ); + } + + @Test + void failsIfCreateImagePullSecretsIsUsedWithoutSecrets() { + testConfig.getRegistry().setCreateImagePullSecrets(true); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> applicationConfigurator.initConfig(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo( + "createImagePullSecrets needs to be used with either registry username and password or the readOnly variants" + ); + } + + @Test + void acceptsReadOnlySecretCredentialsForImagePullSecrets() { + testConfig.getRegistry().setCreateImagePullSecrets(true); + testConfig.getRegistry().setReadOnlyCredentials( + new Credentials(null, null, "registry-read-only-credentials", "gop-job") + ); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getRegistry().getReadOnlyCredentials().getSecretName()) + .isEqualTo("registry-read-only-credentials"); + } + + @Test + void acceptsProxySecretCredentials() { + testConfig.getRegistry().setProxyUsername(""); + testConfig.getRegistry().setProxyPassword(""); + testConfig.getRegistry().setProxyCredentials( + new Credentials(null, null, "registry-proxy-credentials", "gop-job") + ); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getRegistry().getProxyCredentials().getSecretName()) + .isEqualTo("registry-proxy-credentials"); + } + + @Test + void failsIfContentRepoIsSetWithoutMandatoryParams() { + Config.ContentSchema.ContentRepositorySchema repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl(""); + testConfig.getContent().setRepos(List.of(repo)); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> featureContent.preConfigInit(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo("content.repos requires a url parameter."); + + repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.COPY); + repo.setTarget("missing_slash"); + testConfig.getContent().setRepos(List.of(repo)); + + exception = assertThrows(RuntimeException.class, () -> featureContent.preConfigInit(testConfig)); + assertThat(exception.getMessage()).isEqualTo( + "content.target needs / to separate namespace/group from repo name. Repo: abc" + ); + } + + @Test + void failsIfCopyRepoMissesTargetParameter() { + Config.ContentSchema.ContentRepositorySchema repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.COPY); + testConfig.getContent().setRepos(List.of(repo)); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> featureContent.preConfigInit(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo( + "content.repos.type COPY requires content.repos.target to be set. Repo: abc" + ); + } + + @Test + void allowsCopyContentRepoTargetingClusterResources() { + Config.ContentSchema.ContentRepositorySchema repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.COPY); + repo.setTarget("argocd/cluster-resources"); + testConfig.getContent().setRepos(List.of(repo)); + + Throwable exception = null; + try { + featureContent.preConfigInit(testConfig); + } catch (Throwable thrown) { + exception = thrown; + } + + assertThat(exception).isNull(); + } + + @Test + void failsIfFolderBasedRepoHasTargetParameter() { + Config.ContentSchema.ContentRepositorySchema repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.FOLDER_BASED); + repo.setTarget("namespace/repo"); + testConfig.getContent().setRepos(List.of(repo)); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> featureContent.preConfigInit(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo( + "content.repos.type FOLDER_BASED does not support target parameter. Repo: abc" + ); + + repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.FOLDER_BASED); + repo.setTargetRef("someRef"); + testConfig.getContent().setRepos(List.of(repo)); + + exception = assertThrows(RuntimeException.class, () -> featureContent.preConfigInit(testConfig)); + assertThat(exception.getMessage()).isEqualTo( + "content.repos.type FOLDER_BASED does not support targetRef parameter. Repo: abc" + ); + } + + @Test + void failsIfMirrorRepoHasInvalidConfiguration() { + Config.ContentSchema.ContentRepositorySchema repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.MIRROR); + testConfig.getContent().setRepos(List.of(repo)); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> featureContent.preConfigInit(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo( + "content.repos.type MIRROR requires content.repos.target to be set. Repo: abc" + ); + + repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.MIRROR); + repo.setTarget("namespace/repo"); + repo.setPath("non-default-path"); + testConfig.getContent().setRepos(List.of(repo)); + + exception = assertThrows(RuntimeException.class, () -> featureContent.preConfigInit(testConfig)); + assertThat(exception.getMessage()).isEqualTo( + "content.repos.type MIRROR does not support path. Current path: non-default-path. Repo: abc" + ); + + repo = new Config.ContentSchema.ContentRepositorySchema(); + repo.setUrl("abc"); + repo.setType(Config.ContentRepoType.MIRROR); + repo.setTarget("namespace/repo"); + repo.setTemplating(true); + testConfig.getContent().setRepos(List.of(repo)); + + exception = assertThrows(RuntimeException.class, () -> featureContent.preConfigInit(testConfig)); + assertThat(exception.getMessage()).isEqualTo( + "content.repos.type MIRROR does not support templating. Repo: abc" + ); + } + + @Test + void ignoresEmptyLocalHelmChartFolderIfMirrorReposIsNotSet() { + testConfig.getApplication().setMirrorRepos(false); + testConfig.getApplication().setLocalHelmChartFolder(""); + + applicationConfigurator.initConfig(testConfig); + // no exceptions means success + } + + @Test + void baseUrlEvaluatesForAllTools() { + testConfig.getApplication().setBaseUrl("http://localhost"); + testConfig.getFeatures().getArgocd().setActive(true); + testConfig.getFeatures().getMonitoring().setActive(true); + testConfig.getFeatures().getSecrets().setActive(true); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getUrl()).isEqualTo("http://argocd.localhost"); + assertThat(actualConfig.getFeatures().getMonitoring().getGrafanaUrl()).isEqualTo("http://grafana.localhost"); + assertThat(actualConfig.getFeatures().getSecrets().getVault().getUrl()).isEqualTo("http://vault.localhost"); + assertThat(actualConfig.getScm().getScmManager().getIngress()).isEqualTo("scmm.localhost"); + assertThat(actualConfig.getJenkins().getIngress()).isEqualTo("jenkins.localhost"); + } + + @Test + void baseUrlWithUrlHyphensEvaluatesForAllTools() { + testConfig.getApplication().setBaseUrl("http://localhost"); + testConfig.getApplication().setUrlSeparatorHyphen(true); + testConfig.getFeatures().getArgocd().setActive(true); + testConfig.getFeatures().getMonitoring().setActive(true); + testConfig.getFeatures().getSecrets().setActive(true); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getUrl()).isEqualTo("http://argocd-localhost"); + assertThat(actualConfig.getFeatures().getMonitoring().getGrafanaUrl()).isEqualTo("http://grafana-localhost"); + assertThat(actualConfig.getFeatures().getSecrets().getVault().getUrl()).isEqualTo("http://vault-localhost"); + assertThat(actualConfig.getScm().getScmManager().getIngress()).isEqualTo("scmm-localhost"); + assertThat(actualConfig.getJenkins().getIngress()).isEqualTo("jenkins-localhost"); + } + + @Test + void baseUrlAlsoWorksWhenPortIsIncluded() { + testConfig.getApplication().setBaseUrl("http://localhost:8080"); + testConfig.getFeatures().getArgocd().setActive(true); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getUrl()).isEqualTo("http://argocd.localhost:8080"); + } + + @Test + void baseUrlAlsoWorksWhenPortIsIncludedAndUrlHyphensAreSet() { + testConfig.getApplication().setBaseUrl("http://localhost:6502"); + testConfig.getFeatures().getArgocd().setActive(true); + testConfig.getApplication().setUrlSeparatorHyphen(true); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getUrl()).isEqualTo("http://argocd-localhost:6502"); + } + + @Test + void baseUrlDoesNotEvaluateForInactiveTools() { + testConfig.getFeatures().getArgocd().setActive(false); + testConfig.getFeatures().getMail().setActive(false); + testConfig.getFeatures().getMonitoring().setActive(false); + testConfig.getFeatures().getSecrets().setActive(false); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getUrl()).isEqualTo(""); + assertThat(actualConfig.getFeatures().getMonitoring().getGrafanaUrl()).isEqualTo(""); + assertThat(actualConfig.getFeatures().getSecrets().getVault().getUrl()).isEqualTo(""); + } + + @Test + void baseUrlIndividualUrlParamsTakePrecedence() { + testConfig.getApplication().setBaseUrl("http://localhost"); + testConfig.getFeatures().getArgocd().setActive(true); + testConfig.getFeatures().getMail().setActive(true); + testConfig.getFeatures().getMonitoring().setActive(true); + testConfig.getFeatures().getSecrets().setActive(true); + testConfig.getFeatures().getArgocd().setUrl("argocd"); + testConfig.getFeatures().getMonitoring().setGrafanaUrl("grafana"); + testConfig.getFeatures().getSecrets().getVault().setUrl("vault"); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getUrl()).isEqualTo("argocd"); + assertThat(actualConfig.getFeatures().getMonitoring().getGrafanaUrl()).isEqualTo("grafana"); + assertThat(actualConfig.getFeatures().getSecrets().getVault().getUrl()).isEqualTo("vault"); + } + + @Test + void setsNamePrefix() { + testConfig.getApplication().setNamePrefix("my-prefix"); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + assertThat(actualConfig.getApplication().getNamePrefix().toString()).isEqualTo("my-prefix-"); + assertThat(actualConfig.getApplication().getNamePrefixForEnvVars().toString()).isEqualTo("MY_PREFIX_"); + } + + @Test + void setsNamePrefixWhenEndingInHyphen() { + testConfig.getApplication().setNamePrefix("my-prefix-"); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + assertThat(actualConfig.getApplication().getNamePrefix().toString()).isEqualTo("my-prefix-"); + assertThat(actualConfig.getApplication().getNamePrefixForEnvVars().toString()).isEqualTo("MY_PREFIX_"); + } + + @Test + void registrySetsToExternalWhenOnlyRegistryUrlSet() { + testConfig.getRegistry().setProxyUrl(null); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getRegistry().getInternal()).isEqualTo(false); + assertThat(actualConfig.getRegistry().getActive()).isEqualTo(true); + } + + @Test + void registryFailsWhenProxyButNoUsernameAndPasswordSet() { + String expectedException = "Proxy URL needs to be used with proxy-username and proxy-password"; + + testConfig.getRegistry().setProxyUsername(null); + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> applicationConfigurator.initConfig(testConfig) + ); + assertThat(exception.getMessage()).isEqualTo(expectedException); + + testConfig.getRegistry().setProxyUsername("something"); + testConfig.getRegistry().setProxyPassword(null); + exception = assertThrows(RuntimeException.class, () -> applicationConfigurator.initConfig(testConfig)); + assertThat(exception.getMessage()).isEqualTo(expectedException); + + testConfig.getRegistry().setProxyUsername(null); + exception = assertThrows(RuntimeException.class, () -> applicationConfigurator.initConfig(testConfig)); + assertThat(exception.getMessage()).isEqualTo(expectedException); + } + + @Test + void validateEnvConfigAllowsValidEnvEntries() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://100.125.0.1:443"); + testConfig.getFeatures().getArgocd().setEnv(List.of( + Map.of("name", "ENV_VAR_1", "value", "value1"), + Map.of("name", "ENV_VAR_2", "value", "value2") + )); + + // No exception should be thrown + applicationConfigurator.initConfig(testConfig); + } + + @Test + void validateEnvConfigThrowsExceptionForMissingNameInEnvEntry() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://100.125.0.1:443"); + testConfig.getFeatures().getArgocd().setEnv(List.of( + Map.of("name", "ENV_VAR_1", "value", "value1"), + Map.of("value", "value2") + )); + + IllegalArgumentException exception = assertThrows( + IllegalArgumentException.class, () -> { + applicationConfigurator.initConfig(testConfig); + featureArgoCd.postConfigInit(testConfig); + } + ); + + assertThat(exception.getMessage()).contains( + "Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: [value:value2]" + ); + } + + @Test + void validateEnvConfigThrowsExceptionForMissingValueInEnvEntry() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://100.125.0.1:443"); + testConfig.getFeatures().getArgocd().setEnv(List.of( + Map.of("name", "ENV_VAR_1", "value", "value1"), + Map.of("name", "ENV_VAR_2") + )); + + IllegalArgumentException exception = assertThrows( + IllegalArgumentException.class, () -> { + applicationConfigurator.initConfig(testConfig); + featureArgoCd.postConfigInit(testConfig); + } + ); + + assertThat(exception.getMessage()).contains( + "Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: [name:ENV_VAR_2]" + ); + } + + @Test + @SuppressWarnings({"rawtypes", "unchecked"}) + void validateEnvConfigThrowsExceptionForNonMapEnvEntry() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://100.125.0.1:443"); + testConfig.getFeatures().getArgocd().setEnv((List) List.of( + Map.of("name", "ENV_VAR_1", "value", "value1"), + "invalid_entry" + )); + + IllegalArgumentException exception = assertThrows( + IllegalArgumentException.class, () -> { + applicationConfigurator.initConfig(testConfig); + featureArgoCd.postConfigInit(testConfig); + } + ); + + assertThat(exception.getMessage()).contains( + "Each env variable in features.argocd.env must be a map with 'name' and 'value'. Invalid entry found: invalid_entry" + ); + } + + @Test + void validateEnvConfigAllowsEmptyEnvList() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://100.125.0.1:443"); + testConfig.getFeatures().getArgocd().getEnv(); + + // No exception should be thrown + applicationConfigurator.initConfig(testConfig); + } + + @Test + void validateEnvConfigSkipsValidationWhenOperatorIsFalse() { + testConfig.getFeatures().getArgocd().setOperator(false); + testConfig.getFeatures().getArgocd().setEnv(List.of( + Map.of("name", "ENV_VAR_1", "value", "value1"), + Map.of("value", "value2") + )); + + // No exception should be thrown + applicationConfigurator.initConfig(testConfig); + } + + @Test + void shouldSkipResourceInclusionsClusterSetupWhenArgoCdOperatorIsNotEnabled() { + testConfig.getFeatures().getArgocd().setOperator(false); + + // Calling the method should not make any changes to the config + applicationConfigurator.initConfig(testConfig); + + assertThat(testLogger.getLogs().search( + "ArgoCD operator is not enabled. Skipping features.argocd.resourceInclusionsCluster setup." + )).isNotEmpty(); + } + + @Test + void shouldValidateAndAcceptUserProvidedValidResourceInclusionsClusterUrl() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://valid-url.com"); + + applicationConfigurator.initConfig(testConfig); + + assertThat(testConfig.getFeatures().getArgocd().getResourceInclusionsCluster()).isEqualTo( + "https://valid-url.com"); + assertThat(testLogger.getLogs().search( + "Validating user-provided features.argocd.resourceInclusionsCluster URL: https://valid-url.com" + )).isNotEmpty(); + assertThat(testLogger.getLogs().search( + "Found valid URL in features.argocd.resourceInclusionsCluster: https://valid-url.com" + )).isNotEmpty(); + } + + @Test + void userProvidedResourceInclusionsClusterTrumpsEnvironmentVariables() { + applicationConfigurator = configuratorWithEnvironment(Map.of( + "KUBERNETES_SERVICE_HOST", "100.125.0.1", + "KUBERNETES_SERVICE_PORT", "443" + )); + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("https://192.168.0.1:6443"); + + applicationConfigurator.initConfig(testConfig); + + assertThat(testConfig.getFeatures().getArgocd().getResourceInclusionsCluster()) + .isEqualTo("https://192.168.0.1:6443"); + } + + @Test + void shouldThrowExceptionForUserProvidedInvalidResourceInclusionsClusterUrl() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster("invalid-url"); + + IllegalArgumentException exception = assertThrows( + IllegalArgumentException.class, + () -> applicationConfigurator.initConfig(testConfig) + ); + + assertThat(exception.getMessage()).contains( + "Invalid URL for 'features.argocd.resourceInclusionsCluster': invalid-url." + ); + } + + @Test + void shouldSetResourceInclusionsClusterUsingKubernetesEnvVariablesWhenNotProvidedByUser() { + applicationConfigurator = configuratorWithEnvironment(Map.of( + "KUBERNETES_SERVICE_HOST", "127.0.0.1", + "KUBERNETES_SERVICE_PORT", "6443" + )); + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster(null); + + Config actualConfig = applicationConfigurator.initConfig(testConfig); + + assertThat(actualConfig.getFeatures().getArgocd().getResourceInclusionsCluster()) + .isEqualTo("https://127.0.0.1:6443"); + assertThat(testLogger.getLogs().search( + "Successfully set features.argocd.resourceInclusionsCluster via Kubernetes ENV to: https://127.0.0.1:6443" + )).isNotEmpty(); + } + + @Test + void multiTenantModeCentralScmUrl() { + testConfig.getMultiTenant().setUseDedicatedInstance(true); + testConfig.getMultiTenant().getScmManager().setUrl("scmm.localhost/scm/"); + testConfig.getApplication().setNamePrefix("foo"); + applicationConfigurator.initConfig(testConfig); + assertThat(testConfig.getMultiTenant().getScmManager().getUrl()).isEqualTo("scmm.localhost/scm"); + } + + @Test + void shouldThrowExceptionWhenKubernetesEnvVariablesAreNotSetAndResourceInclusionsClusterIsNull() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster(null); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> applicationConfigurator.initConfig(testConfig) + ); + + assertThat(exception.getMessage()).contains( + "Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true. " + + "Ensure Kubernetes environment variables 'KUBERNETES_SERVICE_HOST' and 'KUBERNETES_SERVICE_PORT' are set properly." + ); + } + + @Test + void shouldThrowExceptionWhenKubernetesEnvVariablesAreNotSetAndResourceInclusionsClusterIsEmpty() { + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster(""); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> applicationConfigurator.initConfig(testConfig) + ); + + assertThat(exception.getMessage()).contains( + "Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true. " + + "Ensure Kubernetes environment variables 'KUBERNETES_SERVICE_HOST' and 'KUBERNETES_SERVICE_PORT' are set properly." + ); + } + + @Test + void shouldThrowExceptionForInvalidKubernetesConstructedUrl() { + applicationConfigurator = configuratorWithEnvironment(Map.of( + "KUBERNETES_SERVICE_HOST", "invalid_host", + "KUBERNETES_SERVICE_PORT", "not_a_port" + )); + testConfig.getFeatures().getArgocd().setOperator(true); + testConfig.getFeatures().getArgocd().setResourceInclusionsCluster(null); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> applicationConfigurator.initConfig(testConfig) + ); + + assertThat(exception.getMessage()).contains( + "Could not determine 'features.argocd.resourceInclusionsCluster' which is required when argocd.operator=true." + ); + + assertThat(testLogger.getLogs().search( + "Constructed internal Kubernetes API Server URL: https://invalid_host:not_a_port" + )).isNotEmpty(); + } + + private static ApplicationConfigurator configuratorWithEnvironment(Map environment) { + return new ApplicationConfigurator(environment::get); + } + + @Test + void setsAllToolNamespacesToApplicationNamespaceWhenConfigured() { + Config config = minimalConfig(); + config.getApplication().setNamespace("platform"); + config.getApplication().setNamePrefix("tenant-a"); + + config.getApplication().setGopNamespace("custom-gop"); + config.getRegistry().setNamespace("custom-registry"); + config.getJenkins().setNamespace("custom-jenkins"); + config.getScm().getScmManager().setNamespace("custom-scm"); + config.getFeatures().getArgocd().setNamespace("custom-argocd"); + config.getFeatures().getMonitoring().setNamespace("custom-monitoring"); + config.getFeatures().getSecrets().setNamespace("custom-secrets"); + config.getFeatures().getIngress().setIngressNamespace("custom-ingress"); + config.getFeatures().getCertManager().setNamespace("custom-cert-manager"); + config.getContent().setNamespaces(new ArrayList<>(List.of("old-namespace", "another-namespace"))); + + Config actualConfig = applicationConfigurator.initConfig(config); + + assertThat(actualConfig.getApplication().getGopNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getRegistry().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getJenkins().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getScm().getScmManager().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getFeatures().getArgocd().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getFeatures().getMonitoring().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getFeatures().getSecrets().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getFeatures().getIngress().getIngressNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getFeatures().getCertManager().getNamespace()).isEqualTo("platform"); + assertThat(actualConfig.getContent().getNamespaces()).containsExactly("tenant-a-platform"); + } + + @Test + void keepsIndividualToolNamespacesWhenApplicationNamespaceIsNotConfigured() { + Config config = minimalConfig(); + config.getApplication().setNamespace(""); + config.getApplication().setNamePrefix("tenant-a"); + + config.getApplication().setGopNamespace("custom-gop"); + config.getRegistry().setNamespace("custom-registry"); + config.getJenkins().setNamespace("custom-jenkins"); + config.getScm().getScmManager().setNamespace("custom-scm"); + config.getFeatures().getArgocd().setNamespace("custom-argocd"); + config.getFeatures().getMonitoring().setNamespace("custom-monitoring"); + config.getFeatures().getSecrets().setNamespace("custom-secrets"); + config.getFeatures().getIngress().setIngressNamespace("custom-ingress"); + config.getFeatures().getCertManager().setNamespace("custom-cert-manager"); + config.getContent().setNamespaces(new ArrayList<>(List.of("old-namespace", "another-namespace"))); + + Config actualConfig = applicationConfigurator.initConfig(config); + + assertThat(actualConfig.getApplication().getGopNamespace()).isEqualTo("custom-gop"); + assertThat(actualConfig.getRegistry().getNamespace()).isEqualTo("custom-registry"); + assertThat(actualConfig.getJenkins().getNamespace()).isEqualTo("custom-jenkins"); + assertThat(actualConfig.getScm().getScmManager().getNamespace()).isEqualTo("custom-scm"); + assertThat(actualConfig.getFeatures().getArgocd().getNamespace()).isEqualTo("custom-argocd"); + assertThat(actualConfig.getFeatures().getMonitoring().getNamespace()).isEqualTo("custom-monitoring"); + assertThat(actualConfig.getFeatures().getSecrets().getNamespace()).isEqualTo("custom-secrets"); + assertThat(actualConfig.getFeatures().getIngress().getIngressNamespace()).isEqualTo("custom-ingress"); + assertThat(actualConfig.getFeatures().getCertManager().getNamespace()).isEqualTo("custom-cert-manager"); + assertThat(actualConfig.getContent().getNamespaces()).containsExactly("old-namespace", "another-namespace"); + } + + List getAllFieldNames(Class clazz) { + return getAllFieldNames(clazz, "", new ArrayList<>()); + } + + List getAllFieldNames(Class clazz, String parentField, List fieldNames) { + for (Field field : clazz.getDeclaredFields()) { + String currentField = parentField + field.getName(); + if (!field.getType().isArray() && field.getType().getName().startsWith(Config.class.getPackageName())) { + System.out.println("nested class " + field.getType() + ", " + currentField + " + '.', " + fieldNames); + getAllFieldNames(field.getType(), currentField + ".", fieldNames); + } else if (!field.getName().startsWith("_") && + !field.getName().startsWith("$") && + !field.getName().equals("metaClass")) { + fieldNames.add(currentField); + } + } + return fieldNames; + } + + List getAllKeys(Map map) { + return getAllKeys(map, "", new ArrayList<>()); + } + + List getAllKeys(Map map, String parentKey, List keysList) { + for (Map.Entry entry : map.entrySet()) { + String currentKey = parentKey + entry.getKey(); + Object value = entry.getValue(); + if (value instanceof Map nested && !nested.isEmpty()) { + getAllKeys(nested, currentKey + ".", keysList); + } else { + keysList.add(currentKey); + } + } + return keysList; + } + + private static Config minimalConfig() { + Config config = new Config(); + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setLocalHelmChartFolder("someValue"); + application.setNamePrefix(""); + config.setApplication(application); + + ScmTenantSchema scm = new ScmTenantSchema(); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setUrl(""); + scm.setScmManager(scmManager); + config.setScm(scm); + + return config; + } +} diff --git a/src/test/java/com/cloudogu/gitops/cli/GenerateJsonSchemaTest.java b/src/test/java/com/cloudogu/gitops/cli/GenerateJsonSchemaTest.java new file mode 100644 index 000000000..e7a7611ac --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/cli/GenerateJsonSchemaTest.java @@ -0,0 +1,16 @@ +package com.cloudogu.gitops.cli; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class GenerateJsonSchemaTest { + + @Test + void generatesDocumentationForEnumFieldsWithoutReflectingIntoEnumInternals() { + assertThat(GenerateJsonSchema.generateDocs()) + .contains("| `scm.scmProviderType` | ScmProviderType | `SCM_MANAGER` |") + .contains("`{}`") + .doesNotContain("`[:]`"); + } +} diff --git a/src/test/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliMainTest.java b/src/test/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliMainTest.java new file mode 100644 index 000000000..6cc1e1cdb --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliMainTest.java @@ -0,0 +1,59 @@ +package com.cloudogu.gitops.cli; + +import org.junit.jupiter.api.Test; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +import static org.assertj.core.api.Assertions.assertThat; + +class GitopsPlaygroundCliMainTest { + + @Test + void applicationReturnsExitCodeZeroOnSuccess() { + GitopsPlaygroundCliMain gitopsPlaygroundCliMain = new GitopsPlaygroundCliMain(); + ReturnCode returnCode = gitopsPlaygroundCliMain.exec(new String[]{"--mock"}, MockedCommand.class); + + assertThat(returnCode.ordinal()).isZero(); + } + + @Test + void applicationReturnsNonZeroExitCodeOnException() { + GitopsPlaygroundCliMain gitopsPlaygroundCliMain = new GitopsPlaygroundCliMain(); + ReturnCode returnCode = gitopsPlaygroundCliMain.exec(new String[]{"--mock"}, ThrowingCommand.class); + + assertThat(returnCode.ordinal()).isNotZero(); + } + + @Test + void applicationReturnsNonZeroExitCodeOnInvalidParam() { + ReturnCode returnCode = new GitopsPlaygroundCliMain().exec( + new String[]{"--parameter-that-doesnt-exist ", "--debug"}, + GitopsPlaygroundCli.class + ); + + assertThat(returnCode.ordinal()).isNotZero(); + } + + static class ThrowingCommand extends MockedCommand { + @Override + public ReturnCode run(String[] args) { + throw new RuntimeException("mock"); + } + } + + @SuppressWarnings("unused") + static class MockedCommand extends GitopsPlaygroundCli { + + @Override + public ReturnCode run(String[] args) { + return ReturnCode.SUCCESS; + } + + @Command + void mockedCommand() { + } + + @Option(names = "--mock") + private boolean mock; + } +} diff --git a/src/test/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliTest.java b/src/test/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliTest.java new file mode 100644 index 000000000..6db3a2a84 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/cli/GitopsPlaygroundCliTest.java @@ -0,0 +1,421 @@ +package com.cloudogu.gitops.cli; + +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.LoggerContext; +import ch.qos.logback.classic.encoder.PatternLayoutEncoder; +import ch.qos.logback.core.ConsoleAppender; +import com.cloudogu.gitops.application.Application; +import com.cloudogu.gitops.application.content.ContentLoader; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.destroy.Destroyer; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.tools.common.AbstractTool; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.micronaut.context.ApplicationContext; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; +import org.slf4j.LoggerFactory; + +import java.io.ByteArrayInputStream; +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +@Timeout(value = 10, unit = TimeUnit.SECONDS) +class GitopsPlaygroundCliTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final String ORIGINAL_LOGGING_PATTERN = getLoggingEncoder().getPattern(); + + private final K8sClient k8sClient = mock(K8sClient.class); + private final Application application = mock(Application.class); + private final ApplicationConfigurator applicationConfigurator = mock(ApplicationConfigurator.class); + private final Destroyer destroyer = mock(Destroyer.class); + private final GitopsPlaygroundCliForTest cli = new GitopsPlaygroundCliForTest(); + + @AfterEach + void setup() { + // Restore logging pattern, if modified + getLoggingEncoder().setPattern(ORIGINAL_LOGGING_PATTERN); + } + + @Test + void startsRegularly() { + ReturnCode status = cli.run(new String[]{"--yes"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + verify(applicationConfigurator).initConfig(any(Config.class)); + verify(application).start(); + } + + @Test + void runsConfigLifecycleHooksOnlyForParticipatingTools() { + AbstractTool regularTool = mock(AbstractTool.class); + ContentLoader configLifecycleHook = mock(ContentLoader.class); + when(application.getTools()).thenReturn(List.of(regularTool, configLifecycleHook)); + + ReturnCode status = cli.run(new String[]{"--yes"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + verify(configLifecycleHook).preConfigInit(any(Config.class)); + verify(configLifecycleHook).postConfigInit(any(Config.class)); + verifyNoInteractions(regularTool); + } + + @Test + void startsWithConfigFile() { + String pathToConfigFile = "./src/test/resources/testMainConfig.yaml"; + + assertThat(new File(pathToConfigFile).isFile()) + .withFailMessage("config file for test do not exists anymore.") + .isTrue(); + + ReturnCode status = cli.run(new String[]{"--config-file=" + pathToConfigFile}); + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + + verify(applicationConfigurator).initConfig(any(Config.class)); + verify(application).start(); + } + + @Test + void startsWithConfigMap() { + when(k8sClient.getConfigMap("my-config", "config.yaml")) + .thenReturn("{\"application\": {\"yes\": true}}"); + + ReturnCode status = cli.run(new String[]{"--config-map=my-config"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + verify(applicationConfigurator).initConfig(any(Config.class)); + verify(application).start(); + } + + @Test + void startsWithDocumentedKeycloakOidcProfile() { + ReturnCode status = cli.run(new String[]{"--profile=keycloak"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + assertThat(cli.lastSchema.getFeatures().getArgocd().getOidc().isEnabled()).isTrue(); + assertThat(cli.lastSchema.getFeatures().getArgocd().getOidc().getClientId()).isEqualTo("argocd"); + assertThat(cli.lastSchema.getFeatures().getMonitoring().getOidc().isEnabled()).isTrue(); + assertThat(cli.lastSchema.getFeatures().getMonitoring().getOidc().getClientId()).isEqualTo("grafana"); + assertThat(cli.lastSchema.getFeatures().getSecrets().getVault().getOidc().isEnabled()).isTrue(); + assertThat(cli.lastSchema.getFeatures().getSecrets().getVault().getOidc().getClientId()).isEqualTo("vault"); + assertThat(cli.lastSchema.getJenkins().getOidc().isEnabled()).isTrue(); + assertThat(cli.lastSchema.getJenkins().getOidc().getClientId()).isEqualTo("jenkins"); + } + + @Test + void outputsConfigFile() { + ReturnCode status = cli.run(new String[]{"--output-config-file"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + verify(applicationConfigurator, never()).initConfig(any(Config.class)); + verify(application, never()).start(); + } + + @Test + void outputsVersion() { + GitopsPlaygroundCliForTest localCli = new GitopsPlaygroundCliForTest(); + ReturnCode status = localCli.run(new String[]{"--version"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + verify(applicationConfigurator, never()).initConfig(any(Config.class)); + verify(application, never()).start(); + } + + @Test + void outputsHelp() { + GitopsPlaygroundCliForTest localCli = new GitopsPlaygroundCliForTest(); + ReturnCode status = localCli.run(new String[]{"--help"}); + + assertThat(status).isEqualTo(ReturnCode.SUCCESS); + verify(applicationConfigurator, never()).initConfig(any(Config.class)); + verify(application, never()).start(); + } + + @Test + void returnsErrorWhenApplyingIsNotConfirmed() { + writeViaSystemIn("something"); + ReturnCode status = cli.run(new String[]{}); + + assertThat(status).isEqualTo(ReturnCode.NOT_CONFIRMED); + } + + @Test + void runsWhenApplyingIsConfirmed() { + writeViaSystemIn("y"); + + cli.run(new String[]{}); + + verify(application).start(); + } + + @Test + void runsWithoutConfirmationWhenYesParameterIsSet() { + cli.run(new String[]{"--yes"}); + + verify(application).start(); + } + + @Test + void returnsErrorWhenDestroyingIsNotConfirmed() { + writeViaSystemIn("something"); + + ReturnCode status = cli.run(new String[]{"--destroy"}); + + assertThat(status).isEqualTo(ReturnCode.NOT_CONFIRMED); + } + + @Test + void destroysWhenConfirmed() { + writeViaSystemIn("y"); + + cli.run(new String[]{"--destroy"}); + + verify(destroyer).destroy(); + verify(application, never()).start(); + } + + @Test + void destroysWithoutConfirmationWhenYesParameterIsSet() { + cli.run(new String[]{"--destroy", "--yes"}); + + verify(destroyer).destroy(); + } + + @Test + void setsSimplifiedLoggingPattern() { + cli.run(new String[]{"--yes"}); + + assertThat(getLoggingPattern()).doesNotContain("%logger", "%thread"); + } + + @Test + void keepsSimplifiedLoggingPatternWhenTraceIsEnabled() { + cli.run(new String[]{"--trace", "--yes"}); + + assertThat(getLoggingPattern()).contains("%logger", "%thread"); + } + + @Test + void keepsSimplifiedLoggingPatternWhenDebugIsEnabled() { + cli.run(new String[]{"--debug", "--yes"}); + + assertThat(getLoggingPattern()).contains("%logger", "%thread"); + } + + @Test + void failsOnInvalidConfigFile() throws IOException { + File configFile = File.createTempFile("gop", ".yaml"); + configFile.deleteOnExit(); + java.nio.file.Files.writeString(configFile.toPath(), "something: not-matching-our-schema"); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> cli.run(new String[]{"--config-file=" + configFile, "--yes"}) + ); + assertThat(exception.getMessage()).contains("Config file invalid"); + } + + @Test + void failsOnInvalidConfigMap() { + when(k8sClient.getConfigMap("my-config", "config.yaml")) + .thenReturn("something: not-matching-our-schema"); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> cli.run(new String[]{"--config-map=my-config", "--yes"}) + ); + assertThat(exception.getMessage()).contains("Config file invalid"); + } + + @Test + void precedenceConfigFileOverwritesConfigMapAndCliOverwritesConfigFile() throws IOException { + Map cmConfig = Map.of( + "application", Map.of("username", "cmUser", "password", "cmPw", "namePrefix", "cmPref") + ); + Map fileConfig = Map.of( + "application", Map.of("username", "fileUser", "password", "filePw") + ); + + File configFile = File.createTempFile("gop", ".yaml"); + configFile.deleteOnExit(); + + java.nio.file.Files.writeString(configFile.toPath(), toYaml(fileConfig)); + when(k8sClient.getConfigMap("my-config", "config.yaml")).thenReturn(toYaml(cmConfig)); + + cli.run(new String[]{ + "--config-file=" + configFile, + "--config-map=my-config", + "--username=paramUser", + "--yes" + }); + + assertThat(cli.lastSchema.getApplication().getUsername()).isEqualTo("paramUser"); + assertThat(cli.lastSchema.getApplication().getPassword()).isEqualTo("filePw"); + assertThat(cli.lastSchema.getApplication().getNamePrefix()).isEqualTo("cmPref"); + } + + @Test + void helmNullValuesOverwrite() throws IOException { + Map fileConfig = Map.of( + "features", Map.of( + "monitoring", Map.of( + "helm", Map.of("repoURL", "https://prometheus-community.github.io/helm-chartsTEST") + ) + ) + ); + + File configFile = File.createTempFile("gop", ".yaml"); + configFile.deleteOnExit(); + + java.nio.file.Files.writeString(configFile.toPath(), toYaml(fileConfig)); + + cli.run(new String[]{"--config-file=" + configFile, "--yes"}); + + assertThat(cli.lastSchema.getFeatures().getMonitoring().getHelm().getChart()) + .isEqualTo("kube-prometheus-stack"); + assertThat(cli.lastSchema.getFeatures().getMonitoring().getHelm().getRepoURL()) + .isEqualTo("https://prometheus-community.github.io/helm-chartsTEST"); + assertThat(cli.lastSchema.getFeatures().getMonitoring().getHelm().getVersion()).isEqualTo("80.2.2"); + } + + @Test + void ensureHelmDefaultsAreUsedIfNotSet() throws IOException { + Map fileConfig = Map.of( + "jenkins", Map.of("helm", Map.of("version", "5.8.1")), + "scm", Map.of( + "scmManager", Map.of( + "helm", Map.of( + "values", Map.of("initialDelaySeconds", 120) + ) + ) + ), + "features", Map.of( + "monitoring", Map.of( + "helm", Map.of( + "version", "66.2.1", + "grafanaImage", "localhost:30000/proxy/grafana:latest" + ) + ), + "secrets", Map.of( + "externalSecrets", Map.of("helm", Map.of("chart", "my-secrets")), + "vault", Map.of("helm", Map.of("repoURL", "localhost:3000/proxy/vault:latest")) + ), + "certManager", Map.of( + "helm", Map.of("image", "localhost:30000/proxy/cert-manager-controller:latest") + ) + ) + ); + + File configFile = File.createTempFile("gop", ".yaml"); + configFile.deleteOnExit(); + + java.nio.file.Files.writeString(configFile.toPath(), toYaml(fileConfig)); + + cli.run(new String[]{"--config-file=" + configFile, "--yes"}); + Config myConfig = cli.lastSchema; + assertThat(myConfig.getJenkins().getHelm().getChart()).isEqualTo("jenkins"); + assertThat(myConfig.getJenkins().getHelm().getRepoURL()).isEqualTo("https://charts.jenkins.io"); + assertThat(myConfig.getJenkins().getHelm().getVersion()).isEqualTo("5.8.1"); + + assertThat(myConfig.getScm().getScmManager().getHelm().getChart()).isEqualTo("scm-manager"); + assertThat(myConfig.getScm().getScmManager().getHelm().getRepoURL()) + .isEqualTo("https://packages.scm-manager.org/repository/helm-v2-releases/"); + assertThat(myConfig.getScm().getScmManager().getHelm().getVersion()).isEqualTo("3.11.10"); + assertThat(myConfig.getScm().getScmManager().getHelm().getValues().get("initialDelaySeconds")) + .isEqualTo(120); + + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getChart()).isEqualTo("kube-prometheus-stack"); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getRepoURL()) + .isEqualTo("https://prometheus-community.github.io/helm-charts"); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getVersion()).isEqualTo("66.2.1"); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getGrafanaSidecarImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getPrometheusImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getPrometheusConfigReloaderImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getPrometheusOperatorImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getMonitoring().getHelm().getGrafanaImage()) + .isEqualTo("localhost:30000/proxy/grafana:latest"); + + assertThat(myConfig.getFeatures().getSecrets().getExternalSecrets().getHelm().getChart()).isEqualTo("my-secrets"); + assertThat(myConfig.getFeatures().getSecrets().getExternalSecrets().getHelm().getRepoURL()) + .isEqualTo("https://charts.external-secrets.io"); + assertThat(myConfig.getFeatures().getSecrets().getExternalSecrets().getHelm().getVersion()).isEqualTo("0.9.16"); + + assertThat(myConfig.getFeatures().getSecrets().getVault().getHelm().getChart()).isEqualTo("vault"); + assertThat(myConfig.getFeatures().getSecrets().getVault().getHelm().getRepoURL()) + .isEqualTo("localhost:3000/proxy/vault:latest"); + assertThat(myConfig.getFeatures().getSecrets().getVault().getHelm().getVersion()).isEqualTo("0.34.1"); + + assertThat(myConfig.getFeatures().getCertManager().getHelm().getChart()).isEqualTo("cert-manager"); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getRepoURL()).isEqualTo( + "https://charts.jetstack.io"); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getVersion()).isEqualTo("1.19.4"); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getStartupAPICheckImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getWebhookImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getCainjectorImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getAcmeSolverImage()).isEqualTo(""); + assertThat(myConfig.getFeatures().getCertManager().getHelm().getImage()) + .isEqualTo("localhost:30000/proxy/cert-manager-controller:latest"); + } + + private static String getLoggingPattern() { + return getLoggingEncoder().getPattern(); + } + + private static PatternLayoutEncoder getLoggingEncoder() { + LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory(); + Logger rootLogger = loggerContext.getLogger(Logger.ROOT_LOGGER_NAME); + ConsoleAppender consoleAppender = (ConsoleAppender) rootLogger.getAppender("STDOUT"); + return (PatternLayoutEncoder) consoleAppender.getEncoder(); + } + + private void writeViaSystemIn(String value) { + ByteArrayInputStream inContent = new ByteArrayInputStream((value + "\n").getBytes(StandardCharsets.UTF_8)); + System.setIn(inContent); + } + + private static String toYaml(Map map) throws IOException { + return YAML_MAPPER.writeValueAsString(map); + } + + class GitopsPlaygroundCliForTest extends GitopsPlaygroundCli { + private final ApplicationContext applicationContext = mock(ApplicationContext.class); + private Config lastSchema; + + GitopsPlaygroundCliForTest() { + super(GitopsPlaygroundCliTest.this.k8sClient, GitopsPlaygroundCliTest.this.applicationConfigurator); + + when(applicationConfigurator.initConfig(any(Config.class))).thenAnswer(new Answer() { + @Override + public Config answer(InvocationOnMock invocation) { + lastSchema = invocation.getArgument(0); + return lastSchema; + } + }); + } + + @Override + protected ApplicationContext createApplicationContext() { + when(applicationContext.getBean(Application.class)).thenReturn(application); + when(applicationContext.getBean(Destroyer.class)).thenReturn(destroyer); + + return applicationContext; + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/cli/VersionTest.java b/src/test/java/com/cloudogu/gitops/cli/VersionTest.java new file mode 100644 index 000000000..9a14f18c6 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/cli/VersionTest.java @@ -0,0 +1,17 @@ +package com.cloudogu.gitops.cli; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class VersionTest { + + @Test + void loadsGeneratedVersionName() { + assertThat(Version.NAME) + .isNotBlank() + .doesNotContain("${") + .contains("Copyright 2020 - present Cloudogu GmbH") + .contains("GNU AFFERO GENERAL PUBLIC LICENSE, Version 3"); + } +} diff --git a/src/test/java/com/cloudogu/gitops/config/schema/ConfigTest.java b/src/test/java/com/cloudogu/gitops/config/schema/ConfigTest.java new file mode 100644 index 000000000..6b2480840 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/config/schema/ConfigTest.java @@ -0,0 +1,98 @@ +package com.cloudogu.gitops.config.schema; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Config.VaultMode; +import com.cloudogu.gitops.utils.MapUtils; +import org.junit.jupiter.api.Test; +import picocli.CommandLine; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class ConfigTest { + + private final Config testConfig = createTestConfig(); + + @Test + void convertsToYamlIncludingInternals() { + String config = testConfig.toYaml(true); + + assertThat(config).startsWith("---\nregistry:\n internal: true\n"); + } + + @Test + void convertsConfigMapToYaml() { + String config = testConfig.toYaml(false); + + assertThat(config).startsWith("---\nregistry:\n active: false\n"); + } + + @Test + void createsFromSchemaOverwritingOnlyMapValuesIgnoringNullValues() { + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setUsername("myUser"); + application.setYes(true); + application.setNamePrefix("aPrefix"); + + Config.RegistrySchema registry = new Config.RegistrySchema(); + registry.setInternalPort(42); + + Config expectedValues = new Config(); + expectedValues.setApplication(application); + expectedValues.setRegistry(registry); + + Config actualValues = Config.fromMap(expectedValues.toMap()); + + assertThat(actualValues.getApplication().getUsername()).isEqualTo(expectedValues.getApplication().getUsername()); + assertThat(actualValues.getApplication().getYes()).isEqualTo(expectedValues.getApplication().getYes()); + assertThat(actualValues.getApplication().getNamePrefix()).isEqualTo(expectedValues.getApplication().getNamePrefix()); + assertThat(actualValues.getRegistry().getInternalPort()).isEqualTo(expectedValues.getRegistry().getInternalPort()); + } + + @Test + void parsesLowercaseVaultModeFromConfigAndPreservesExternalRepresentation() { + Map input = Map.of( + "features", Map.of( + "secrets", Map.of( + "vault", Map.of("mode", "dev") + ) + ) + ); + Config config = Config.fromMap(input); + + assertThat(config.getFeatures().getSecrets().getVault().getMode()).isEqualTo(VaultMode.DEV); + + Map configMap = config.toMap(); + Map features = MapUtils.asStringObjectMap(configMap.get("features")); + Map secrets = MapUtils.asStringObjectMap(features.get("secrets")); + Map vault = MapUtils.asStringObjectMap(secrets.get("vault")); + assertThat(vault.get("mode")).isEqualTo("dev"); + } + + @Test + void parsesLowercaseVaultModeFromCli() { + Config config = new Config(); + + new CommandLine(config).parseArgs("--vault=dev"); + + assertThat(config.getFeatures().getSecrets().getVault().getMode()).isEqualTo(VaultMode.DEV); + } + + @Test + void getsTenantNameFromConfig() { + testConfig.getApplication().setNamePrefix("testprefix-"); + + assertThat(testConfig.getApplication().getTenantName()).isEqualTo("testprefix"); + } + + private static Config createTestConfig() { + Config.RegistrySchema registry = new Config.RegistrySchema(); + registry.setTwoRegistries(true); + registry.setInternalPort(123); + + Config config = new Config(); + config.setRegistry(registry); + return config; + } +} diff --git a/src/test/java/com/cloudogu/gitops/config/schema/CredentialsReferenceConfigTest.java b/src/test/java/com/cloudogu/gitops/config/schema/CredentialsReferenceConfigTest.java new file mode 100644 index 000000000..1c6fb1d72 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/config/schema/CredentialsReferenceConfigTest.java @@ -0,0 +1,114 @@ +package com.cloudogu.gitops.config.schema; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmCentralSchema; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class CredentialsReferenceConfigTest { + + @Test + void copyConstructorKeepsOnlyReferenceMetadata() { + Credentials original = new Credentials( + "resolved-user", + "resolved-password", + "tool-credentials", + "gop-job", + "custom-user", + "custom-password" + ); + + Credentials copy = new Credentials(original); + + assertThat(copy.getUsername()).isNull(); + assertThat(copy.getPassword()).isNull(); + assertThat(copy.getSecretName()).isEqualTo("tool-credentials"); + assertThat(copy.getSecretNamespace()).isEqualTo("gop-job"); + assertThat(copy.getUsernameKey()).isEqualTo("custom-user"); + assertThat(copy.getPasswordKey()).isEqualTo("custom-password"); + } + + @Test + void storesSecretReferencesWithoutChangingPlainCredentials() { + Config config = new Config(); + Credentials reference = secretReference("tool-credentials"); + + config.getApplication().setUsername("application-user"); + config.getApplication().setPassword("application-password"); + config.getApplication().setCredentials(reference); + + config.getJenkins().setUsername("jenkins-user"); + config.getJenkins().setPassword("jenkins-password"); + config.getJenkins().setCredentials(reference); + + config.getFeatures().getMail().setSmtpUser("smtp-user"); + config.getFeatures().getMail().setSmtpPassword("smtp-password"); + config.getFeatures().getMail().setCredentials(reference); + + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setPassword("registry-password"); + config.getRegistry().setCredentials(reference); + config.getRegistry().setProxyCredentials(reference); + config.getRegistry().setReadOnlyCredentials(reference); + + assertThat(config.getApplication().getCredentials()).isSameAs(reference); + assertThat(config.getApplication().getUsername()).isEqualTo("application-user"); + assertThat(config.getApplication().getPassword()).isEqualTo("application-password"); + assertThat(config.getJenkins().getCredentials()).isSameAs(reference); + assertThat(config.getJenkins().getUsername()).isEqualTo("jenkins-user"); + assertThat(config.getJenkins().getPassword()).isEqualTo("jenkins-password"); + assertThat(config.getFeatures().getMail().getCredentials()).isSameAs(reference); + assertThat(config.getFeatures().getMail().getSmtpUser()).isEqualTo("smtp-user"); + assertThat(config.getFeatures().getMail().getSmtpPassword()).isEqualTo("smtp-password"); + assertThat(config.getRegistry().getCredentials()).isSameAs(reference); + assertThat(config.getRegistry().getProxyCredentials()).isSameAs(reference); + assertThat(config.getRegistry().getReadOnlyCredentials()).isSameAs(reference); + assertThat(config.getRegistry().getUsername()).isEqualTo("registry-user"); + assertThat(config.getRegistry().getPassword()).isEqualTo("registry-password"); + } + + @Test + void scmConfigsPreferSecretReferences() { + Credentials reference = secretReference("scm-credentials"); + + ScmTenantSchema.GitlabTenantConfig tenantGitlab = new ScmTenantSchema.GitlabTenantConfig(); + tenantGitlab.setCredentials(reference); + ScmTenantSchema.ScmManagerTenantConfig tenantScmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + tenantScmManager.setCredentials(reference); + ScmCentralSchema.GitlabCentralConfig centralGitlab = new ScmCentralSchema.GitlabCentralConfig(); + centralGitlab.setCredentials(reference); + ScmCentralSchema.ScmManagerCentralConfig centralScmManager = new ScmCentralSchema.ScmManagerCentralConfig(); + centralScmManager.setCredentials(reference); + + assertThat(tenantGitlab.getCredentials()).isSameAs(reference); + assertThat(tenantScmManager.getCredentials()).isSameAs(reference); + assertThat(centralGitlab.getCredentials()).isSameAs(reference); + assertThat(centralScmManager.getCredentials()).isSameAs(reference); + } + + @Test + void scmConfigsKeepPlainCredentialsAsFallback() { + ScmTenantSchema.GitlabTenantConfig gitlab = new ScmTenantSchema.GitlabTenantConfig(); + gitlab.setUsername("gitlab-user"); + gitlab.setPassword("gitlab-token"); + + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setUsername("scmm-user"); + scmManager.setPassword("scmm-password"); + + assertThat(gitlab.getCredentials().getUsername()).isEqualTo("gitlab-user"); + assertThat(gitlab.getCredentials().getPassword()).isEqualTo("gitlab-token"); + assertThat(scmManager.getCredentials().getUsername()).isEqualTo("scmm-user"); + assertThat(scmManager.getCredentials().getPassword()).isEqualTo("scmm-password"); + } + + private static Credentials secretReference(String secretName) { + Credentials reference = new Credentials(); + reference.setSecretName(secretName); + reference.setSecretNamespace("gop-job"); + return reference; + } +} diff --git a/src/test/java/com/cloudogu/gitops/config/schema/JsonConfigValidatorTest.java b/src/test/java/com/cloudogu/gitops/config/schema/JsonConfigValidatorTest.java new file mode 100644 index 000000000..c5dc57571 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/config/schema/JsonConfigValidatorTest.java @@ -0,0 +1,63 @@ +package com.cloudogu.gitops.config.schema; + +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import java.util.Map; +import java.util.stream.Stream; + +import static org.junit.jupiter.api.Assertions.assertThrows; + +class JsonConfigValidatorTest { + + static Stream validSchemas() { + return Stream.of( + Arguments.of( + "multiple values", + Map.of("features", Map.of("argocd", Map.of("url", "http://localhost/argocd"))) + ) + ); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("validSchemas") + void testValidSchemas(String description, Map schema) { + JsonSchemaValidator.validate(schema); + } + + static Stream invalidSchemas() { + return Stream.of( + Arguments.of( + "wrong type for registry.internalPort", + Map.of("registry", Map.of("internalPort", "this should be a number")) + ), + Arguments.of( + "invalid additional key within registry", + Map.of("registry", Map.of("url", "", "unexpectedKey", "this should error")) + ), + Arguments.of( + "invalid additional key on root level", + Map.of( + "registry", Map.of("url", ""), + "unexpectedKey", "this should not exist" + ) + ), + Arguments.of( + "specifying dynamic value", + Map.of( + "application", Map.of( + "namePrefix", "prefix", + "namePrefixForEnvVars", "prefix" + ) + ) + ) + ); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("invalidSchemas") + void testInvalidSchemas(String description, Map schema) { + assertThrows(RuntimeException.class, () -> JsonSchemaValidator.validate(schema)); + } +} diff --git a/src/test/java/com/cloudogu/gitops/config/schema/JsonSchemaGeneratorTest.java b/src/test/java/com/cloudogu/gitops/config/schema/JsonSchemaGeneratorTest.java new file mode 100644 index 000000000..173211ae3 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/config/schema/JsonSchemaGeneratorTest.java @@ -0,0 +1,27 @@ +package com.cloudogu.gitops.config.schema; + +import org.junit.jupiter.api.Test; +import tools.jackson.databind.ObjectMapper; + +import java.io.File; +import java.io.IOException; + +import static org.assertj.core.api.Assertions.assertThat; + +class JsonSchemaGeneratorTest { + + @Test + void configurationSchemaIsNotOutOfDate() throws IOException { + ObjectMapper objectMapper = new ObjectMapper(); + String expected = objectMapper.writeValueAsString( + objectMapper.readTree(new JsonSchemaGenerator().createSchema().toString()) + ); + String actual = objectMapper.writeValueAsString( + objectMapper.readTree(new File(System.getProperty("user.dir"), "docs/configuration.schema.json")) + ); + + assertThat(actual) + .as("Config in docs/configuration.schema.json must be updated. Run GenerateJsonSchema class.") + .isEqualTo(expected); + } +} diff --git a/src/test/java/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptorTest.java b/src/test/java/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptorTest.java new file mode 100644 index 000000000..9d4c325fc --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/dependencyinjection/okhttp/RetryInterceptorTest.java @@ -0,0 +1,181 @@ +package com.cloudogu.gitops.dependencyinjection.okhttp; + +import com.github.tomakehurst.wiremock.junit5.WireMockExtension; +import okhttp3.Interceptor; +import okhttp3.OkHttpClient; +import okhttp3.Protocol; +import okhttp3.Request; +import okhttp3.Response; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.TrustManager; +import javax.net.ssl.X509TrustManager; +import java.io.IOException; +import java.net.SocketTimeoutException; +import java.security.GeneralSecurityException; +import java.security.SecureRandom; +import java.security.cert.X509Certificate; +import java.util.concurrent.TimeUnit; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.get; +import static com.github.tomakehurst.wiremock.client.WireMock.getRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo; +import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class RetryInterceptorTest { + + private static final int OKHTTPCLIENT_TIMEOUT = 1000; + + @RegisterExtension + static final WireMockExtension wireMock = WireMockExtension.newInstance() + .options(wireMockConfig() + .dynamicPort() + .dynamicHttpsPort()) + .build(); + + @BeforeEach + void resetWireMock() { + wireMock.resetAll(); + } + + @Test + void retriesThreeTimesOn500() throws IOException, GeneralSecurityException { + String path = "/retry-500"; + + wireMock.stubFor(get(urlEqualTo(path)) + .inScenario("Retry Scenario") + .whenScenarioStateIs("Started") + .willReturn(aResponse().withStatus(500)) + .willSetStateTo("First Retry")); + + wireMock.stubFor(get(urlEqualTo(path)) + .inScenario("Retry Scenario") + .whenScenarioStateIs("First Retry") + .willReturn(aResponse().withStatus(500)) + .willSetStateTo("Second Retry")); + + wireMock.stubFor(get(urlEqualTo(path)) + .inScenario("Retry Scenario") + .whenScenarioStateIs("Second Retry") + .willReturn(aResponse() + .withStatus(200) + .withBody("Successful Result"))); + + OkHttpClient client = createClient(); + Response response = client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute(); + assertThat(response.body().string()).isEqualTo("Successful Result"); + wireMock.verify(3, getRequestedFor(urlEqualTo(path))); + } + + @Test + void retriesThreeTimesOn500WithHttps() throws IOException, GeneralSecurityException { + String path = "/retry-500"; + + wireMock.stubFor(get(urlEqualTo(path)) + .inScenario("HTTPS Retry Scenario") + .whenScenarioStateIs("Started") + .willReturn(aResponse().withStatus(500)) + .willSetStateTo("First Retry")); + + wireMock.stubFor(get(urlEqualTo(path)) + .inScenario("HTTPS Retry Scenario") + .whenScenarioStateIs("First Retry") + .willReturn(aResponse().withStatus(500)) + .willSetStateTo("Second Retry")); + + wireMock.stubFor(get(urlEqualTo(path)) + .inScenario("HTTPS Retry Scenario") + .whenScenarioStateIs("Second Retry") + .willReturn(aResponse() + .withStatus(200) + .withBody("Successful Result"))); + + OkHttpClient client = createClient(); + Response response = client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute(); + assertThat(response.body().string()).isEqualTo("Successful Result"); + wireMock.verify(3, getRequestedFor(urlEqualTo(path))); + } + + @Test + void retriesOnTimeout() throws IOException { + Request request = new Request.Builder().url("http://localhost/timeout-test").build(); + Interceptor.Chain chain = mock(Interceptor.Chain.class); + Response successfulResponse = new Response.Builder() + .request(request) + .protocol(Protocol.HTTP_1_1) + .code(200) + .message("OK") + .build(); + + when(chain.request()).thenReturn(request); + when(chain.proceed(request)) + .thenThrow(new SocketTimeoutException("Read timed out")) + .thenReturn(successfulResponse); + + try (Response response = new RetryInterceptor(3, 0).intercept(chain)) { + assertThat(response.code()).isEqualTo(200); + } + verify(chain, times(2)).proceed(request); + } + + @Test + void failsAfterThirdRetry() throws GeneralSecurityException { + String path = "/always-fail"; + + wireMock.stubFor(get(urlEqualTo(path)) + .willReturn(aResponse().withStatus(500))); + + OkHttpClient client = createClient(); + + IOException exception = assertThrows( + IOException.class, () -> + client.newCall(new Request.Builder().url(wireMock.baseUrl() + path).build()).execute() + ); + + assertThat(exception.getMessage()).contains("500"); + wireMock.verify(4, getRequestedFor(urlEqualTo(path))); + } + + private OkHttpClient createClient() throws GeneralSecurityException { + return createClient(OKHTTPCLIENT_TIMEOUT); + } + + private OkHttpClient createClient(int timeout) throws GeneralSecurityException { + X509TrustManager trustManager = new X509TrustManager() { + @Override + public void checkClientTrusted(X509Certificate[] chain, String authType) { + } + + @Override + public void checkServerTrusted(X509Certificate[] chain, String authType) { + } + + @Override + public X509Certificate[] getAcceptedIssuers() { + return new X509Certificate[0]; + } + }; + TrustManager[] trustAllCerts = new TrustManager[]{trustManager}; + + SSLContext sslContext = SSLContext.getInstance("TLS"); + sslContext.init(null, trustAllCerts, new SecureRandom()); + + return new OkHttpClient.Builder() + .addInterceptor(new RetryInterceptor(3, 0)) + .connectTimeout(timeout, TimeUnit.MILLISECONDS) + .readTimeout(timeout, TimeUnit.MILLISECONDS) + .sslSocketFactory(sslContext.getSocketFactory(), trustManager) + .hostnameVerifier((hostname, session) -> true) + .build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/destroy/DestroyerDependencyInjectionTest.java b/src/test/java/com/cloudogu/gitops/destroy/DestroyerDependencyInjectionTest.java new file mode 100644 index 000000000..f20e08dcb --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/destroy/DestroyerDependencyInjectionTest.java @@ -0,0 +1,37 @@ +package com.cloudogu.gitops.destroy; + +import com.cloudogu.gitops.config.Config; +import io.micronaut.context.ApplicationContext; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class DestroyerDependencyInjectionTest { + + @Test + void canCreateBean() { + Config config = Config.fromMap(Map.of( + "scm", Map.of( + "scmManager", Map.of( + "url", "http://localhost:9091/scm", + "username", "admin", + "password", "admin" + ) + ), + "jenkins", Map.of( + "url", "http://localhost:9090", + "username", "admin", + "password", "admin" + ), + "application", Map.of("insecure", true) + )); + + Destroyer destroyer = ApplicationContext.run() + .registerSingleton(config) + .getBean(Destroyer.class); + + assertThat(destroyer.getDestructionHandlers()).hasSize(3); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategyTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategyTest.java new file mode 100644 index 000000000..537c57f3f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationStrategyTest.java @@ -0,0 +1,357 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.config.scm.ScmTenantSchema.ScmManagerTenantConfig; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class ArgoCdApplicationStrategyTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + + private File localTempDir; + private DeploymentContext context; + private RepositoryWorkspace repositoryWorkspace; + + @Test + void deploysFeatureUsingArgoCd() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(); + File valuesYaml = File.createTempFile("values", "yaml"); + + strategy.deployFeature( + "repoURL", + "repoName", + "chartName", + "version", + "foo-namespace", + "releaseName", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + File argoCdApplicationYaml = new File(localTempDir, "apps/argocd/applications/releaseName.yaml"); + + assertThat(Files.readString(argoCdApplicationYaml.toPath())).isEqualTo(""" + --- + apiVersion: "argoproj.io/v1alpha1" + kind: "Application" + metadata: + name: "foo-repoName" + namespace: "foo-argocd" + spec: + destination: + server: "https://kubernetes.default.svc" + namespace: "foo-namespace" + project: "cluster-resources" + sources: + - repoURL: "repoURL" + chart: "chartName" + targetRevision: "version" + helm: + releaseName: "releaseName" + valueFiles: + - "$values/apps/repoName/repoName-gop-helm.yaml" + - "$values/apps/repoName/repoName-user-values.yaml" + ignoreMissingValueFiles: true + - repoURL: "http://scmm.scm-manager.svc.cluster.local/scm/repo/argocd/cluster-resources.git" + targetRevision: "main" + ref: "values" + path: "apps/repoName" + directory: + recurse: true + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - "ServerSideApply=true" + - "CreateNamespace=true" + """); + } + + @Test + @SuppressWarnings("unchecked") + void deploysFeatureUsingArgoCdFromGitRepo() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(); + File valuesYaml = File.createTempFile("values", "yaml"); + + strategy.deployFeature( + "repoURL", + "repoName", + "chartName", + "version", + "namespace", + "releaseName", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.GIT, + context, + repositoryWorkspace + ); + + File argoCdApplicationYaml = new File(localTempDir, "apps/argocd/applications/releaseName.yaml"); + Map result = YAML_MAPPER.readValue(argoCdApplicationYaml, YAML_MAP_TYPE); + Map spec = (Map) result.get("spec"); + List> sources = (List>) spec.get("sources"); + + assertThat(sources.get(0)).containsKey("path"); + assertThat(sources.get(0).get("path")).isEqualTo("chartName"); + } + + @Test + void deploysFeatureWithArgoCdOperatorTrueSettingCreateNamespaceToFalse() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(true); + File valuesYaml = File.createTempFile("values", "yaml"); + Files.writeString( + valuesYaml.toPath(), """ + param1: value1 + param2: value2 + """ + ); + + strategy.deployFeature( + "repoURL", + "repoName", + "chartName", + "version", + "namespace", + "releaseName", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + File argoCdApplicationYaml = new File(localTempDir, "apps/argocd/applications/releaseName.yaml"); + + assertThat(Files.readString(argoCdApplicationYaml.toPath())).contains("CreateNamespace=false"); + } + + @Test + void deploysFeatureWithArgoCdOperatorFalseSettingCreateNamespaceToTrue() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(false); + File valuesYaml = File.createTempFile("values", "yaml"); + Files.writeString( + valuesYaml.toPath(), """ + param1: value1 + param2: value2 + """ + ); + + strategy.deployFeature( + "repoURL", + "repoName", + "chartName", + "version", + "namespace", + "releaseName", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + File argoCdApplicationYaml = new File(localTempDir, "apps/argocd/applications/releaseName.yaml"); + + assertThat(Files.readString(argoCdApplicationYaml.toPath())).contains("CreateNamespace=true"); + } + + @Test + @SuppressWarnings("unchecked") + void deploysScmManagerAsBootstrapApplicationWithoutValuesSource() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(); + File valuesYaml = File.createTempFile("values", "yaml"); + Files.writeString( + valuesYaml.toPath(), """ + fullnameOverride: tenant1-scmm + service: + type: NodePort + """ + ); + + strategy.deployFeature( + "repoURL", + "scm-manager", + "scm-manager", + "3.11.6", + "tenant1-scm-manager", + "tenant1-scmm", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + File argoCdApplicationYaml = new File(localTempDir, "apps/argocd/applications/tenant1-scmm.yaml"); + Map result = YAML_MAPPER.readValue(argoCdApplicationYaml, YAML_MAP_TYPE); + Map spec = (Map) result.get("spec"); + List> sources = (List>) spec.get("sources"); + Map helm = (Map) sources.get(0).get("helm"); + + assertThat(sources).hasSize(1); + assertThat(sources.get(0).get("repoURL")).isEqualTo("repoURL"); + assertThat(sources.get(0).get("chart")).isEqualTo("scm-manager"); + assertThat(helm.get("releaseName")).isEqualTo("tenant1-scmm"); + assertThat(helm.get("values").toString()).contains("fullnameOverride: tenant1-scmm"); + } + + @Test + void deploysScmManagerAsBootstrapApplicationWithoutWritingExternalValueFiles() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(); + File valuesYaml = File.createTempFile("values", "yaml"); + Files.writeString( + valuesYaml.toPath(), """ + fullnameOverride: tenant1-scmm + """ + ); + + strategy.deployFeature( + "repoURL", + "scm-manager", + "scm-manager", + "3.11.6", + "tenant1-scm-manager", + "tenant1-scmm", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + assertThat(new File(localTempDir, "apps/scm-manager/scm-manager-gop-helm.yaml")).doesNotExist(); + assertThat(new File(localTempDir, "apps/scm-manager/scm-manager-user-values.yaml")).doesNotExist(); + } + + @Test + void deploysNormalFeatureWithGopAndUserValuesFiles() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(); + File valuesYaml = File.createTempFile("values", "yaml"); + Files.writeString( + valuesYaml.toPath(), """ + param1: value1 + """ + ); + + strategy.deployFeature( + "repoURL", + "repoName", + "chartName", + "version", + "namespace", + "releaseName", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + assertThat(Files.readString(new File(localTempDir, "apps/repoName/repoName-gop-helm.yaml").toPath())) + .contains("param1: value1"); + + assertThat(new File(localTempDir, "apps/repoName/repoName-user-values.yaml")).exists(); + } + + @Test + @SuppressWarnings("unchecked") + void usesWorkspaceClusterResourcesRepositoryAsValuesSource() throws IOException { + ArgoCdApplicationStrategy strategy = createStrategy(); + File valuesYaml = File.createTempFile("values", "yaml"); + + strategy.deployFeature( + "repoURL", + "repoName", + "chartName", + "version", + "namespace", + "releaseName", + valuesYaml.toPath(), + DeploymentStrategy.RepoType.HELM, + context, + repositoryWorkspace + ); + + File argoCdApplicationYaml = new File(localTempDir, "apps/argocd/applications/releaseName.yaml"); + Map result = YAML_MAPPER.readValue(argoCdApplicationYaml, YAML_MAP_TYPE); + Map spec = (Map) result.get("spec"); + List> sources = (List>) spec.get("sources"); + + assertThat(sources.get(1).get("repoURL")) + .isEqualTo("http://scmm.scm-manager.svc.cluster.local/scm/repo/argocd/cluster-resources.git"); + + assertThat(sources.get(1).get("path")).isEqualTo("apps/repoName"); + } + + private ArgoCdApplicationStrategy createStrategy() { + return createStrategy(false); + } + + private ArgoCdApplicationStrategy createStrategy(boolean argocdOperator) { + Config config = new Config(); + + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setNamePrefix("foo-"); + application.setGitName("Cloudogu"); + application.setGitEmail("hello@cloudogu.com"); + config.setApplication(application); + + ScmManagerTenantConfig scmManager = new ScmManagerTenantConfig(); + scmManager.setUsername("dont-care-username"); + scmManager.setPassword("dont-care-password"); + ScmTenantSchema scm = new ScmTenantSchema(); + scm.setScmManager(scmManager); + config.setScm(scm); + + Config.ArgoCDSchema argoCd = new Config.ArgoCDSchema(); + argoCd.setOperator(argocdOperator); + Config.FeaturesSchema features = new Config.FeaturesSchema(); + features.setArgocd(argoCd); + config.setFeatures(features); + + ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + + TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, new FileSystemUtils()) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, scmManagerMock); + + assertThat(repo) + .as("TestGitRepoFactory must create cluster-resources GitRepo") + .isNotNull(); + + localTempDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoProvider.create("argocd/cluster-resources", scmManagerMock); + + repositoryWorkspace = new RepositoryWorkspace(clusterResourcesRepo); + context = new ContextBuilder(config).build(); + + ArgoCdApplicationTargetResolver targetResolver = new ArgoCdApplicationTargetResolver(config); + + return new ArgoCdApplicationStrategy(targetResolver); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTargetResolverTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTargetResolverTest.java new file mode 100644 index 000000000..4fd6dd36e --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/ArgoCdApplicationTargetResolverTest.java @@ -0,0 +1,67 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.MultiTenantSchema; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class ArgoCdApplicationTargetResolverTest { + + @Test + void resolvesTargetForSingleTenantDeployment() { + Config config = createConfig(); + + ArgoCdApplicationTarget target = new ArgoCdApplicationTargetResolver(config) + .resolve(new ContextBuilder(config).build(), "repo-name"); + + assertThat(target.getApplicationName()).isEqualTo("foo-repo-name"); + assertThat(target.getNamespace()).isEqualTo("foo-argocd"); + assertThat(target.getProject()).isEqualTo("cluster-resources"); + assertThat(target.isCreateDestinationNamespace()).isTrue(); + } + + @Test + void resolvesTargetForMultiTenantDeployment() { + Config config = createConfig(); + config.getMultiTenant().setUseDedicatedInstance(true); + config.getMultiTenant().setCentralArgocdNamespace("central-argocd"); + + ArgoCdApplicationTarget target = new ArgoCdApplicationTargetResolver(config) + .resolve(new ContextBuilder(config).build(), "repo-name"); + + assertThat(target.getApplicationName()).isEqualTo("foo-repo-name"); + assertThat(target.getNamespace()).isEqualTo("central-argocd"); + assertThat(target.getProject()).isEqualTo("foo"); + assertThat(target.isCreateDestinationNamespace()).isTrue(); + } + + @Test + void disablesDestinationNamespaceCreationInOperatorMode() { + Config config = createConfig(); + config.getFeatures().getArgocd().setOperator(true); + + ArgoCdApplicationTarget target = new ArgoCdApplicationTargetResolver(config) + .resolve(new ContextBuilder(config).build(), "repo-name"); + + assertThat(target.isCreateDestinationNamespace()).isFalse(); + } + + private static Config createConfig() { + Config config = new Config(); + + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setNamePrefix("foo-"); + config.setApplication(application); + + Config.ArgoCDSchema argoCd = new Config.ArgoCDSchema(); + argoCd.setNamespace("argocd"); + Config.FeaturesSchema features = new Config.FeaturesSchema(); + features.setArgocd(argoCd); + config.setFeatures(features); + + config.setMultiTenant(new MultiTenantSchema()); + return config; + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/deployment/DeployerTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/DeployerTest.java new file mode 100644 index 000000000..fe7f10e89 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/DeployerTest.java @@ -0,0 +1,124 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import jakarta.inject.Provider; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.InOrder; + +import java.nio.file.Path; + +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.verifyNoMoreInteractions; +import static org.mockito.Mockito.when; + +class DeployerTest { + + private static final String REPO_URL = "https://example.com/repo.git"; + private static final String REPO_NAME = "repo-name"; + private static final String CHART_OR_PATH = "chart-or-path"; + private static final String VERSION = "1.2.3"; + private static final String NAMESPACE = "namespace"; + private static final String RELEASE_NAME = "release-name"; + private static final RepoType REPO_TYPE = RepoType.HELM; + + private Provider argoCdStrategyProvider; + private ArgoCdApplicationStrategy argoCdStrategy; + private HelmStrategy helmStrategy; + private Path helmValuesPath; + private Deployer deployer; + private DeploymentContext context; + private RepositoryWorkspace workspace; + + @BeforeEach + @SuppressWarnings("unchecked") + void setup() { + argoCdStrategyProvider = mock(Provider.class); + argoCdStrategy = mock(ArgoCdApplicationStrategy.class); + helmStrategy = mock(HelmStrategy.class); + helmValuesPath = mock(Path.class); + context = mock(DeploymentContext.class); + workspace = mock(RepositoryWorkspace.class); + + deployer = new Deployer(argoCdStrategyProvider, helmStrategy); + } + + @Test + void deploysViaArgoCdWhenArgoCdIsEnabledAndInitByHelmIsDisabled() { + when(argoCdStrategyProvider.get()).thenReturn(argoCdStrategy); + + deployFeature(false); + + verify(argoCdStrategyProvider).get(); + verify(argoCdStrategy).deployFeature( + REPO_URL, + REPO_NAME, + CHART_OR_PATH, + VERSION, + NAMESPACE, + RELEASE_NAME, + helmValuesPath, + REPO_TYPE, + context, + workspace + ); + verifyNoInteractions(helmStrategy); + verifyNoMoreInteractions(argoCdStrategyProvider, argoCdStrategy); + } + + @Test + void deploysViaHelmBeforeArgoCdWhenArgoCdIsEnabledAndInitByHelmIsEnabled() { + when(argoCdStrategyProvider.get()).thenReturn(argoCdStrategy); + + deployFeature(true); + + InOrder inOrder = inOrder(helmStrategy, argoCdStrategyProvider, argoCdStrategy); + inOrder.verify(helmStrategy).deployFeature( + REPO_URL, + REPO_NAME, + CHART_OR_PATH, + VERSION, + NAMESPACE, + RELEASE_NAME, + helmValuesPath, + REPO_TYPE, + context, + workspace + ); + inOrder.verify(argoCdStrategyProvider).get(); + inOrder.verify(argoCdStrategy).deployFeature( + REPO_URL, + REPO_NAME, + CHART_OR_PATH, + VERSION, + NAMESPACE, + RELEASE_NAME, + helmValuesPath, + REPO_TYPE, + context, + workspace + ); + verifyNoMoreInteractions(helmStrategy, argoCdStrategyProvider, argoCdStrategy); + } + + private void deployFeature(boolean initByHelm) { + deployer.deployFeature( + REPO_URL, + REPO_NAME, + CHART_OR_PATH, + VERSION, + NAMESPACE, + RELEASE_NAME, + helmValuesPath, + REPO_TYPE, + initByHelm, + context, + workspace + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/deployment/HelmStrategyTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/HelmStrategyTest.java new file mode 100644 index 000000000..94b5bd79f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/deployment/HelmStrategyTest.java @@ -0,0 +1,88 @@ +package com.cloudogu.gitops.infrastructure.deployment; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; + +class HelmStrategyTest { + + private final HelmClient helmClient = mock(HelmClient.class); + + @Test + void deploysFeatureUsingHelmClient() throws IOException { + Path valuesYaml = Files.createTempFile("", ""); + DeploymentContext context = new ContextBuilder(createConfig()).build(); + + createStrategy().deployFeature( + "repoURL", + "repoName", + "chart", + "version", + "foo-namespace", + "releaseName", + valuesYaml, + DeploymentStrategy.RepoType.HELM, + context, + null + ); + + verify(helmClient).addRepo("repoName", "repoURL"); + verify(helmClient).upgrade( + "releaseName", + "repoName/chart", + Map.of( + "namespace", "foo-namespace", + "version", "version", + "values", valuesYaml.toString() + ) + ); + } + + @Test + void failsToDeployFromGit() { + DeploymentContext context = new ContextBuilder(createConfig()).build(); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> createStrategy().deployFeature( + "http://repoURL", + "repoName", + "chart", + "version", + "namespace", + "releaseName", + Path.of("values.yaml"), + DeploymentStrategy.RepoType.GIT, + context, + null + ) + ); + + assertThat(exception.getMessage()).isEqualTo( + "Unable to deploy helm chart via Helm CLI from Git URL, because helm does not support this out of the box.\n" + + "Repo URL: http://repoURL" + ); + } + + protected HelmStrategy createStrategy() { + return new HelmStrategy(helmClient); + } + + private Config createConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("foo-"); + return config; + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/git/GitRepoFactoryTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/git/GitRepoFactoryTest.java new file mode 100644 index 000000000..7c835228f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/git/GitRepoFactoryTest.java @@ -0,0 +1,58 @@ +package com.cloudogu.gitops.infrastructure.git; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.utils.FileSystemUtils; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class GitRepoFactoryTest { + + private final Config config = createConfig(); + private final GitRepoFactory factory = new GitRepoFactory(config, new FileSystemUtils()); + + @Test + void createsRepoWithEmptyNamePrefix() { + GitRepo repo = factory.create("expectedRepoTarget", new ScmManagerProviderMock()); + + assertThat(repo.getRepoTarget()).isEqualTo("expectedRepoTarget"); + } + + @Test + void createsRepoWithNamePrefix() { + config.getApplication().setNamePrefix("abc-"); + + GitRepo repo = factory.create("expectedRepoTarget", new ScmManagerProviderMock()); + + assertThat(repo.getRepoTarget()).isEqualTo("abc-expectedRepoTarget"); + } + + @Test + void createsRepoWithNamePrefixWhenInNamespaceThirdPartyDependencies() { + config.getApplication().setNamePrefix("abc-"); + + GitRepo repo = factory.create( + GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES + "/foo", + new ScmManagerProviderMock() + ); + + assertThat(repo.getRepoTarget()).isEqualTo( + config.getApplication().getNamePrefix() + GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES + "/foo" + ); + } + + private static Config createConfig() { + Config config = new Config(); + config.getApplication().setGitName("Cloudogu"); + config.getApplication().setGitEmail("hello@cloudogu.com"); + + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setUsername("dont-care-username"); + scmManager.setPassword("dont-care-password"); + config.getScm().setScmManager(scmManager); + + return config; + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/git/GitRepoTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/git/GitRepoTest.java new file mode 100644 index 000000000..b1e17f680 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/git/GitRepoTest.java @@ -0,0 +1,218 @@ +package com.cloudogu.gitops.infrastructure.git; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.utils.FileSystemUtils; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.revwalk.RevCommit; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mock; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class GitRepoTest { + + public static final String expectedNamespace = "namespace"; + public static final String expectedRepo = "repo"; + + private final Config config = Config.fromMap(Map.of( + "application", Map.of( + "gitName", "Cloudogu", + "gitEmail", "hello@cloudogu.com" + ), + "scm", Map.of( + "scmManager", Map.of( + "username", "dont-care-username", + "password", "dont-care-password" + ) + ) + )); + + private final TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, new FileSystemUtils()); + + @Mock + GitProvider gitProvider; + + private ScmManagerProviderMock scmManagerMock; + + @BeforeEach + void setup() { + scmManagerMock = new ScmManagerProviderMock(); + } + + @Test + void writesFile() throws IOException { + GitRepo repo = getRepo("", scmManagerMock); + repo.writeFile("test.txt", "the file's content"); + + File expectedFile = new File(repo.getAbsoluteLocalRepoTmpDir(), "test.txt"); + assertThat(Files.readString(expectedFile.toPath())).isEqualTo("the file's content"); + } + + @Test + void overwritesFile() throws IOException { + GitRepo repo = getRepo("", scmManagerMock); + String tempDir = repo.getAbsoluteLocalRepoTmpDir(); + + File existingFile = new File(tempDir, "already-exists.txt"); + existingFile.createNewFile(); + Files.writeString(existingFile.toPath(), "already existing content"); + + repo.writeFile("already-exists.txt", "overwritten content"); + + File expectedFile = new File(tempDir, "already-exists.txt"); + assertThat(Files.readString(expectedFile.toPath())).isEqualTo("overwritten content"); + } + + @Test + void writesFileAndCreatesSubdirectory() throws IOException { + GitRepo repo = getRepo("", scmManagerMock); + String tempDir = repo.getAbsoluteLocalRepoTmpDir(); + repo.writeFile("subdirectory/test.txt", "the file's content"); + + File expectedFile = new File(tempDir, "subdirectory/test.txt"); + assertThat(Files.readString(expectedFile.toPath())).isEqualTo("the file's content"); + } + + @Test + void throwsErrorWhenDirectoryConflictsWithExistingFile() { + GitRepo repo = getRepo("", scmManagerMock); + String tempDir = repo.getAbsoluteLocalRepoTmpDir(); + new File(tempDir, "test.txt").mkdir(); + + assertThrows(FileNotFoundException.class, () -> repo.writeFile("test.txt", "the file's content")); + } + + @Test + void usesRepositoryTargetAsProvided() { + config.getApplication().setNamePrefix("abc-"); + + GitRepo repo = new GitRepo(config, scmManagerMock, "expectedRepoTarget", new FileSystemUtils()); + + assertThat(repo.getRepoTarget()).isEqualTo("expectedRepoTarget"); + } + + @Test + void clonesAndChecksOutMain() throws GitAPIException, IOException { + GitRepo repo = getRepo("", scmManagerMock); + + repo.cloneRepo(); + File head = new File(repo.getAbsoluteLocalRepoTmpDir(), ".git/HEAD"); + assertThat(Files.readString(head.toPath())).isEqualTo("ref: refs/heads/main\n"); + assertThat(new File(repo.getAbsoluteLocalRepoTmpDir(), "README.md")).exists(); + } + + @Test + void pushesChangesToRemoteDirectory() throws GitAPIException, IOException { + GitRepo repo = getRepo("", scmManagerMock); + + repo.cloneRepo(); + File readme = new File(repo.getAbsoluteLocalRepoTmpDir(), "README.md"); + Files.writeString(readme.toPath(), "This text should be in the readme afterwards"); + repo.commitAndPush("The commit message"); + + List commits = new ArrayList<>(); + Git.open(new File(repo.getAbsoluteLocalRepoTmpDir())) + .log().setMaxCount(1).all().call().forEach(commits::add); + assertThat(commits.size()).isEqualTo(1); + assertThat(commits.get(0).getFullMessage()).isEqualTo("The commit message"); + assertThat(commits.get(0).getAuthorIdent().getEmailAddress()).isEqualTo("hello@cloudogu.com"); + assertThat(commits.get(0).getAuthorIdent().getName()).isEqualTo("Cloudogu"); + assertThat(commits.get(0).getCommitterIdent().getEmailAddress()).isEqualTo("hello@cloudogu.com"); + assertThat(commits.get(0).getCommitterIdent().getName()).contains("Cloudogu - GOP v"); + + List tags = Git.open(new File(repo.getAbsoluteLocalRepoTmpDir())).tagList().call(); + assertThat(tags.size()).isEqualTo(0); + } + + @Test + void pushesChangesToRemoteDirectoryWithTag() throws GitAPIException, IOException { + GitRepo repo = getRepo("", scmManagerMock); + String expectedTag = "1.0"; + + repo.cloneRepo(); + File readme = new File(repo.getAbsoluteLocalRepoTmpDir(), "README.md"); + Files.writeString(readme.toPath(), "This text should be in the readme afterwards"); + // Create existing tag to test for idempotence + Git.open(new File(repo.getAbsoluteLocalRepoTmpDir())).tag().setName(expectedTag).call(); + + repo.commitAndPush("The commit message", expectedTag); + + List tags = Git.open(new File(repo.getAbsoluteLocalRepoTmpDir())).tagList().call(); + assertThat(tags.size()).isEqualTo(1); + assertThat(tags.get(0).getName()).isEqualTo("refs/tags/" + expectedTag); + // It would be a good idea to check if the git tag is set on the commit. + // However, it's extremely complicated with jgit + // The "official" example code throws an exception here: Ref peeledRef = repository.getRefDatabase().peel(ref) + // https://github.com/centic9/jgit-cookbook/blob/d923e18b2ce2e55761858fd2e8e402dd252e0766/src/main/java/org/dstadler/jgit/porcelain/ListTags.java + // 🤷 + } + + @Test + void createsRepositoryAndSetsPermissionWhenNewAndUsernamePresent() { + String repoTarget = "foo/bar"; + GitRepo repo = getRepo(repoTarget, scmManagerMock); + scmManagerMock.setNextCreateResults(new ArrayList<>(List.of(true))); // simulate "new repo" + scmManagerMock.setGitOpsUsername("foo-gitops"); // username available + + boolean created = repo.createRepositoryAndSetPermission("testdescription", true); + + assertThat(created).isTrue(); + + // Verify that repo was created + assertThat(scmManagerMock.getCreatedRepos()).containsExactly(repoTarget); + + // Verify permission call + assertThat(scmManagerMock.getPermissionCalls()).hasSize(1); + Map call = scmManagerMock.getPermissionCalls().get(0); + assertThat(call.get("repoTarget")).isEqualTo(repoTarget); + assertThat(call.get("principal")).isEqualTo("foo-gitops"); + assertThat(call.get("role")).isEqualTo(AccessRole.WRITE); + assertThat(call.get("scope")).isEqualTo(Scope.USER); + } + + @Test + void doesNotSetPermissionWhenNoGitOpsUsernameIsConfigured() { + String repoTarget = "foo/bar"; + ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + GitRepo repo = getRepo(repoTarget, scmManagerMock); + + scmManagerMock.setNextCreateResults(new ArrayList<>(List.of(true))); // repo is new + scmManagerMock.setGitOpsUsername(null); // no username + + boolean created = repo.createRepositoryAndSetPermission("desc", true); + + assertThat(created).isTrue(); + + // Repo created + assertThat(scmManagerMock.getCreatedRepos()).containsExactly(repoTarget); + + // No permission calls because username missing + assertThat(scmManagerMock.getPermissionCalls()).isEmpty(); + } + + private GitRepo getRepo(String repoTarget, ScmManagerProviderMock scmManagerMock) { + return repoProvider.create(repoTarget, scmManagerMock); + } + + @SuppressWarnings("unused") + private GitRepo getRepo(ScmManagerProviderMock scmManagerMock) { + return getRepo(expectedNamespace + "/" + expectedRepo, scmManagerMock); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerProviderTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerProviderTest.java new file mode 100644 index 000000000..fb4d9de29 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerProviderTest.java @@ -0,0 +1,214 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.ScmManagerConfig; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.RepositoryApi; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.NetworkingUtils; +import okhttp3.internal.http.RealResponseBody; +import okio.BufferedSource; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import retrofit2.Call; +import retrofit2.Response; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.net.URI; +import java.net.URISyntaxException; +import java.util.HashSet; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.argThat; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.atLeastOnce; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class ScmManagerProviderTest { + + @Mock + ScmManagerConfig scmmCfg; + @Mock + ScmManagerUrlResolver urls; + @Mock + ScmManagerApiClient apiClient; + @Mock + RepositoryApi repoApi; + @Mock + K8sClient k8s; + @Mock + NetworkingUtils net; + + @BeforeEach + void setup() throws URISyntaxException { + lenient().when(scmmCfg.getGitOpsUsername()).thenReturn("gitops-bot"); + + lenient().when(urls.inClusterBase()).thenReturn(new URI("http://scmm.ns.svc.cluster.local/scm")); + lenient().when(urls.inClusterRepoPrefix()).thenReturn("http://scmm.ns.svc.cluster.local/scm/repo/fv40-"); + lenient().when(urls.clientApiBase()).thenReturn(new URI("http://nodeport/scm/api/v2/")); + + lenient().when(apiClient.repositoryApi()).thenReturn(repoApi); + } + + private ScmManagerProvider newScmManager() throws ReflectiveOperationException { + ScmManagerProvider scmManager = new ScmManagerProvider( + scmmCfg, new Credentials("user", "password"), k8s, net, "fv40-", true, false, "fv40-" + ); + setField(scmManager, "urls", urls); + setField(scmManager, "apiClient", apiClient); + return scmManager; + } + + private static void setField(ScmManagerProvider scmManager, String fieldName, Object value) + throws ReflectiveOperationException { + Field field = ScmManagerProvider.class.getDeclaredField(fieldName); + field.setAccessible(true); + field.set(scmManager, value); + } + + private static Call callReturningSuccess(int code) throws IOException { + @SuppressWarnings("unchecked") + Call call = mock(Call.class); + when(call.execute()).thenReturn(Response.success(code, null)); + return call; + } + + private static Call callReturningError(int code) throws IOException { + @SuppressWarnings("unchecked") + Call call = mock(Call.class); + RealResponseBody body = new RealResponseBody("ignored", 0, mock(BufferedSource.class)); + when(call.execute()).thenReturn(Response.error(code, body)); + return call; + } + + @Test + void createRepositoryReturnsTrueOn201AndFalseOnSubsequent409ForTheSameRepo() + throws IOException, ReflectiveOperationException { + ScmManagerProvider scmManager = newScmManager(); + + Call created = callReturningSuccess(201); + Call conflict = callReturningError(409); + Set seen = new HashSet<>(); + + when(repoApi.create(any(Repository.class), anyBoolean())) + .thenAnswer(inv -> { + Repository repository = inv.getArgument(0); + if (seen.contains(repository.getFullRepoName())) { + return conflict; + } + + seen.add(repository.getFullRepoName()); + return created; + }); + + assertTrue(scmManager.createRepository("team/demo", "Demo repo", true)); + assertFalse(scmManager.createRepository("team/demo", "Demo repo", true)); + assertTrue(scmManager.createRepository("team/other", null, false)); + + verify(repoApi, times(3)).create(any(Repository.class), anyBoolean()); + } + + @Test + void setRepositoryPermissionMapsMaintainToWriteAndHandles201409() + throws IOException, ReflectiveOperationException { + ScmManagerProvider scmManager = newScmManager(); + + Call created = callReturningSuccess(201); + Call conflict = callReturningError(409); + Set seen = new HashSet<>(); + + when(repoApi.createPermission(anyString(), anyString(), any(Permission.class))) + .thenAnswer(inv -> { + String namespace = inv.getArgument(0); + String repoName = inv.getArgument(1); + String key = namespace + "/" + repoName; + + if (seen.contains(key)) { + return conflict; + } + + seen.add(key); + return created; + }); + + assertDoesNotThrow(() -> + scmManager.setRepositoryPermission("namespace/repo1", "devs", AccessRole.MAINTAIN, Scope.GROUP) + ); + + assertDoesNotThrow(() -> + scmManager.setRepositoryPermission("namespace/repo1", "devs", AccessRole.MAINTAIN, Scope.GROUP) + ); + + verify(repoApi, atLeastOnce()).createPermission( + eq("namespace"), + eq("repo1"), + argThat(permission -> permission.groupPermission() && permission.role() == Permission.Role.WRITE) + ); + } + + @Test + void urlRepoPrefixRepoUrlVariantsProtocolAndHostComeFromUrlResolver() throws ReflectiveOperationException { + when(urls.inClusterRepoUrl(anyString())) + .thenAnswer(answer -> "http://scmm.ns.svc.cluster.local/scm/repo/" + answer.getArgument(0)); + when(urls.clientRepoUrl(anyString())) + .thenAnswer(answer -> "http://nodeport/scm/repo/" + answer.getArgument(0)); + + ScmManagerProvider scmManager = newScmManager(); + + assertEquals("http://scmm.ns.svc.cluster.local/scm", scmManager.getUrl()); + assertEquals("http://scmm.ns.svc.cluster.local/scm/repo/fv40-", scmManager.repoPrefix()); + + assertEquals( + "http://scmm.ns.svc.cluster.local/scm/repo/team/app", + scmManager.repoUrl("team/app", RepoUrlScope.IN_CLUSTER) + ); + assertEquals( + "http://nodeport/scm/repo/team/app", + scmManager.repoUrl("team/app", RepoUrlScope.CLIENT) + ); + + assertEquals("http", scmManager.getProtocol()); + assertEquals("scmm.ns.svc.cluster.local", scmManager.getHost()); + } + + @Test + void prometheusMetricsEndpointIsDelegatedToUrlResolver() throws URISyntaxException, ReflectiveOperationException { + when(urls.prometheusEndpoint()).thenReturn(new URI("http://nodeport/scm/api/v2/metrics/prometheus")); + + ScmManagerProvider scmManager = newScmManager(); + + assertEquals( + new URI("http://nodeport/scm/api/v2/metrics/prometheus"), + scmManager.prometheusMetricsEndpoint() + ); + } + + @Test + void runtimeCredentialsAndGitOpsUsernameAreAvailable() throws ReflectiveOperationException { + ScmManagerProvider scmManager = newScmManager(); + + assertEquals("user", scmManager.getCredentials().getUsername()); + assertEquals("password", scmManager.getCredentials().getPassword()); + assertEquals("gitops-bot", scmManager.getGitOpsUsername()); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolverTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolverTest.java new file mode 100644 index 000000000..17fe469d3 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/ScmManagerUrlResolverTest.java @@ -0,0 +1,322 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.NetworkingUtils; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.HashMap; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoMoreInteractions; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class ScmManagerUrlResolverTest { + + private Config config; + + @Mock + private K8sClient k8s; + + @Mock + private NetworkingUtils net; + + @BeforeEach + void setUp() { + config = new Config(); + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setNamePrefix("fv40-"); + application.setRunningInsideK8s(false); + config.setApplication(application); + } + + private ScmManagerUrlResolver resolverWith() { + return resolverWith(Map.of(), "fv40-"); + } + + private ScmManagerUrlResolver resolverWith(Map args) { + return resolverWith(args, "fv40-"); + } + + private ScmManagerUrlResolver resolverWith(Map args, String servicePrefix) { + ScmTenantSchema.ScmManagerTenantConfig scmmConfig = new ScmTenantSchema.ScmManagerTenantConfig(); + scmmConfig.setInternal(args.containsKey("internal") ? (Boolean) args.get("internal") : true); + scmmConfig.setNamespace(args.containsKey("namespace") ? (String) args.get("namespace") : "scm-manager"); + scmmConfig.setUrl(args.containsKey("url") ? (String) args.get("url") : ""); + scmmConfig.setIngress(args.containsKey("ingress") ? (String) args.get("ingress") : ""); + + return new ScmManagerUrlResolver( + scmmConfig, + k8s, + net, + config.getApplication().getNamePrefix(), + config.getApplication().getRunningInsideK8s(), + servicePrefix + ); + } + + @Test + void clientBaseTenantInternalOutsideK8sUsesPrefixedNodePortLookupAndAppendsScmOnlyOnce() { + when(k8s.waitForNodePort("fv40-scmm", "fv40-scm-manager")).thenReturn("30080"); + when(net.findClusterBindAddress()).thenReturn("10.0.0.1"); + + ScmManagerUrlResolver resolver = resolverWith(); + var base1 = resolver.clientBase(); + var base2 = resolver.clientBase(); + + assertEquals("http://10.0.0.1:30080/scm", base1.toString()); + assertEquals(base1, base2); + + verify(k8s, times(1)).waitForNodePort("fv40-scmm", "fv40-scm-manager"); + verify(net, times(1)).findClusterBindAddress(); + verifyNoMoreInteractions(k8s, net); + } + + @Test + void clientBaseCentralInternalOutsideK8sKeepsUnprefixedServiceNameAndNamespace() { + when(k8s.waitForNodePort("scmm", "scm-manager")).thenReturn("30080"); + when(net.findClusterBindAddress()).thenReturn("10.0.0.1"); + + ScmManagerUrlResolver resolver = resolverWith(Map.of(), ""); + + assertEquals("http://10.0.0.1:30080/scm", resolver.clientBase().toString()); + + verify(k8s).waitForNodePort("scmm", "scm-manager"); + verify(net).findClusterBindAddress(); + verifyNoMoreInteractions(k8s, net); + } + + @Test + void clientApiBaseAppendsApiToClientBase() { + when(k8s.waitForNodePort("fv40-scmm", "fv40-scm-manager")).thenReturn("30080"); + when(net.findClusterBindAddress()).thenReturn("10.0.0.1"); + + ScmManagerUrlResolver resolver = resolverWith(); + + assertEquals("http://10.0.0.1:30080/scm/api/", resolver.clientApiBase().toString()); + } + + @Test + void clientRepoUrlTrimsRepoTargetAndRemovesTrailingSlash() { + when(k8s.waitForNodePort("fv40-scmm", "fv40-scm-manager")).thenReturn("30080"); + when(net.findClusterBindAddress()).thenReturn("10.0.0.1"); + + ScmManagerUrlResolver resolver = resolverWith(); + + assertEquals( + "http://10.0.0.1:30080/scm/repo/ns/project", + resolver.clientRepoUrl(" ns/project ") + ); + } + + @Test + void inClusterBaseTenantInternalUsesPrefixedServiceDns() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(); + + assertEquals( + "http://fv40-scmm.fv40-scm-manager.svc.cluster.local/scm", + resolver.inClusterBase().toString() + ); + } + + @Test + void inClusterBaseTenantInternalPrefixesCustomNamespaceWhenNeeded() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(Map.of("namespace", "custom-ns")); + + assertEquals( + "http://fv40-scmm.fv40-custom-ns.svc.cluster.local/scm", + resolver.inClusterBase().toString() + ); + } + + @Test + void inClusterBaseTenantInternalDoesNotDuplicateAlreadyPrefixedNamespace() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(Map.of("namespace", "fv40-scm-manager")); + + assertEquals( + "http://fv40-scmm.fv40-scm-manager.svc.cluster.local/scm", + resolver.inClusterBase().toString() + ); + } + + @Test + void inClusterBaseCentralInternalUsesUnprefixedServiceDns() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(Map.of(), ""); + + assertEquals( + "http://scmm.scm-manager.svc.cluster.local/scm", + resolver.inClusterBase().toString() + ); + } + + @Test + void inClusterBaseExternalUsesExternalBaseAndScm() { + ScmManagerUrlResolver resolver = resolverWith(Map.of( + "internal", false, + "url", "https://fv40-scmm.external" + )); + + assertEquals("https://fv40-scmm.external/scm", resolver.inClusterBase().toString()); + } + + @Test + void inClusterRepoUrlBuildsFullTenantInClusterRepoUrlWithoutTrailingSlash() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(); + + assertEquals( + "http://fv40-scmm.fv40-scm-manager.svc.cluster.local/scm/repo/admin/admin", + resolver.inClusterRepoUrl("admin/admin") + ); + } + + @Test + void inClusterRepoPrefixTenantServiceUsesServicePrefixAndRepoNamespaceUsesApplicationNamePrefix() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(); + + assertEquals( + "http://fv40-scmm.fv40-scm-manager.svc.cluster.local/scm/repo/fv40-", + resolver.inClusterRepoPrefix() + ); + } + + @Test + void inClusterRepoPrefixCentralServiceStaysUnprefixedButRepoNamespaceStillUsesApplicationNamePrefix() { + config.getApplication().setRunningInsideK8s(true); + + ScmManagerUrlResolver resolver = resolverWith(Map.of(), ""); + + assertEquals( + "http://scmm.scm-manager.svc.cluster.local/scm/repo/fv40-", + resolver.inClusterRepoPrefix() + ); + } + + @Test + void inClusterRepoPrefixEmptyApplicationNamePrefixYieldsBaseRepoPath() { + config.getApplication().setRunningInsideK8s(true); + config.getApplication().setNamePrefix(" "); + + ScmManagerUrlResolver resolver = resolverWith(Map.of(), ""); + + assertEquals( + "http://scmm.scm-manager.svc.cluster.local/scm/repo/", + resolver.inClusterRepoPrefix() + ); + } + + @Test + void externalBasePrefersUrlOverIngress() { + ScmManagerUrlResolver resolver = resolverWith(Map.of( + "internal", false, + "url", "https://scmm.external", + "ingress", "ingress.example.org" + )); + + assertEquals("https://scmm.external/scm", resolver.inClusterBase().toString()); + } + + @Test + void externalBaseUsesIngressWhenUrlIsMissing() { + Map args = new HashMap<>(); + args.put("internal", false); + args.put("url", null); + args.put("ingress", "ingress.example.org"); + ScmManagerUrlResolver resolver = resolverWith(args); + + assertEquals("http://ingress.example.org/scm", resolver.inClusterBase().toString()); + } + + @Test + void externalBaseThrowsWhenNeitherUrlNorIngressIsSet() { + Map args = new HashMap<>(); + args.put("internal", false); + args.put("url", null); + args.put("ingress", null); + ScmManagerUrlResolver resolver = resolverWith(args); + + IllegalArgumentException exception = assertThrows( + IllegalArgumentException.class, + resolver::inClusterBase + ); + + assertTrue(exception.getMessage().contains( + "Either scmm.url or scmm.ingress must be set when internal=false" + )); + } + + @Test + void nodePortBaseTenantFallsBackToPrefixedDefaultNamespaceWhenNoneProvided() { + when(k8s.waitForNodePort(eq("fv40-scmm"), eq("fv40-scm-manager"))).thenReturn("30080"); + when(net.findClusterBindAddress()).thenReturn("10.0.0.1"); + + Map args = new HashMap<>(); + args.put("namespace", null); + ScmManagerUrlResolver resolver = resolverWith(args); + + assertEquals("http://10.0.0.1:30080/scm", resolver.clientBase().toString()); + } + + @Test + void nodePortBaseCentralFallsBackToUnprefixedDefaultNamespaceWhenNoneProvided() { + when(k8s.waitForNodePort(eq("scmm"), eq("scm-manager"))).thenReturn("30080"); + when(net.findClusterBindAddress()).thenReturn("10.0.0.1"); + + Map args = new HashMap<>(); + args.put("namespace", null); + ScmManagerUrlResolver resolver = resolverWith(args, ""); + + assertEquals("http://10.0.0.1:30080/scm", resolver.clientBase().toString()); + } + + @Test + void ensureScmAddsScmIfMissingAndKeepsItIfPresent() { + ScmManagerUrlResolver resolverWithoutScm = resolverWith(Map.of( + "internal", false, + "url", "https://fv40-scmm.localhost" + )); + assertEquals("https://fv40-scmm.localhost/scm", resolverWithoutScm.clientBase().toString()); + + ScmManagerUrlResolver resolverWithScm = resolverWith(Map.of( + "internal", false, + "url", "https://fv40-scmm.localhost/scm" + )); + assertEquals("https://fv40-scmm.localhost/scm", resolverWithScm.clientBase().toString()); + } + + @Test + void prometheusEndpointResolves() { + ScmManagerUrlResolver resolver = resolverWith(Map.of( + "internal", false, + "url", "https://fv40-scmm.localhost" + )); + + assertEquals( + "https://fv40-scmm.localhost/scm/api/v2/metrics/prometheus", + resolver.prometheusEndpoint().toString() + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApiTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApiTest.java new file mode 100644 index 000000000..620764fa5 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/git/providers/scmmanager/api/UsersApiTest.java @@ -0,0 +1,64 @@ +package com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api; + +import com.cloudogu.gitops.config.Credentials; +import com.github.tomakehurst.wiremock.junit5.WireMockExtension; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import javax.net.ssl.SSLHandshakeException; +import java.io.IOException; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.delete; +import static com.github.tomakehurst.wiremock.client.WireMock.deleteRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class UsersApiTest { + + @RegisterExtension + static final WireMockExtension wireMock = WireMockExtension.newInstance() + .options(wireMockConfig() + .dynamicPort() + .dynamicHttpsPort()) + .build(); + + private final Credentials credentials = new Credentials("user", "pass"); + + @Test + void allowsSelfSignedCertificatesWhenUsingInsecureOption() throws IOException { + wireMock.stubFor(delete(urlPathEqualTo("/scm/api/v2/users/test-user")) + .willReturn(aResponse().withStatus(204))); + + UsersApi api = usersApi(true, true); + var response = api.delete("test-user").execute(); + + assertThat(response.isSuccessful()).isTrue(); + wireMock.verify(1, deleteRequestedFor(urlPathEqualTo("/scm/api/v2/users/test-user"))); + } + + @Test + void doesNotAllowSelfSignedCertificatesByDefault() { + wireMock.stubFor(delete(urlPathEqualTo("/scm/api/v2/users/test-user")) + .willReturn(aResponse().withStatus(204))); + + UsersApi api = usersApi(false, true); + + assertThrows(SSLHandshakeException.class, () -> api.delete("test-user").execute()); + + wireMock.verify(0, deleteRequestedFor(urlPathEqualTo("/scm/api/v2/users/test-user"))); + } + + private UsersApi usersApi(boolean insecure, boolean useHttps) { + return new ScmManagerApiClient(apiBaseUrl(useHttps), credentials, insecure).usersApi(); + } + + private String apiBaseUrl(boolean useHttps) { + if (useHttps) { + return "https://localhost:" + wireMock.getRuntimeInfo().getHttpsPort() + "/scm/api/"; + } + return wireMock.baseUrl() + "/scm/api/"; + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManagerTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManagerTest.java new file mode 100644 index 000000000..e86fd35a6 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/GlobalPropertyManagerTest.java @@ -0,0 +1,113 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.contains; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class GlobalPropertyManagerTest { + + @Test + void setsGlobalProperty() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + GlobalPropertyManager propertyManager = new GlobalPropertyManager(client); + + when(client.runScript(anyString())).thenReturn("Done"); + propertyManager.setGlobalProperty("the-key", "the-value"); + + verify(client).runScript(""" + instance = Jenkins.getInstance() + globalNodeProperties = instance.getGlobalNodeProperties() + envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) + + def newEnvVarsNodeProperty + def envVars + + if ( envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0 ) { + newEnvVarsNodeProperty = new hudson.slaves.EnvironmentVariablesNodeProperty() + globalNodeProperties.add(newEnvVarsNodeProperty) + envVars = newEnvVarsNodeProperty.getEnvVars() + } else { + envVars = envVarsNodePropertyList.get(0).getEnvVars() + + } + + envVars.put('the-key', 'the-value') + + instance.save() + print("Done") + """); + } + + @Test + void throwsWhenThereWasAnErrorWhenCreatingGlobalProperty() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())) + .thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]"); + + assertThrows( + RuntimeException.class, + () -> new GlobalPropertyManager(client).setGlobalProperty("the-key", "the-value") + ); + } + + @Test + void deletesGlobalProperty() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + GlobalPropertyManager propertyManager = new GlobalPropertyManager(client); + + when(client.runScript(anyString())).thenReturn("Nothing to do"); + propertyManager.deleteGlobalProperty("the-key"); + + verify(client).runScript(""" + def instance = Jenkins.getInstance() + def globalNodeProperties = instance.getGlobalNodeProperties() + def envVarsNodePropertyList = globalNodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class) + + if (envVarsNodePropertyList == null || envVarsNodePropertyList.size() == 0) { + print("Nothing to do") + return + } + + envVars = envVarsNodePropertyList.get(0).getEnvVars() + envVars.remove('the-key') + print("Done") + """); + } + + @Test + void throwsWhenThereWasAnErrorWhenDeletingGlobalProperty() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())) + .thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]"); + + assertThrows( + RuntimeException.class, + () -> new GlobalPropertyManager(client).deleteGlobalProperty("the-key") + ); + } + + @Test + void escapesSingleQuotesInKeyAndValueToAvoidBreakingOutOfTheGroovyScript() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("Done"); + + new GlobalPropertyManager(client).setGlobalProperty("the'key", "the'value"); + + verify(client).runScript(contains("envVars.put('the\\'key', 'the\\'value')")); + } + + @Test + void rejectsValuesContainingBackslashes() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + + assertThrows( + IllegalArgumentException.class, + () -> new GlobalPropertyManager(client).setGlobalProperty("the-key", "the\\value") + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClientTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClientTest.java new file mode 100644 index 000000000..27c4a8a22 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/JenkinsApiClientTest.java @@ -0,0 +1,344 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.config.Config; +import com.github.tomakehurst.wiremock.junit5.WireMockExtension; +import io.micronaut.context.ApplicationContext; +import okhttp3.FormBody; +import okhttp3.JavaNetCookieJar; +import okhttp3.OkHttpClient; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocketFactory; +import javax.net.ssl.TrustManager; +import javax.net.ssl.X509TrustManager; +import java.net.CookieManager; +import java.security.SecureRandom; +import java.security.cert.X509Certificate; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.equalTo; +import static com.github.tomakehurst.wiremock.client.WireMock.get; +import static com.github.tomakehurst.wiremock.client.WireMock.getRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.matching; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class JenkinsApiClientTest { + + @RegisterExtension + static WireMockExtension wireMock = WireMockExtension.newInstance() + .options(wireMockConfig() + .dynamicPort() + .dynamicHttpsPort()) + .build(); + + @Test + void runsScriptWithCrumb() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .willReturn(aResponse() + .withStatus(200) + .withBody("ok"))); + + OkHttpClient httpClient = getUnsafeOkHttpClient().newBuilder() + .cookieJar(new JavaNetCookieJar(new CookieManager())) + .build(); + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JenkinsApiClient apiClient = new JenkinsApiClient(config, httpClient); + + String result = apiClient.runScript("println('ok')"); + assertThat(result).isEqualTo("ok"); + + wireMock.verify( + 1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .withHeader("Authorization", matching("Basic .*")) + ); + + wireMock.verify( + 1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText")) + .withHeader("Authorization", matching("Basic .*")) + .withHeader("Jenkins-Crumb", equalTo("the-crumb")) + ); + } + + @Test + void usesRuntimeCredentialsWhenConfigured() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .willReturn(aResponse().withStatus(200).withBody("ok"))); + + Config config = new Config(); + config.getJenkins().setUrl(wireMock.baseUrl() + "/jenkins"); + config.getJenkins().setUsername("fallback-user"); + config.getJenkins().setPassword("fallback-password"); + JenkinsApiClient apiClient = new JenkinsApiClient(config, getUnsafeOkHttpClient()); + apiClient.setRuntimeCredentials(new ResolvedCredentials("secret-user", "secret-password")); + + apiClient.runScript("println('ok')"); + + String authorization = okhttp3.Credentials.basic("secret-user", "secret-password"); + wireMock.verify( + 1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .withHeader("Authorization", equalTo(authorization)) + ); + wireMock.verify( + 1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText")) + .withHeader("Authorization", equalTo(authorization)) + ); + } + + @Test + void addsCrumbToSendRequest() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/foobar")) + .willReturn(aResponse().withStatus(200))); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JenkinsApiClient client = new JenkinsApiClient(config, getUnsafeOkHttpClient()); + client.postRequestWithCrumb("foobar"); + + wireMock.verify(1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))); + wireMock.verify( + 1, postRequestedFor(urlPathEqualTo("/jenkins/foobar")) + .withHeader("Jenkins-Crumb", equalTo("the-crumb")) + ); + } + + @Test + void addsCrumbAndPostDataToSendRequest() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/foobar")) + .willReturn(aResponse().withStatus(200))); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JenkinsApiClient client = new JenkinsApiClient(config, getUnsafeOkHttpClient()); + client.postRequestWithCrumb("foobar", new FormBody.Builder().add("key", "value with spaces").build()); + + wireMock.verify(1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))); + wireMock.verify( + 1, postRequestedFor(urlPathEqualTo("/jenkins/foobar")) + .withHeader("Jenkins-Crumb", equalTo("the-crumb")) + .withFormParam("key", equalTo("value with spaces")) + ); + } + + @Test + void allowsSelfSignedCertificatesWhenUsingInsecure() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .willReturn(aResponse() + .withStatus(200) + .withBody("ok"))); + + Config config = new Config(); + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setInsecure(true); + config.setApplication(application); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl().replace("http://", "https://") + "/jenkins"); + config.setJenkins(jenkins); + + JenkinsApiClient apiClient = ApplicationContext.run() + .registerSingleton(config) + .getBean(JenkinsApiClient.class); + + String result = apiClient.runScript("println('ok')"); + assertThat(result).isEqualTo("ok"); + + wireMock.verify( + 1, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .withHeader("Authorization", matching("Basic .*")) + ); + + wireMock.verify( + 1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText")) + .withHeader("Authorization", matching("Basic .*")) + .withHeader("Jenkins-Crumb", equalTo("the-crumb")) + ); + } + + @Test + void retriesOnInvalidCrumb() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .inScenario("Invalid Crumb Retry") + .whenScenarioStateIs("Started") + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-invalid-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}")) + .willSetStateTo("First Crumb")); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .inScenario("Invalid Crumb Retry") + .whenScenarioStateIs("First Crumb") + .withHeader("Jenkins-Crumb", equalTo("the-invalid-crumb")) + .willReturn(aResponse() + .withStatus(403) + .withBody( + "{\"servlet\":\"Stapler\", \"message\":\"No valid crumb was included in the request\", \"url\":\"/scriptText\", \"status\":\"403\"}")) + .willSetStateTo("Invalid Crumb Response")); + + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .inScenario("Invalid Crumb Retry") + .whenScenarioStateIs("Invalid Crumb Response") + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-second-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}")) + .willSetStateTo("Second Crumb")); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .inScenario("Invalid Crumb Retry") + .whenScenarioStateIs("Second Crumb") + .withHeader("Jenkins-Crumb", equalTo("the-second-crumb")) + .willReturn(aResponse() + .withStatus(200) + .withBody("ok"))); + + OkHttpClient httpClient = getUnsafeOkHttpClient(); + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JenkinsApiClient apiClient = new JenkinsApiClient(config, httpClient); + apiClient.setMaxRetries(3); + apiClient.setWaitPeriodInMs(0); + + String result = apiClient.runScript("println('ok')"); + assertThat(result).isEqualTo("ok"); + + wireMock.verify(2, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))); + wireMock.verify(2, postRequestedFor(urlPathEqualTo("/jenkins/scriptText"))); + } + + @Test + void retriesOnInvalidCrumbAreLimited() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-invalid-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .willReturn(aResponse() + .withStatus(403) + .withBody( + "{\"servlet\":\"Stapler\", \"message\":\"No valid crumb was included in the request\", \"url\":\"/scriptText\", \"status\":\"403\"}"))); + + OkHttpClient httpClient = getUnsafeOkHttpClient(); + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JenkinsApiClient apiClient = new JenkinsApiClient(config, httpClient); + apiClient.setMaxRetries(3); + apiClient.setWaitPeriodInMs(0); + + assertThrows(RuntimeException.class, () -> apiClient.runScript("println('ok')")); + + wireMock.verify(3, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))); + wireMock.verify(3, postRequestedFor(urlPathEqualTo("/jenkins/scriptText"))); + } + + @Test + void retriesWhenFetchingCrumbFails() { + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .inScenario("Crumb Fetch Retry") + .whenScenarioStateIs("Started") + .willReturn(aResponse() + .withStatus(401) + .withBody("error")) + .willSetStateTo("First Attempt Failed")); + + wireMock.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .inScenario("Crumb Fetch Retry") + .whenScenarioStateIs("First Attempt Failed") + .willReturn(aResponse() + .withStatus(200) + .withBody("{\"crumb\": \"the-invalid-crumb\", \"crumbRequestField\": \"Jenkins-Crumb\"}"))); + + wireMock.stubFor(post(urlPathEqualTo("/jenkins/scriptText")) + .willReturn(aResponse() + .withStatus(200) + .withBody("ok"))); + + OkHttpClient httpClient = getUnsafeOkHttpClient(); + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMock.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JenkinsApiClient apiClient = new JenkinsApiClient(config, httpClient); + apiClient.setMaxRetries(3); + apiClient.setWaitPeriodInMs(0); + + String result = apiClient.runScript("println('ok')"); + assertThat(result).isEqualTo("ok"); + + wireMock.verify(2, getRequestedFor(urlPathEqualTo("/jenkins/crumbIssuer/api/json"))); + wireMock.verify(1, postRequestedFor(urlPathEqualTo("/jenkins/scriptText"))); + } + + private static OkHttpClient getUnsafeOkHttpClient() { + try { + TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager() { + @Override + public void checkClientTrusted(X509Certificate[] chain, String authType) { + } + + @Override + public void checkServerTrusted(X509Certificate[] chain, String authType) { + } + + @Override + public X509Certificate[] getAcceptedIssuers() { + return new X509Certificate[0]; + } + } + }; + + SSLContext sslContext = SSLContext.getInstance("SSL"); + sslContext.init(null, trustAllCerts, new SecureRandom()); + SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory(); + + return new OkHttpClient.Builder() + .sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0]) + .hostnameVerifier((hostname, session) -> true) + .build(); + } catch (Exception e) { + throw new RuntimeException(e); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/JobManagerTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/JobManagerTest.java new file mode 100644 index 000000000..66d77da39 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/JobManagerTest.java @@ -0,0 +1,303 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import com.cloudogu.gitops.config.Config; +import com.github.tomakehurst.wiremock.WireMockServer; +import okhttp3.OkHttpClient; +import org.junit.jupiter.api.Test; + +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; + +import static com.github.tomakehurst.wiremock.client.WireMock.aResponse; +import static com.github.tomakehurst.wiremock.client.WireMock.containing; +import static com.github.tomakehurst.wiremock.client.WireMock.equalTo; +import static com.github.tomakehurst.wiremock.client.WireMock.get; +import static com.github.tomakehurst.wiremock.client.WireMock.ok; +import static com.github.tomakehurst.wiremock.client.WireMock.okJson; +import static com.github.tomakehurst.wiremock.client.WireMock.post; +import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo; +import static com.github.tomakehurst.wiremock.client.WireMock.urlPathMatching; +import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.options; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class JobManagerTest { + + @Test + void createsCredential() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathMatching(".*createCredentials.*")) + .willReturn(ok())); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + jobManager.createCredential("the-jobname", "the-id", "the-username", "the-password", "some description"); + + wireMockServer.verify(postRequestedFor(urlPathEqualTo( + "/jenkins/job/the-jobname/credentials/store/folder/domain/_/createCredentials"))); + + var requests = wireMockServer.findAll(postRequestedFor(urlPathMatching(".*createCredentials.*"))); + assertThat(requests).hasSize(1); + + String requestBody = requests.get(0).getBodyAsString(); + assertThat(URLDecoder.decode(requestBody, StandardCharsets.UTF_8)) + .isEqualTo( + "json={\"credentials\":{\"scope\":\"GLOBAL\",\"id\":\"the-id\",\"username\":\"the-username\",\"password\":\"the-password\",\"description\":\"some description\",\"$class\":\"com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl\"}}"); + + } finally { + wireMockServer.stop(); + } + } + + @Test + void throwsWhenCreatingCredentialFails() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathMatching(".*createCredentials.*")) + .willReturn(aResponse().withStatus(404))); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> jobManager.createCredential( + "the-jobname", + "the-id", + "the-username", + "the-password", + "some description" + ) + ); + assertThat(exception.getMessage()).isEqualTo( + "Could not create credential id=the-id,job=the-jobname. StatusCode: 404"); + } finally { + wireMockServer.stop(); + } + } + + @Test + void startsJob() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathMatching("/jenkins/job/the-jobname/build.*")) + .willReturn(ok())); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + jobManager.startJob("the-jobname"); + + wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname/build")) + .withQueryParam("delay", equalTo("0sec"))); + + } finally { + wireMockServer.stop(); + } + } + + @Test + void throwsWhenStartingJobFails() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathMatching("/jenkins/job/the-jobname/build.*")) + .willReturn(aResponse().withStatus(400))); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> jobManager.startJob("the-jobname") + ); + assertThat(exception.getMessage()).isEqualTo( + "Could not trigger build of Jenkins job: the-jobname. StatusCode: 400"); + } finally { + wireMockServer.stop(); + } + } + + @Test + void throwsWhenJobContainsInvalidCharacters() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + JobManager jobManager = new JobManager(client); + + RuntimeException exception = assertThrows(RuntimeException.class, () -> jobManager.deleteJob("foo'foo")); + assertThat(exception.getMessage()).isEqualTo("Job name cannot contain quotes."); + } + + @Test + void throwsWhenJobDeletionFails() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + JobManager jobManager = new JobManager(client); + + RuntimeException exception = assertThrows(RuntimeException.class, () -> jobManager.deleteJob("foo-foo")); + assertThat(exception.getMessage()).isEqualTo("Could not delete job foo-foo"); + } + + @Test + void deletesJob() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + JobManager jobManager = new JobManager(client); + + when(client.runScript(anyString())).thenReturn("null"); + jobManager.deleteJob("foo"); + verify(client).runScript("print(Jenkins.instance.getItem('foo')?.delete())"); + } + + @Test + void checksExistingJob() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) + .willReturn(ok())); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + boolean exists = jobManager.jobExists("the-jobname"); + + assertThat(exists).isEqualTo(true); + wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))); + } finally { + wireMockServer.stop(); + } + } + + @Test + void checksNonExistingJob() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) + .willReturn(aResponse().withStatus(404))); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + boolean exists = jobManager.jobExists("the-jobname"); + assertThat(exists).isEqualTo(false); + wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))); + } finally { + wireMockServer.stop(); + } + } + + @Test + void createsJob() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) + .willReturn(aResponse().withStatus(404))); + wireMockServer.stubFor(post(urlPathMatching("/jenkins/createItem.*")) + .willReturn(ok())); + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + boolean created = jobManager.createJob("the-jobname", "http://scm", "ns", "creds"); + + assertThat(created).isEqualTo(true); + + wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))); + wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/createItem")) + .withQueryParam("name", equalTo("the-jobname")) + .withRequestBody(containing("http://scm")) + .withRequestBody(containing("ns")) + .withRequestBody(containing("creds"))); + + } finally { + wireMockServer.stop(); + } + } + + @Test + void ignoresExistingJob() { + WireMockServer wireMockServer = new WireMockServer(options().dynamicPort()); + wireMockServer.start(); + + try { + wireMockServer.stubFor(get(urlPathEqualTo("/jenkins/crumbIssuer/api/json")) + .willReturn(okJson("{\"crumb\":\"the-crumb\"}"))); + + wireMockServer.stubFor(post(urlPathEqualTo("/jenkins/job/the-jobname")) + .willReturn(ok())); // 200 OK means "Job Exists" + + Config config = new Config(); + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setUrl(wireMockServer.baseUrl() + "/jenkins"); + config.setJenkins(jenkins); + JobManager jobManager = new JobManager(new JenkinsApiClient(config, new OkHttpClient())); + + boolean created = jobManager.createJob("the-jobname", "http://scm", "ns", "creds"); + + assertThat(created).isEqualTo(false); + wireMockServer.verify(postRequestedFor(urlPathEqualTo("/jenkins/job/the-jobname"))); + wireMockServer.verify(0, postRequestedFor(urlPathEqualTo("/jenkins/createItem"))); + + } finally { + wireMockServer.stop(); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/UserManagerTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/UserManagerTest.java new file mode 100644 index 000000000..33e5fbfab --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/jenkins/UserManagerTest.java @@ -0,0 +1,145 @@ +package com.cloudogu.gitops.infrastructure.jenkins; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class UserManagerTest { + + @Test + void createsUserSuccessfully() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("the-user"); + + new UserManager(client).createUser("the-user", "hunter2"); + verify(client).runScript(anyString()); + } + + @Test + void createsUserWithQuotesSuccessfully() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("the-'user"); + + new UserManager(client).createUser("the-'user", "code''injection"); + verify(client).runScript(""" + def realm = Jenkins.getInstance().getSecurityRealm() + def user = realm.createAccount('the-\\'user', 'code\\'\\'injection') + + print(user) + """); + } + + @Test + void throwsWhenBackslashesArePassed() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + + assertThrows( + IllegalArgumentException.class, + () -> new UserManager(client).createUser("the-\\'user", "hunter2") + ); + } + + @Test + void throwsWhenThereWasAnError() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())) + .thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]"); + + assertThrows( + RuntimeException.class, + () -> new UserManager(client).createUser("the-user", "hunter2") + ); + } + + @Test + void grantsPermissionForUser() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("true"); + when(client.runScript("print(Jenkins.getInstance().getAuthorizationStrategy().class)")) + .thenReturn("class hudson.security.GlobalMatrixAuthorizationStrategy"); + + new UserManager(client).grantPermission("the-'user", UserManager.Permissions.METRICS_VIEW); + + verify(client).runScript("print(Jenkins.getInstance().getAuthorizationStrategy().class)"); + verify(client).runScript(""" + import org.jenkinsci.plugins.matrixauth.PermissionEntry + import org.jenkinsci.plugins.matrixauth.AuthorizationType + + def permissions = Jenkins.getInstance().getAuthorizationStrategy().getGrantedPermissionEntries() + permissions.computeIfAbsent(jenkins.metrics.api.Metrics.VIEW) { + new HashSet<>() + } + print(permissions[jenkins.metrics.api.Metrics.VIEW].add(new PermissionEntry(AuthorizationType.USER, 'the-\\'user'))) + """); + } + + @Test + void throwsWhenGrantingPermissionFailed() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())) + .thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]"); + + assertThrows( + RuntimeException.class, + () -> new UserManager(client).grantPermission("the-'user", UserManager.Permissions.METRICS_VIEW) + ); + } + + @Test + void checksWhetherMatrixBasedAuthorizationIsEnabled() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("class hudson.security.GlobalMatrixAuthorizationStrategy"); + + assertThat(new UserManager(client).isUsingMatrixBasedPermissions()).isTrue(); + } + + @Test + void checksWhetherMatrixBasedAuthorizationIsDisabled() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn( + "class hudson.security.FullControlOnceLoggedInAuthorizationStrategy"); + + assertThat(new UserManager(client).isUsingMatrixBasedPermissions()).isFalse(); + } + + @Test + void checksWhetherSecurityRealmWithoutLocalUserCreationIsUsedForCas() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("class org.jenkinsci.plugins.cas.CasSecurityRealm"); + + assertThat(new UserManager(client).isUsingSecurityRealmWithoutLocalUserCreation()).isTrue(); + } + + @Test + void checksWhetherSecurityRealmWithoutLocalUserCreationIsUsedForOic() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("class org.jenkinsci.plugins.oic.OicSecurityRealm"); + + assertThat(new UserManager(client).isUsingSecurityRealmWithoutLocalUserCreation()).isTrue(); + } + + @Test + void checksWhetherLocalUserCreationIsSupported() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())).thenReturn("class hudson.security.HudsonPrivateSecurityRealm"); + + assertThat(new UserManager(client).isUsingSecurityRealmWithoutLocalUserCreation()).isFalse(); + } + + @Test + void throwsWhenDeterminingSecurityRealmErrors() { + JenkinsApiClient client = mock(JenkinsApiClient.class); + when(client.runScript(anyString())) + .thenReturn("groovy.lang.MissingPropertyException: No such property: asd for class: Script1[...]"); + + assertThrows( + RuntimeException.class, + () -> new UserManager(client).isUsingSecurityRealmWithoutLocalUserCreation() + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientTest.java new file mode 100644 index 000000000..cc2c9dc87 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/kubernetes/api/K8sClientTest.java @@ -0,0 +1,1887 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.api; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.utils.Tuple; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.ObjectMapper; +import io.fabric8.kubernetes.api.model.ConfigMapBuilder; +import io.fabric8.kubernetes.api.model.GenericKubernetesResource; +import io.fabric8.kubernetes.api.model.GenericKubernetesResourceBuilder; +import io.fabric8.kubernetes.api.model.NamespaceBuilder; +import io.fabric8.kubernetes.api.model.NodeBuilder; +import io.fabric8.kubernetes.api.model.NodeListBuilder; +import io.fabric8.kubernetes.api.model.PodBuilder; +import io.fabric8.kubernetes.api.model.PodListBuilder; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.api.model.ServiceBuilder; +import io.fabric8.kubernetes.api.model.StatusBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Base64; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicBoolean; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +@EnableKubernetesMockClient +@SuppressWarnings("unchecked") +class K8sClientTest { + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + private static final TypeReference> JSON_MAP_TYPE = new TypeReference<>() { + }; + private static final TypeReference>> JSON_LIST_TYPE = new TypeReference<>() { + }; + + KubernetesMockServer server; + KubernetesClient client; + + private K8sClient k8sApiClient; + + @TempDir + Path tempDir; + + @BeforeEach + void setup() { + k8sApiClient = new K8sClient(); + k8sApiClient.setClient(client); + k8sApiClient.sleepTimeMillis = 10; // Speed up tests + k8sApiClient.defaultRetries = 3; + } + + // ======================================== + // Node Operations Tests + // ======================================== + + @Test + void waitForNodeReturnsFirstNodeName() { + // Given + var node = new NodeBuilder() + .withNewMetadata() + .withName("test-node-1") + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/nodes") + .andReturn(200, new NodeListBuilder().withItems(node).build()) + .once(); + + // When + String nodeName = k8sApiClient.waitForNode(); + + // Then + assertThat(nodeName).isEqualTo("test-node-1"); + } + + @Test + void waitForNodeRetriesWhenNoNodesAvailable() { + // Given + server.expect() + .get() + .withPath("/api/v1/nodes") + .andReturn(200, new NodeListBuilder().build()) + .times(2); + + var node = new NodeBuilder() + .withNewMetadata() + .withName("test-node-1") + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/nodes") + .andReturn(200, new NodeListBuilder().withItems(node).build()) + .once(); + + // When + String nodeName = k8sApiClient.waitForNode(); + + // Then + assertThat(nodeName).isEqualTo("test-node-1"); + } + + @Test + void waitForNodeThrowsExceptionAfterMaxRetries() { + // Given + server.expect() + .get() + .withPath("/api/v1/nodes") + .andReturn(200, new NodeListBuilder().build()) + .times(k8sApiClient.defaultRetries + 1); + + // When/Then + var exception = assertThrows(RuntimeException.class, () -> k8sApiClient.waitForNode()); + assertThat(exception.getMessage()).contains("Failed to retrieve node"); + } + + @Test + void waitForInternalNodeIpReturnsNodeInternalIP() { + // Given - First call for waitForNode + var node = new NodeBuilder() + .withNewMetadata() + .withName("test-node-1") + .endMetadata() + .withNewStatus() + .addNewAddress() + .withType("InternalIP") + .withAddress("192.168.1.100") + .endAddress() + .endStatus() + .build(); + + server.expect() + .get() + .withPath("/api/v1/nodes") + .andReturn(200, new NodeListBuilder().withItems(node).build()) + .once(); + + // Second call for waitForInternalNodeIp + server.expect() + .get() + .withPath("/api/v1/nodes/test-node-1") + .andReturn(200, node) + .once(); + + // When + String ip = k8sApiClient.waitForInternalNodeIp(); + + // Then + assertThat(ip).isEqualTo("192.168.1.100"); + } + + @Test + void waitForInternalNodeIpIgnoresIPv6Addresses() { + // Given + var node = new NodeBuilder() + .withNewMetadata() + .withName("test-node-1") + .endMetadata() + .withNewStatus() + .addNewAddress() + .withType("InternalIP") + .withAddress("192.168.1.100") + .endAddress() + .addNewAddress() + .withType("InternalIP") + .withAddress("fe80::1") + .endAddress() + .endStatus() + .build(); + + server.expect() + .get() + .withPath("/api/v1/nodes") + .andReturn(200, new NodeListBuilder().withItems(node).build()) + .once(); + + server.expect() + .get() + .withPath("/api/v1/nodes/test-node-1") + .andReturn(200, node) + .once(); + + // When + String ip = k8sApiClient.waitForInternalNodeIp(); + + // Then + assertThat(ip).isEqualTo("192.168.1.100"); + } + + // ======================================== + // Service Operations Tests + // ======================================== + + @Test + void waitForNodePortReturnsServiceNodePort() { + // Given + var service = new ServiceBuilder() + .withNewMetadata() + .withName("test-service") + .withNamespace("test-ns") + .endMetadata() + .withNewSpec() + .addNewPort() + .withPort(8080) + .withNodePort(30080) + .endPort() + .endSpec() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/services/test-service") + .andReturn(200, service) + .once(); + + // When + String nodePort = k8sApiClient.waitForNodePort("test-service", "test-ns"); + + // Then + assertThat(nodePort).isEqualTo("30080"); + } + + @Test + void createServiceNodePortCreatesServiceWithNodePort() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/default/services") + .andReturn( + 201, new ServiceBuilder() + .withNewMetadata() + .withName("my-service") + .withNamespace("default") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createServiceNodePort("my-service", "8080:80", "30000", ""); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("POST"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/default/services"); + + Map body = parseJson(request.getUtf8Body()); + Map metadata = (Map) body.get("metadata"); + Map spec = (Map) body.get("spec"); + List> ports = (List>) spec.get("ports"); + + assertThat(metadata.get("name")).isEqualTo("my-service"); + assertThat(metadata.get("namespace")).isEqualTo("default"); + assertThat(spec.get("type")).isEqualTo("NodePort"); + assertThat(ports).hasSize(1); + assertThat(ports.get(0).get("port")).isEqualTo(8080); + assertThat(ports.get(0).get("targetPort")).isEqualTo(80); + assertThat(ports.get(0).get("nodePort")).isEqualTo(30000); + } + + @Test + void createServiceNodePortCreatesServiceWithoutExplicitNodePort() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/test-ns/services") + .andReturn( + 201, new ServiceBuilder() + .withNewMetadata() + .withName("my-service") + .withNamespace("test-ns") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createServiceNodePort("my-service", "8080:80", "", "test-ns"); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("POST"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/test-ns/services"); + + Map body = parseJson(request.getUtf8Body()); + Map spec = (Map) body.get("spec"); + List> ports = (List>) spec.get("ports"); + + assertThat(spec.get("type")).isEqualTo("NodePort"); + assertThat(ports).hasSize(1); + assertThat(ports.get(0).get("port")).isEqualTo(8080); + assertThat(ports.get(0).get("targetPort")).isEqualTo(80); + assertThat(ports.get(0).get("nodePort")).isNull(); + } + + @Test + void patchServiceNodePortUpdatesServicePort() throws InterruptedException { + // Given + var service = new ServiceBuilder() + .withNewMetadata() + .withName("test-service") + .withNamespace("test-ns") + .endMetadata() + .withNewSpec() + .addNewPort() + .withName("http") + .withPort(8080) + .withNodePort(30080) + .endPort() + .endSpec() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/services/test-service") + .andReturn(200, service) + .once(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/services/test-service") + .andReturn(200, service) + .once(); + + server.expect() + .patch() + .withPath("/api/v1/namespaces/test-ns/services/test-service") + .andReturn(200, service) + .once(); + + // When + k8sApiClient.patchServiceNodePort("test-service", "test-ns", "http", 30090); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("PATCH"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/test-ns/services/test-service"); + assertThat(parseJsonList(request.getUtf8Body())).containsExactly(Map.of( + "op", "replace", + "path", "/spec/ports/0/nodePort", + "value", 30090 + )); + } + + @Test + void patchServiceNodePortThrowsExceptionForInvalidParameters() { + // When/Then + var exception = assertThrows( + IllegalArgumentException.class, + () -> k8sApiClient.patchServiceNodePort("", "test-ns", "http", 30000) + ); + assertThat(exception.getMessage()).contains("Service name"); + } + + @Test + void patchServiceNodePortThrowsExceptionWhenPortNotFound() { + // Given + var service = new ServiceBuilder() + .withNewMetadata() + .withName("test-service") + .withNamespace("test-ns") + .endMetadata() + .withNewSpec() + .addNewPort() + .withName("http") + .withPort(8080) + .endPort() + .endSpec() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/services/test-service") + .andReturn(200, service) + .once(); + + // When/Then + var exception = assertThrows( + RuntimeException.class, + () -> k8sApiClient.patchServiceNodePort("test-service", "test-ns", "https", 30000) + ); + assertThat(exception.getMessage()).contains("Port with name https not found"); + } + + // ======================================== + // Namespace Operations Tests + // ======================================== + + @Test + void createNamespaceCreatesNewNamespace() throws InterruptedException { + // Given + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns") + .andReturn(404, "") + .once(); + server.expect() + .post() + .withPath("/api/v1/namespaces") + .andReturn( + 201, new NamespaceBuilder() + .withNewMetadata() + .withName("test-ns") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createNamespace("test-ns"); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("POST"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces"); + Map body = parseJson(request.getUtf8Body()); + assertThat(body.get("kind")).isEqualTo("Namespace"); + assertThat(((Map) body.get("metadata")).get("name")).isEqualTo("test-ns"); + } + + @Test + void createNamespaceCreatesOpenShiftProjectWhenOpenshiftConfigIsEnabled() throws InterruptedException { + // Given + Config config = Config.fromMap(Map.of("application", Map.of("openshift", true))); + k8sApiClient.setGopConfig(config); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-project") + .andReturn(404, "") + .once(); + + server.expect() + .post() + .withPath("/apis/project.openshift.io/v1/projects") + .andReturn( + 201, new GenericKubernetesResourceBuilder() + .withApiVersion("project.openshift.io/v1") + .withKind("Project") + .withNewMetadata() + .withName("test-project") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createNamespace("test-project"); + + // Then + Map requestBody = parseJson(server.getLastRequest().getUtf8Body()); + assertThat(requestBody.get("kind")).isEqualTo("Project"); + assertThat(((Map) requestBody.get("metadata")).get("name")).isEqualTo("test-project"); + } + + @Test + void createNamespaceCreatesKubernetesNamespaceWhenOpenshiftConfigIsDisabled() throws InterruptedException { + // Given + Config config = Config.fromMap(Map.of("application", Map.of("openshift", false))); + k8sApiClient.setGopConfig(config); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns") + .andReturn(404, "") + .once(); + + server.expect() + .post() + .withPath("/api/v1/namespaces") + .andReturn( + 201, new NamespaceBuilder() + .withNewMetadata() + .withName("test-ns") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createNamespace("test-ns"); + + // Then + Map requestBody = parseJson(server.getLastRequest().getUtf8Body()); + assertThat(requestBody.get("kind")).isEqualTo("Namespace"); + assertThat(((Map) requestBody.get("metadata")).get("name")).isEqualTo("test-ns"); + } + + @Test + void createNamespaceDoesNotCreateExistingNamespace() throws InterruptedException { + // Given + var namespace = new NamespaceBuilder() + .withNewMetadata() + .withName("test-ns") + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns") + .andReturn(200, namespace) + .once(); + + // When + k8sApiClient.createNamespace("test-ns"); + + // Then + assertThat(server.getLastRequest().getMethod()).isEqualTo("GET"); + assertThat(server.getLastRequest().getPath()).isEqualTo("/api/v1/namespaces/test-ns"); + } + + @Test + void createNamespaceCreatesKubernetesNamespaceWhenConfigIsNull() throws InterruptedException { + // Given + k8sApiClient.setGopConfig(null); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns") + .andReturn(404, "") + .once(); + + server.expect() + .post() + .withPath("/api/v1/namespaces") + .andReturn( + 201, new NamespaceBuilder() + .withNewMetadata() + .withName("test-ns") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createNamespace("test-ns"); + + // Then + Map requestBody = parseJson(server.getLastRequest().getUtf8Body()); + assertThat(requestBody.get("kind")).isEqualTo("Namespace"); + assertThat(((Map) requestBody.get("metadata")).get("name")).isEqualTo("test-ns"); + } + + @Test + void createNamespaceDoesNotCreateOpenShiftProjectWhenNamespaceAlreadyExists() throws InterruptedException { + // Given + Config config = Config.fromMap(Map.of("application", Map.of("openshift", true))); + k8sApiClient.setGopConfig(config); + + var namespace = new NamespaceBuilder() + .withNewMetadata() + .withName("existing-project") + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/existing-project") + .andReturn(200, namespace) + .once(); + + // When + k8sApiClient.createNamespace("existing-project"); + + // Then + assertThat(server.getLastRequest().getMethod()).isEqualTo("GET"); + assertThat(server.getLastRequest().getPath()).isEqualTo("/api/v1/namespaces/existing-project"); + } + + @Test + void createNamespaceThrowsExceptionForInvalidName() { + // When/Then + var exception = assertThrows(IllegalArgumentException.class, () -> k8sApiClient.createNamespace("")); + assertThat(exception.getMessage()).contains("Namespace name must be provided"); + } + + @Test + void createNamespacesCreatesMultipleNamespaces() throws InterruptedException { + // Given + server.expect().get().withPath("/api/v1/namespaces/ns1").andReturn(404, "").once(); + server.expect() + .post() + .withPath("/api/v1/namespaces") + .andReturn(201, new NamespaceBuilder().withNewMetadata().withName("ns1").endMetadata().build()) + .once(); + server.expect().get().withPath("/api/v1/namespaces/ns2").andReturn(404, "").once(); + server.expect() + .post() + .withPath("/api/v1/namespaces") + .andReturn(201, new NamespaceBuilder().withNewMetadata().withName("ns2").endMetadata().build()) + .once(); + + // When + k8sApiClient.createNamespaces(List.of("ns1", "ns2")); + + // Then + assertThat(server.getRequestCount()).isEqualTo(4); + assertThat(server.takeRequest().getPath()).isEqualTo("/api/v1/namespaces/ns1"); + Map firstCreate = parseJson(server.takeRequest().getUtf8Body()); + assertThat(((Map) firstCreate.get("metadata")).get("name")).isEqualTo("ns1"); + assertThat(server.takeRequest().getPath()).isEqualTo("/api/v1/namespaces/ns2"); + Map secondCreate = parseJson(server.takeRequest().getUtf8Body()); + assertThat(((Map) secondCreate.get("metadata")).get("name")).isEqualTo("ns2"); + } + + @Test + void namespaceExistsReturnsTrueForExistingNamespace() { + // Given + var namespace = new NamespaceBuilder() + .withNewMetadata() + .withName("test-ns") + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns") + .andReturn(200, namespace) + .once(); + + // When + boolean exists = k8sApiClient.namespaceExists("test-ns"); + + // Then + assertThat(exists).isTrue(); + } + + @Test + void namespaceExistsReturnsFalseForNonExistingNamespace() { + // Given + server.expect() + .get() + .withPath("/api/v1/namespaces/non-existing") + .andReturn(404, "") + .once(); + + // When + boolean exists = k8sApiClient.namespaceExists("non-existing"); + + // Then + assertThat(exists).isFalse(); + } + + // ======================================== + // Secret Operations Tests + // ======================================== + + @Test + void createSecretCreatesGenericSecret() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/test-ns/secrets") + .andReturn( + 201, new SecretBuilder() + .withNewMetadata() + .withName("my-secret") + .withNamespace("test-ns") + .endMetadata() + .withType("Opaque") + .build() + ) + .once(); + + // When + k8sApiClient.createSecret( + "Opaque", "my-secret", "test-ns", + new Tuple("username", "admin"), + new Tuple("password", "secret") + ); + + // Then + var request = server.getLastRequest(); + Map body = parseJson(request.getUtf8Body()); + assertThat(request.getMethod()).isEqualTo("POST"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/test-ns/secrets"); + assertThat(body.get("type")).isEqualTo("Opaque"); + assertThat(((Map) body.get("metadata")).get("name")).isEqualTo("my-secret"); + assertThat(((Map) body.get("stringData"))) + .containsEntry("username", "admin") + .containsEntry("password", "secret"); + } + + @Test + void createSecretUpdatesAnExistingSecretWithoutDeletingIt() throws InterruptedException { + var secret = new SecretBuilder() + .withNewMetadata() + .withName("my-secret") + .withNamespace("test-ns") + .endMetadata() + .withType("Opaque") + .build(); + + server.expect() + .post() + .withPath("/api/v1/namespaces/test-ns/secrets") + .andReturn(409, new StatusBuilder().withCode(409).build()) + .once(); + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/secrets/my-secret") + .andReturn(200, secret) + .once(); + server.expect() + .put() + .withPath("/api/v1/namespaces/test-ns/secrets/my-secret") + .andReturn(200, secret) + .once(); + + k8sApiClient.createSecret("Opaque", "my-secret", "test-ns", new Tuple("username", "admin")); + + assertThat(server.getRequestCount()).isEqualTo(3); + assertThat(server.takeRequest().getMethod()).isEqualTo("POST"); + assertThat(server.takeRequest().getMethod()).isEqualTo("GET"); + var updateRequest = server.takeRequest(); + assertThat(updateRequest.getMethod()).isEqualTo("PUT"); + assertThat(updateRequest.getPath()).isEqualTo("/api/v1/namespaces/test-ns/secrets/my-secret"); + assertThat(updateRequest.getUtf8Body()).contains("\"username\":\"admin\""); + } + + @Test + void createImagePullSecretCreatesDockerRegistrySecret() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/default/secrets") + .andReturn( + 201, new SecretBuilder() + .withNewMetadata() + .withName("my-registry") + .withNamespace("default") + .endMetadata() + .withType("kubernetes.io/dockerconfigjson") + .build() + ) + .once(); + + // When + k8sApiClient.createImagePullSecret("my-registry", "", "docker.io", "user\"name", "pa\"ss"); + + Map requestBody = parseJson(server.getLastRequest().getUtf8Body()); + Map dockerConfig = parseJson((String) ((Map) requestBody.get("stringData")).get( + ".dockerconfigjson")); + assertThat(((Map) ((Map) dockerConfig.get("auths")).get("docker.io")).get( + "username")).isEqualTo("user\"name"); + assertThat(((Map) ((Map) dockerConfig.get("auths")).get("docker.io")).get( + "password")).isEqualTo("pa\"ss"); + } + + @Test + void getArgoCDNamespacesSecretRetrievesSecretData() { + // Given + var secret = new SecretBuilder() + .withNewMetadata() + .withName("argocd-secret") + .withNamespace("argocd") + .endMetadata() + .withData(Map.of( + "namespaces", + Base64.getEncoder().encodeToString("ns1,ns2".getBytes(StandardCharsets.UTF_8)) + )) + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/argocd/secrets/argocd-secret") + .andReturn(200, secret) + .once(); + + // When + String data = k8sApiClient.getArgoCDNamespacesSecret("argocd-secret", "argocd"); + + // Then + assertThat(data).isEqualTo(Base64.getEncoder().encodeToString("ns1,ns2".getBytes(StandardCharsets.UTF_8))); + } + + @Test + void getCredentialsFromSecretExtractsUsernameAndPassword() { + // Given + var secret = new SecretBuilder() + .withNewMetadata() + .withName("my-secret") + .withNamespace("test-ns") + .endMetadata() + .withData(Map.of( + "username", Base64.getEncoder().encodeToString("admin".getBytes(StandardCharsets.UTF_8)), + "password", Base64.getEncoder().encodeToString("secret123".getBytes(StandardCharsets.UTF_8)) + )) + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/secrets/my-secret") + .andReturn(200, secret) + .once(); + + // When + Credentials creds = k8sApiClient.getCredentialsFromSecret("my-secret", "test-ns"); + + // Then + assertThat(creds.getUsername()).isEqualTo("admin"); + assertThat(creds.getPassword()).isEqualTo("secret123"); + } + + @Test + void getCredentialsFromSecretWithCredentialsObject() { + // Given + var inputCreds = new Credentials(); + inputCreds.setSecretName("my-secret"); + inputCreds.setSecretNamespace("test-ns"); + inputCreds.setUsernameKey("user"); + inputCreds.setPasswordKey("pass"); + + var secret = new SecretBuilder() + .withNewMetadata() + .withName("my-secret") + .withNamespace("test-ns") + .endMetadata() + .withData(Map.of( + "user", Base64.getEncoder().encodeToString("testuser".getBytes(StandardCharsets.UTF_8)), + "pass", Base64.getEncoder().encodeToString("testpass".getBytes(StandardCharsets.UTF_8)) + )) + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/secrets/my-secret") + .andReturn(200, secret) + .once(); + + // When + Credentials result = k8sApiClient.getCredentialsFromSecret(inputCreds); + + // Then + assertThat(result.getUsername()).isEqualTo("testuser"); + assertThat(result.getPassword()).isEqualTo("testpass"); + } + + // ======================================== + // ConfigMap Operations Tests + // ======================================== + + @Test + void createConfigMapFromFileCreatesConfigmap() throws IOException, InterruptedException { + // Given + Path testFile = tempDir.resolve("test.txt"); + Files.writeString(testFile, "test content"); + + server.expect() + .post() + .withPath("/api/v1/namespaces/default/configmaps") + .andReturn( + 201, new ConfigMapBuilder() + .withNewMetadata() + .withName("my-config") + .withNamespace("default") + .endMetadata() + .build() + ) + .once(); + + // When + k8sApiClient.createConfigMapFromFile("my-config", "", testFile.toString()); + + // Then + var request = server.getLastRequest(); + Map body = parseJson(request.getUtf8Body()); + assertThat(request.getMethod()).isEqualTo("POST"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/default/configmaps"); + assertThat(((Map) body.get("metadata")).get("name")).isEqualTo("my-config"); + assertThat(((Map) body.get("data"))).containsEntry("test.txt", "test content"); + } + + @Test + void createConfigMapFromFileThrowsExceptionForNonExistingFile() { + // When/Then + var exception = assertThrows( + RuntimeException.class, + () -> k8sApiClient.createConfigMapFromFile("my-config", "", "/non/existing/file.txt") + ); + assertThat(exception.getMessage()).contains("File not found"); + } + + @Test + void getConfigMapRetrievesValueFromConfigmap() { + // Given + var configMap = new ConfigMapBuilder() + .withNewMetadata() + .withName("my-config") + .withNamespace("test") + .endMetadata() + .withData(Map.of("key1", "value1", "key2", "value2")) + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test/configmaps/my-config") + .andReturn(200, configMap) + .once(); + + // When + String value = k8sApiClient.getConfigMap("my-config", "key1"); + + // Then + assertThat(value).isEqualTo("value1"); + } + + @Test + void getConfigMapThrowsExceptionForNonExistingKey() { + // Given + var configMap = new ConfigMapBuilder() + .withNewMetadata() + .withName("my-config") + .withNamespace("test") + .endMetadata() + .withData(Map.of("key1", "value1")) + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test/configmaps/my-config") + .andReturn(200, configMap) + .once(); + + // When/Then + var exception = assertThrows( + RuntimeException.class, + () -> k8sApiClient.getConfigMap("my-config", "non-existing-key") + ); + assertThat(exception.getMessage()).contains("Could not fetch non-existing-key"); + } + + // ======================================== + // Resource Management Tests + // ======================================== + + @Test + void applyYamlAppliesResourcesFromFile() throws IOException { + // Given + Path yamlFile = tempDir.resolve("test.yaml"); + Files.writeString( + yamlFile, """ + apiVersion: v1 + kind: Namespace + metadata: + name: test-ns + """ + ); + + server.expect() + .post() + .withPath("/api/v1/namespaces") + .andReturn( + 201, new NamespaceBuilder() + .withNewMetadata() + .withName("test-ns") + .endMetadata() + .build() + ) + .once(); + + // When + String result = k8sApiClient.applyYaml(yamlFile.toString()); + + // Then + assertThat(result).contains("Applied 1 resource(s)"); + } + + @Test + void applyYamlAppliesGenericKubernetesResourceViaDiscovery() throws IOException { + // Given + Path yamlFile = tempDir.resolve("app-project.yaml"); + Files.writeString( + yamlFile, """ + apiVersion: argoproj.io/v1alpha1 + kind: AppProject + metadata: + name: argocd + namespace: argocd + spec: + description: AppProject for ArgoCD-specific applications. + """ + ); + + server.expect() + .get() + .withPath("/apis") + .andReturn( + 200, Map.of( + "groups", List.of(Map.of( + "name", "argoproj.io", + "preferredVersion", Map.of("version", "v1alpha1"), + "versions", List.of(Map.of("version", "v1alpha1")) + )) + ) + ) + .once(); + + server.expect() + .get() + .withPath("/apis/argoproj.io/v1alpha1") + .andReturn( + 200, Map.of( + "resources", List.of(Map.of( + "name", "appprojects", + "singularName", "appproject", + "namespaced", true, + "kind", "AppProject", + "shortNames", List.of() + )) + ) + ) + .once(); + + GenericKubernetesResource appProject = new GenericKubernetesResourceBuilder() + .withApiVersion("argoproj.io/v1alpha1") + .withKind("AppProject") + .withNewMetadata() + .withName("argocd") + .withNamespace("argocd") + .endMetadata() + .addToAdditionalProperties( + "spec", Map.of("description", "AppProject for ArgoCD-specific applications.") + ) + .build(); + + AtomicBoolean appProjectWasApplied = new AtomicBoolean(false); + server.expect() + .post() + .withPath("/apis/argoproj.io/v1alpha1/namespaces/argocd/appprojects") + .andReply( + 201, request -> { + appProjectWasApplied.set(true); + return appProject; + } + ) + .once(); + + // When + String result = k8sApiClient.applyYaml(yamlFile.toString()); + + // Then + assertThat(result).contains("Applied 1 resource(s)"); + assertThat(appProjectWasApplied.get()).isTrue(); + } + + @Test + void applyYamlFallsBackToCrdWhenDiscoveryDoesNotExposeGenericResource() throws IOException { + // Given + Path yamlFile = tempDir.resolve("app-project-crd-fallback.yaml"); + Files.writeString( + yamlFile, """ + apiVersion: argoproj.io/v1alpha1 + kind: AppProject + metadata: + name: argocd + namespace: argocd + spec: + description: AppProject for ArgoCD-specific applications. + """ + ); + + server.expect() + .get() + .withPath("/apis") + .andReturn(200, Map.of("groups", List.of())) + .once(); + + server.expect() + .get() + .withPath("/apis/apiextensions.k8s.io/v1/customresourcedefinitions") + .andReturn( + 200, Map.of( + "apiVersion", "apiextensions.k8s.io/v1", + "kind", "CustomResourceDefinitionList", + "items", List.of(Map.of( + "apiVersion", "apiextensions.k8s.io/v1", + "kind", "CustomResourceDefinition", + "metadata", Map.of("name", "appprojects.argoproj.io"), + "spec", Map.of( + "group", "argoproj.io", + "scope", "Namespaced", + "names", Map.of( + "kind", "AppProject", + "plural", "appprojects", + "singular", "appproject" + ), + "versions", List.of(Map.of( + "name", "v1alpha1", + "served", true, + "storage", true + )) + ) + )) + ) + ) + .once(); + + GenericKubernetesResource appProject = new GenericKubernetesResourceBuilder() + .withApiVersion("argoproj.io/v1alpha1") + .withKind("AppProject") + .withNewMetadata() + .withName("argocd") + .withNamespace("argocd") + .endMetadata() + .addToAdditionalProperties( + "spec", Map.of("description", "AppProject for ArgoCD-specific applications.") + ) + .build(); + + AtomicBoolean appProjectWasApplied = new AtomicBoolean(false); + server.expect() + .post() + .withPath("/apis/argoproj.io/v1alpha1/namespaces/argocd/appprojects") + .andReply( + 201, request -> { + appProjectWasApplied.set(true); + return appProject; + } + ) + .once(); + + // When + String result = k8sApiClient.applyYaml(yamlFile.toString()); + + // Then + assertThat(result).contains("Applied 1 resource(s)"); + assertThat(appProjectWasApplied.get()).isTrue(); + } + + @Test + void applyYamlThrowsExceptionForNonExistingFileOrDirectory() { + // When/Then + var exception = assertThrows(RuntimeException.class, () -> k8sApiClient.applyYaml("/non/existing/file.yaml")); + + assertThat(exception.getMessage()).contains("File or directory not found"); + assertThat(exception.getMessage()).contains("/non/existing/file.yaml"); + } + + @Test + void labelAddsLabelsToResource() throws InterruptedException { + // Given + var pod = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("default") + .withLabels(Map.of("existing", "label")) + .endMetadata() + .build(); + + server.expect().get().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + server.expect().get().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + server.expect().patch().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + + // When + k8sApiClient.label( + "pod", "test-pod", "default", + new Tuple("app", "myapp"), + new Tuple("version", "1.0") + ); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("PATCH"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/default/pods/test-pod"); + assertThat(parseJsonList(request.getUtf8Body())).containsExactlyInAnyOrder( + Map.of("op", "add", "path", "/metadata/labels/app", "value", "myapp"), + Map.of("op", "add", "path", "/metadata/labels/version", "value", "1.0") + ); + } + + @Test + void labelRemoveRemovesLabelsFromResource() throws InterruptedException { + // Given + var pod = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("default") + .withLabels(Map.of("app", "myapp", "version", "1.0")) + .endMetadata() + .build(); + + server.expect().get().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + server.expect().get().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + server.expect().patch().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + + // When + k8sApiClient.labelRemove("pod", "test-pod", "default", "version"); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("PATCH"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/default/pods/test-pod"); + assertThat(parseJsonList(request.getUtf8Body())).containsExactly( + Map.of("op", "remove", "path", "/metadata/labels/version") + ); + } + + @Test + void patchPatchesResourceWithStrategicMerge() throws InterruptedException { + // Given + var pod = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("default") + .endMetadata() + .build(); + + server.expect().get().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + server.expect().patch().withPath("/api/v1/namespaces/default/pods/test-pod").andReturn(200, pod).once(); + + // When + k8sApiClient.patch( + "pod", "test-pod", "default", "strategic", Map.of( + "metadata", Map.of("labels", Map.of("new", "label"))) + ); + + // Then + var request = server.getLastRequest(); + Map body = parseJson(request.getUtf8Body()); + assertThat(request.getMethod()).isEqualTo("PATCH"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/default/pods/test-pod"); + assertThat((Map) ((Map) body.get("metadata")).get("labels")) + .containsEntry("new", "label"); + } + + @Test + void patchRejectsAnUnknownPatchType() { + var exception = assertThrows( + IllegalArgumentException.class, + () -> k8sApiClient.patch("pod", "test-pod", "default", "unknown", Map.of()) + ); + + assertThat(exception.getMessage()).isEqualTo("Unsupported patch type: unknown"); + } + + @Test + void deleteRemovesResourcesByLabelSelector() throws InterruptedException { + // Given + server.expect() + .delete() + .withPath("/api/v1/namespaces/test-ns/pods?labelSelector=app%3Dmyapp") + .andReturn(200, new StatusBuilder().build()) + .once(); + + // When + k8sApiClient.delete("pod", "test-ns", new Tuple("app", "myapp")); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("DELETE"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/test-ns/pods?labelSelector=app%3Dmyapp"); + } + + @Test + void deleteWithoutSelectorsRemovesAllResourcesOfTheType() throws InterruptedException { + server.expect() + .delete() + .withPath("/api/v1/namespaces/test-ns/pods") + .andReturn(200, new StatusBuilder().build()) + .once(); + + k8sApiClient.delete("pod", "test-ns"); + + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("DELETE"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/test-ns/pods"); + } + + @Test + void deleteRemovesSpecificResourceByName() throws InterruptedException { + // Given + server.expect() + .delete() + .withPath("/api/v1/namespaces/test-ns/pods/test-pod") + .andReturn(200, new StatusBuilder().build()) + .once(); + + // When + k8sApiClient.delete("pod", "test-ns", "test-pod"); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("DELETE"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/test-ns/pods/test-pod"); + } + + @Test + void runCreatesPodWithImage() { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/default/pods") + .andReturn( + 201, new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .endMetadata() + .build() + ) + .once(); + + // When + String result = k8sApiClient.run("test-pod", "nginx:latest", "", Map.of()); + + // Then + assertThat(result).contains("pod/test-pod created"); + } + + @Test + void runAppliesPodOverridesInsteadOfGeneratedParameterValues() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/jenkins/pods") + .andReturn( + 201, new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .endMetadata() + .build() + ) + .once(); + + String overrideImage = "bash:42"; + Map overrides = Map.of( + "spec", Map.of( + "containers", List.of(Map.of( + "name", "override-container", + "image", overrideImage, + "args", List.of("cat", "/etc/group"), + "volumeMounts", List.of(Map.of("name", "group", "mountPath", "/etc/group", "readOnly", true)) + )), + "nodeSelector", Map.of("node", "jenkins"), + "volumes", List.of(Map.of("name", "group", "hostPath", Map.of("path", "/etc/group"))) + ) + ); + + // When + k8sApiClient.run("test-pod", "nginx:latest", "jenkins", overrides); + + // Then + Map requestBody = parseJson(server.getLastRequest().getUtf8Body()); + assertThat(((Map) requestBody.get("metadata")).get("name")).isEqualTo("test-pod"); + assertThat(((Map) requestBody.get("metadata")).get("namespace")).isEqualTo("jenkins"); + assertThat(((Map) ((Map) requestBody.get("spec")).get("nodeSelector")).get( + "node")).isEqualTo("jenkins"); + + List> containers = (List>) ((Map) requestBody.get("spec")).get( + "containers"); + assertThat(containers).hasSize(1); + Map container = containers.get(0); + assertThat(container.get("name")).isEqualTo("override-container"); + assertThat(container.get("image")).isEqualTo("bash:42"); + List args = (List) container.get("args"); + assertThat(args).containsExactly("cat", "/etc/group"); + + List> volumeMounts = (List>) container.get("volumeMounts"); + Map volumeMount = volumeMounts.get(0); + assertThat(volumeMount.get("mountPath")).isEqualTo("/etc/group"); + assertThat(volumeMount.get("readOnly")).isEqualTo(true); + + List> volumes = (List>) ((Map) requestBody.get("spec")).get( + "volumes"); + Map volume = volumes.get(0); + assertThat(((Map) volume.get("hostPath")).get("path")).isEqualTo("/etc/group"); + } + + @Test + void runReturnsPodLogsAndRemovesPodForInteractiveRmMode() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/jenkins/pods") + .andReturn( + 201, new PodBuilder() + .withNewMetadata() + .withName("gid-pod") + .endMetadata() + .build() + ) + .once(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/jenkins/pods/gid-pod") + .andReturn( + 200, new PodBuilder() + .withNewMetadata() + .withName("gid-pod") + .endMetadata() + .withNewStatus() + .withPhase("Succeeded") + .endStatus() + .build() + ) + .once(); + + var succeededPod = new PodBuilder() + .withNewMetadata() + .withName("gid-pod") + .endMetadata() + .withNewStatus() + .withPhase("Succeeded") + .endStatus() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/jenkins/pods?fieldSelector=metadata.name%3Dgid-pod") + .andReturn(200, new PodListBuilder().withItems(succeededPod).build()) + .once(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/jenkins/pods/gid-pod/log?pretty=false") + .andReturn(200, "root:x:0:\ndocker:x:42:\n") + .once(); + + server.expect() + .delete() + .withPath("/api/v1/namespaces/jenkins/pods/gid-pod") + .andReturn(200, new StatusBuilder().build()) + .once(); + + // When + String result = k8sApiClient.run("gid-pod", "bash:42", "jenkins", "--restart=Never", "-ti", "--rm", "--quiet"); + + // Then + assertThat(result).isEqualTo("root:x:0:\ndocker:x:42:\n"); + + Map createRequest = parseJson(server.takeRequest().getUtf8Body()); + assertThat(((Map) createRequest.get("spec")).get("restartPolicy")).isEqualTo("Never"); + } + + // ======================================== + // Query Operations Tests + // ======================================== + + @Test + void getCustomResourceReturnsListOfCustomResources() { + // Given + server.expect() + .get() + .withPath("/apis") + .andReturn( + 200, Map.of( + "groups", List.of(Map.of( + "name", "example.io", + "preferredVersion", Map.of("version", "v1"), + "versions", List.of(Map.of("version", "v1")) + )) + ) + ) + .once(); + server.expect() + .get() + .withPath("/apis/example.io/v1") + .andReturn( + 200, Map.of( + "resources", List.of(Map.of( + "name", "widgets", + "singularName", "widget", + "namespaced", true, + "kind", "Widget", + "shortNames", List.of() + )) + ) + ) + .once(); + server.expect() + .get() + .withPath("/apis/example.io/v1/widgets") + .andReturn( + 200, Map.of( + "apiVersion", "example.io/v1", + "kind", "WidgetList", + "items", List.of( + Map.of( + "apiVersion", + "example.io/v1", + "kind", + "Widget", + "metadata", + Map.of("namespace", "ns-a", "name", "widget-a") + ), + Map.of( + "apiVersion", + "example.io/v1", + "kind", + "Widget", + "metadata", + Map.of("namespace", "ns-b", "name", "widget-b") + ) + ) + ) + ) + .once(); + + // When + List result = k8sApiClient.getCustomResource("widget"); + + // Then + assertThat(result).containsExactly( + new K8sClient.CustomResource("ns-a", "widget-a"), + new K8sClient.CustomResource("ns-b", "widget-b") + ); + } + + @Test + void getAnnotationRetrievesAnnotationValue() { + // Given + var pod = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("default") + .withAnnotations(Map.of("key1", "value1", "key2", "value2")) + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/default/pods/test-pod") + .andReturn(200, pod) + .once(); + + // When + String value = k8sApiClient.getAnnotation("pod", "test-pod", "key1", "default"); + + // Then + assertThat(value).isEqualTo("value1"); + } + + @Test + void getAnnotationReturnsNullForNonExistingAnnotation() { + // Given + var pod = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("default") + .withAnnotations(Map.of("key1", "value1")) + .endMetadata() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/default/pods/test-pod") + .andReturn(200, pod) + .once(); + + // When + String value = k8sApiClient.getAnnotation("pod", "test-pod", "non-existing", "default"); + + // Then + assertThat(value).isNull(); + } + + @Test + void getCurrentContextReturnsContextName() { + // When + String context = k8sApiClient.getCurrentContext(); + + // Then + assertThat(context).isNotNull(); + // Note: Actual value depends on mock client configuration + } + + // ======================================== + // Wait Operations Tests + // ======================================== + + @Test + void waitForResourcePhaseWaitsForPodToReachRunningPhase() { + // Given + var podRunning = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("test-ns") + .endMetadata() + .withNewStatus() + .withPhase("Running") + .endStatus() + .build(); + server.expect().get().withPath("/api/v1/namespaces/test-ns/pods/test-pod").andReturn(200, podRunning).once(); + + // When + k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 5, 1); + + // Then + assertThat(server.getRequestCount()).isEqualTo(1); + } + + @Test + void waitForResourcePhaseRetriesUntilPhaseIsReached() { + // Given + var podPending = new PodBuilder() + .withNewMetadata().withName("test-pod").withNamespace("test-ns").endMetadata() + .withNewStatus().withPhase("Pending").endStatus() + .build(); + var podRunning = new PodBuilder() + .withNewMetadata().withName("test-pod").withNamespace("test-ns").endMetadata() + .withNewStatus().withPhase("Running").endStatus() + .build(); + server.expect().get().withPath("/api/v1/namespaces/test-ns/pods/test-pod").andReturn(200, podPending).once(); + server.expect().get().withPath("/api/v1/namespaces/test-ns/pods/test-pod").andReturn(200, podPending).once(); + server.expect().get().withPath("/api/v1/namespaces/test-ns/pods/test-pod").andReturn(200, podRunning).once(); + + // When + k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 10, 1); + + // Then + assertThat(server.getRequestCount()).isEqualTo(3); + } + + @Test + void waitForResourcePhaseThrowsExceptionOnTimeout() { + // Given + var podPending = new PodBuilder() + .withNewMetadata() + .withName("test-pod") + .withNamespace("test-ns") + .endMetadata() + .withNewStatus() + .withPhase("Pending") + .endStatus() + .build(); + + server.expect() + .get() + .withPath("/api/v1/namespaces/test-ns/pods/test-pod") + .andReturn(200, podPending) + .always(); + + // When/Then + var exception = assertThrows( + RuntimeException.class, + () -> k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 2, 1) + ); + assertThat(exception.getMessage()).contains("Timeout reached"); + } + + @Test + void waitForResourcePhaseWithDefaultTimeout() { + // Given + var podRunning = new PodBuilder() + .withNewMetadata().withName("test-pod").withNamespace("test-ns").endMetadata() + .withNewStatus().withPhase("Running").endStatus() + .build(); + server.expect().get().withPath("/api/v1/namespaces/test-ns/pods/test-pod").andReturn(200, podRunning).always(); + + // When + k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running"); + + // Then + assertThat(server.getRequestCount()).isEqualTo(1); + } + + @Test + void waitForResourcePhaseValidatesParameters() { + // When/Then + var exception = assertThrows( + IllegalArgumentException.class, + () -> k8sApiClient.waitForResourcePhase("", "test-pod", "test-ns", "Running", 60, 1) + ); + assertThat(exception.getMessage()).contains("Resource type"); + + exception = assertThrows( + IllegalArgumentException.class, + () -> k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 0, 1) + ); + assertThat(exception.getMessage()).contains("Timeout"); + + exception = assertThrows( + IllegalArgumentException.class, + () -> k8sApiClient.waitForResourcePhase("pod", "test-pod", "test-ns", "Running", 60, 0) + ); + assertThat(exception.getMessage()).contains("check interval"); + } + + // ======================================== + // Edge Cases and Error Handling Tests + // ======================================== + + @Test + void resolvesDefaultNamespaceForEmptyString() throws InterruptedException { + // Given + server.expect() + .post() + .withPath("/api/v1/namespaces/default/secrets") + .andReturn( + 201, new SecretBuilder() + .withNewMetadata().withName("test-secret").withNamespace("default").endMetadata() + .withType("Opaque") + .build() + ) + .once(); + + // When + k8sApiClient.createSecret("Opaque", "test-secret", "", new Tuple("key", "value")); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("POST"); + assertThat(request.getPath()).isEqualTo("/api/v1/namespaces/default/secrets"); + Map body = parseJson(request.getUtf8Body()); + assertThat(((Map) body.get("metadata")).get("namespace")).isEqualTo("default"); + } + + @Test + void handlesMultipleResourceTypesInGetResourceClient() throws InterruptedException { + // Given + var deployment = new io.fabric8.kubernetes.api.model.apps.DeploymentBuilder() + .withNewMetadata().withName("test-deploy").withNamespace("default").endMetadata() + .build(); + server.expect() + .get() + .withPath("/apis/apps/v1/namespaces/default/deployments/test-deploy") + .andReturn(200, deployment) + .once(); + server.expect() + .delete() + .withPath("/apis/apps/v1/namespaces/default/deployments/test-deploy") + .andReturn(200, new StatusBuilder().build()) + .once(); + + // When + k8sApiClient.delete("deployment", "default", "test-deploy"); + + // Then + var request = server.getLastRequest(); + assertThat(request.getMethod()).isEqualTo("DELETE"); + assertThat(request.getPath()).isEqualTo("/apis/apps/v1/namespaces/default/deployments/test-deploy"); + } + + @Test + void customResourceClassIsImmutable() { + // When + var cr = new K8sClient.CustomResource("test-ns", "test-name"); + + // Then + assertThat(cr.namespace()).isEqualTo("test-ns"); + assertThat(cr.name()).isEqualTo("test-name"); + } + + @Test + void waitForResourcePhaseResolvesArgoCDCustomResourceViaDiscovery() { + // Given + server.expect() + .get() + .withPath("/apis") + .andReturn( + 200, Map.of( + "groups", List.of(Map.of( + "name", "argoproj.io", + "preferredVersion", Map.of("version", "v1beta1"), + "versions", List.of(Map.of("version", "v1beta1")) + )) + ) + ) + .once(); + + server.expect() + .get() + .withPath("/apis/argoproj.io/v1beta1") + .andReturn( + 200, Map.of( + "resources", List.of(Map.of( + "name", "argocds", + "singularName", "argocd", + "namespaced", true, + "kind", "ArgoCD", + "shortNames", List.of() + )) + ) + ) + .once(); + + GenericKubernetesResource argocdResource = new GenericKubernetesResourceBuilder() + .withApiVersion("argoproj.io/v1beta1") + .withKind("ArgoCD") + .withNewMetadata() + .withName("argocd") + .withNamespace("argocd") + .endMetadata() + .addToAdditionalProperties("status", Map.of("phase", "Available")) + .build(); + + AtomicBoolean argocdResourceWasRequested = new AtomicBoolean(false); + + server.expect() + .get() + .withPath("/apis/argoproj.io/v1beta1/namespaces/argocd/argocds/argocd") + .andReply( + 200, request -> { + argocdResourceWasRequested.set(true); + return argocdResource; + } + ) + .once(); + + // When + k8sApiClient.waitForResourcePhase("argocd", "argocd", "argocd", "Available", 5, 1); + + // Then + assertThat(argocdResourceWasRequested.get()).isTrue(); + assertThat(argocdResource.getApiVersion()).isEqualTo("argoproj.io/v1beta1"); + assertThat(argocdResource.getKind()).isEqualTo("ArgoCD"); + assertThat(argocdResource.getMetadata().getName()).isEqualTo("argocd"); + assertThat(argocdResource.getMetadata().getNamespace()).isEqualTo("argocd"); + } + + @Test + void throwsKubernetesApiResourceNotFoundExceptionWhenCustomResourceCannotBeResolved() { + // Given + server.expect() + .get() + .withPath("/apis") + .andReturn( + 200, Map.of( + "groups", List.of(Map.of( + "name", "argoproj.io", + "preferredVersion", Map.of("version", "v1beta1"), + "versions", List.of(Map.of("version", "v1beta1")) + )) + ) + ) + .once(); + + server.expect() + .get() + .withPath("/apis/argoproj.io/v1beta1") + .andReturn( + 200, Map.of( + "resources", List.of(Map.of( + "name", "argocds", + "singularName", "argocd", + "namespaced", true, + "kind", "ArgoCD", + "shortNames", List.of() + )) + ) + ) + .once(); + + // When/Then + var exception = assertThrows( + K8sClient.KubernetesApiResourceNotFoundException.class, + () -> k8sApiClient.getAnnotation("does-not-exist", "some-resource", "some-annotation", "argocd") + ); + + assertThat(exception.getMessage()) + .isEqualTo("No API resource found for custom resource type 'does-not-exist'"); + } + + private static Map parseJson(String json) { + try { + return OBJECT_MAPPER.readValue(json, JSON_MAP_TYPE); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + private static List> parseJsonList(String json) { + try { + return OBJECT_MAPPER.readValue(json, JSON_LIST_TYPE); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + +} diff --git a/src/test/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinitionTest.java b/src/test/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinitionTest.java new file mode 100644 index 000000000..e6f9a4e13 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/infrastructure/kubernetes/rbac/RbacDefinitionTest.java @@ -0,0 +1,341 @@ +package com.cloudogu.gitops.infrastructure.kubernetes.rbac; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.util.Arrays; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class RbacDefinitionTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + private final Config config = Config.fromMap(Map.of( + "scm", Map.of( + "scmManager", Map.of( + "username", "user", + "password", "pass", + "url", "http://localhost" + ) + ), + "application", Map.of( + "namePrefix", "", + "insecure", false, + "gitName", "Test User", + "gitEmail", "test@example.com" + ) + )); + + private final GitRepo repo = new GitRepo(config, null, "my-repo", new FileSystemUtils()); + + @Test + void generatesAtLeastOneRbacYamlFile() { + new RbacDefinition(Role.Variant.ARGOCD) + .withName("access") + .withNamespace("testing") + .withServiceAccountsFrom("testing", List.of("reader")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File outputDir = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac"); + File[] yamlFiles = outputDir.listFiles((FileFilter) file -> file.getName().endsWith(".yaml")); + List fileNames = Arrays.stream(yamlFiles).map(File::getName).toList(); + + assertThat(yamlFiles).isNotEmpty(); + assertThat(fileNames).anyMatch(name -> name.contains("role") || name.contains("rolebinding")); + } + + @Test + void failsIfNameIsMissing() { + IllegalArgumentException ex = assertThrows( + IllegalArgumentException.class, () -> + new RbacDefinition(Role.Variant.ARGOCD) + .withNamespace("testing") + .withServiceAccountsFrom("testing", List.of("reader")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate() + ); + + assertThat(ex.getMessage()).contains("name must not be blank"); + } + + @Test + void failsIfNamespaceIsMissing() { + IllegalArgumentException ex = assertThrows( + IllegalArgumentException.class, () -> + new RbacDefinition(Role.Variant.ARGOCD) + .withName("access") + .withServiceAccountsFrom("testing", List.of("reader")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate() + ); + + assertThat(ex.getMessage()).contains("namespace must not be blank"); + } + + @Test + void failsIfServiceAccountsAreEmpty() { + IllegalArgumentException ex = assertThrows( + IllegalArgumentException.class, () -> + new RbacDefinition(Role.Variant.ARGOCD) + .withName("access") + .withNamespace("testing") + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .withServiceAccounts(List.of()) + .generate() + ); + + assertThat(ex.getMessage()).contains("At least one service account"); + } + + @Test + void acceptsServiceAccountsViaWithServiceAccountsDirectly() { + ServiceAccountRef serviceAccount = new ServiceAccountRef("myns", "mysa"); + + new RbacDefinition(Role.Variant.ARGOCD) + .withName("direct") + .withNamespace("myns") + .withServiceAccounts(List.of(serviceAccount)) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File file = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac/rolebinding-direct-myns.yaml"); + assertThat(file).exists(); + } + + @Test + void customSubfolderIsRespected() { + String custom = "custom-dir"; + new RbacDefinition(Role.Variant.ARGOCD) + .withName("custom") + .withNamespace("testing") + .withSubfolder(custom) + .withServiceAccountsFrom("testing", List.of("reader")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File out = new File(repo.getAbsoluteLocalRepoTmpDir(), custom); + File[] yamlFiles = out.listFiles((FileFilter) file -> file.getName().endsWith(".yaml")); + List fileNames = Arrays.stream(yamlFiles).map(File::getName).toList(); + + assertThat(yamlFiles).isNotEmpty(); + assertThat(fileNames).anyMatch(name -> name.contains("role") || name.contains("rolebinding")); + } + + @Test + void multipleServiceAccountsAreRenderedCorrectly() { + new RbacDefinition(Role.Variant.ARGOCD) + .withName("multi") + .withNamespace("testing") + .withServiceAccountsFrom("testing", List.of("reader", "writer", "admin")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File[] files = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac").listFiles(); + List fileNames = Arrays.stream(files).map(File::getName).toList(); + assertThat(fileNames).anyMatch(name -> name.contains("role")); + } + + @Test + void customRoleAndBindingFileNamesAreRendered() { + new RbacDefinition(Role.Variant.ARGOCD) + .withName("myrole") + .withNamespace("custom-ns") + .withServiceAccountsFrom("custom-ns", List.of("sa1")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File outputDir = new File(repo.getAbsoluteLocalRepoTmpDir(), "rbac"); + List fileNames = Arrays.stream(outputDir.listFiles()).map(File::getName).toList(); + + assertThat(fileNames).contains("role-myrole-custom-ns.yaml", "rolebinding-myrole-custom-ns.yaml"); + } + + @Test + void subfolderCanBeNested() { + String nested = "some/nested/path"; + new RbacDefinition(Role.Variant.ARGOCD) + .withName("nestedtest") + .withNamespace("ns") + .withServiceAccountsFrom("ns", List.of("sa1")) + .withSubfolder(nested) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File outputDir = new File(repo.getAbsoluteLocalRepoTmpDir(), nested); + List fileNames = Arrays.stream(outputDir.listFiles()).map(File::getName).toList(); + + assertThat(fileNames).contains("role-nestedtest-ns.yaml", "rolebinding-nestedtest-ns.yaml"); + } + + @Test + void failsIfRepoIsNotSet() { + IllegalStateException ex = assertThrows( + IllegalStateException.class, () -> + new RbacDefinition(Role.Variant.ARGOCD) + .withName("failtest") + .withNamespace("ns") + .withServiceAccountsFrom("ns", List.of("sa1")) + .withTemplateConfig(rbacConfig()) + .generate() + ); + + assertThat(ex.getMessage()).contains("SCMM repo must be set using withRepo() before calling generate()"); + } + + @Test + @SuppressWarnings("unchecked") + void renderedRolebindingYamlContainsCorrectServiceAccounts() throws IOException { + List serviceAccounts = List.of("reader", "writer"); + String namespace = "rbac-test"; + + new RbacDefinition(Role.Variant.ARGOCD) + .withName("test") + .withNamespace(namespace) + .withServiceAccountsFrom(namespace, serviceAccounts) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + String path = "rbac/rolebinding-test-" + namespace + ".yaml"; + File file = new File(repo.getAbsoluteLocalRepoTmpDir(), path); + Map yaml = YAML_MAPPER.readValue(file, YAML_MAP_TYPE); + Map metadata = (Map) yaml.get("metadata"); + + assertThat(metadata.get("name")).isEqualTo("test"); + assertThat(metadata.get("namespace")).isEqualTo(namespace); + + List> subjects = (List>) yaml.get("subjects"); + List names = subjects.stream().map(subject -> (String) subject.get("name")).toList(); + assertThat(names).containsExactlyInAnyOrderElementsOf(serviceAccounts); + + List namespaces = subjects.stream().map(subject -> (String) subject.get("namespace")).toList(); + assertThat(namespaces).containsOnly(namespace); + + Map roleRef = (Map) yaml.get("roleRef"); + assertThat(roleRef.get("name")).isEqualTo("test"); + assertThat(roleRef.get("kind")).isEqualTo("Role"); + } + + @Test + @SuppressWarnings("unchecked") + void renderedRoleYamlContainsCorrectMetadata() throws IOException { + String name = "myrole"; + String namespace = "custom-ns"; + + new RbacDefinition(Role.Variant.ARGOCD) + .withName(name) + .withNamespace(namespace) + .withServiceAccountsFrom(namespace, List.of("sa1")) + .withRepo(repo) + .withTemplateConfig(rbacConfig()) + .generate(); + + String path = "rbac/role-" + name + "-" + namespace + ".yaml"; + File file = new File(repo.getAbsoluteLocalRepoTmpDir(), path); + Map yaml = YAML_MAPPER.readValue(file, YAML_MAP_TYPE); + Map metadata = (Map) yaml.get("metadata"); + + assertThat(metadata.get("name")).isEqualTo(name); + assertThat(metadata.get("namespace")).isEqualTo(namespace); + } + + @Test + @SuppressWarnings("unchecked") + void rendersNodeAccessRulesInArgocdRoleOnlyWhenNotOnOpenShift() throws IOException { + config.getApplication().setOpenshift(false); + + GitRepo tempRepo = new GitRepo(config, null, "rbac-test", new FileSystemUtils()); + + new RbacDefinition(Role.Variant.ARGOCD) + .withName("nodecheck") + .withNamespace("monitoring") + .withServiceAccountsFrom("monitoring", List.of("sa1")) + .withRepo(tempRepo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File roleFile = new File(tempRepo.getAbsoluteLocalRepoTmpDir(), "rbac/role-nodecheck-monitoring.yaml"); + Map yaml = YAML_MAPPER.readValue(roleFile, YAML_MAP_TYPE); + List> rules = (List>) yaml.get("rules"); + + assertThat(rules).anyMatch(rule -> { + List resources = (List) rule.get("resources"); + List verbs = (List) rule.get("verbs"); + return resources.containsAll(List.of("nodes", "nodes/metrics")) + && verbs.containsAll(List.of("get", "list", "watch")); + }); + } + + @Test + @SuppressWarnings("unchecked") + void doesNotRenderNodeAccessRulesInArgocdRoleWhenOnOpenShift() throws IOException { + config.getApplication().setOpenshift(true); + + GitRepo tempRepo = new GitRepo(config, null, "rbac-test", new FileSystemUtils()); + + new RbacDefinition(Role.Variant.ARGOCD) + .withName("nodecheck") + .withNamespace("monitoring") + .withServiceAccountsFrom("monitoring", List.of("sa1")) + .withRepo(tempRepo) + .withTemplateConfig(rbacConfig()) + .generate(); + + File roleFile = new File(tempRepo.getAbsoluteLocalRepoTmpDir(), "rbac/role-nodecheck-monitoring.yaml"); + Map yaml = YAML_MAPPER.readValue(roleFile, YAML_MAP_TYPE); + List> rules = (List>) yaml.get("rules"); + + assertThat(rules).noneMatch(rule -> { + List resources = (List) rule.get("resources"); + return resources.contains("nodes") && resources.contains("nodes/metrics"); + }); + } + + @Test + void failsIfConfigIsNotSet() { + IllegalArgumentException ex = assertThrows( + IllegalArgumentException.class, () -> + new RbacDefinition(Role.Variant.ARGOCD) + .withName("failtest") + .withNamespace("ns") + .withServiceAccountsFrom("ns", List.of("sa")) + .withRepo(repo) + .generate() + ); + + assertThat(ex.getMessage()).contains("Config must not be null"); + } + + private Map rbacConfig() { + return Map.of( + "application", Map.of("openshift", config.getApplication().getOpenshift()), + "features", Map.of( + "monitoring", Map.of("active", config.getFeatures().getMonitoring().getActive()), + "secrets", Map.of("active", config.getFeatures().getSecrets().getActive()) + ) + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/Polling.java b/src/test/java/com/cloudogu/gitops/integration/Polling.java new file mode 100644 index 000000000..466fa1c0a --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/Polling.java @@ -0,0 +1,65 @@ +package com.cloudogu.gitops.integration; + +import java.time.Duration; +import java.time.Instant; +import java.util.function.BooleanSupplier; + +public final class Polling { + + private Polling() { + } + + public static void until(BooleanSupplier condition, Duration timeout, Duration pollInterval) { + untilAsserted(() -> { + if (!condition.getAsBoolean()) { + throw new AssertionError("Condition is not fulfilled"); + } + }, timeout, pollInterval); + } + + public static void untilAsserted(Runnable assertion, Duration timeout, Duration pollInterval) { + if (timeout.isNegative()) { + throw new IllegalArgumentException("Timeout must not be negative"); + } + if (pollInterval.isNegative()) { + throw new IllegalArgumentException("Poll interval must not be negative"); + } + + Instant deadline = Instant.now().plus(timeout); + Throwable lastFailure; + do { + try { + assertion.run(); + return; + } catch (RuntimeException | AssertionError failure) { + lastFailure = failure; + } + + sleepUntilNextAttempt(pollInterval, deadline); + } while (Instant.now().isBefore(deadline)); + + throw new TimeoutException(timeout, lastFailure); + } + + private static void sleepUntilNextAttempt(Duration pollInterval, Instant deadline) { + long remainingMillis = Duration.between(Instant.now(), deadline).toMillis(); + if (remainingMillis <= 0) { + return; + } + + long sleepMillis = Math.min(pollInterval.toMillis(), remainingMillis); + try { + Thread.sleep(sleepMillis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Interrupted while waiting for condition", e); + } + } + + public static final class TimeoutException extends RuntimeException { + + private TimeoutException(Duration timeout, Throwable cause) { + super("Condition was not fulfilled within " + timeout, cause); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/PollingTest.java b/src/test/java/com/cloudogu/gitops/integration/PollingTest.java new file mode 100644 index 000000000..7e10fc331 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/PollingTest.java @@ -0,0 +1,57 @@ +package com.cloudogu.gitops.integration; + +import org.junit.jupiter.api.Test; + +import java.time.Duration; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class PollingTest { + + @Test + void retriesUntilAssertionSucceeds() { + AtomicInteger attempts = new AtomicInteger(); + + Polling.untilAsserted( + () -> assertThat(attempts.incrementAndGet()).isGreaterThanOrEqualTo(3), + Duration.ofSeconds(1), + Duration.ZERO + ); + + assertThat(attempts).hasValue(3); + } + + @Test + void reportsLastFailureOnTimeout() { + assertThatThrownBy(() -> Polling.untilAsserted( + () -> { + throw new AssertionError("not ready"); + }, + Duration.ZERO, + Duration.ZERO + )) + .isInstanceOf(Polling.TimeoutException.class) + .hasCauseInstanceOf(AssertionError.class) + .hasRootCauseMessage("not ready"); + } + + @Test + void restoresInterruptStatus() { + Thread.currentThread().interrupt(); + try { + assertThatThrownBy(() -> Polling.until( + () -> false, + Duration.ofSeconds(1), + Duration.ofMillis(1) + )) + .isInstanceOf(IllegalStateException.class) + .hasMessage("Interrupted while waiting for condition") + .hasCauseInstanceOf(InterruptedException.class); + assertThat(Thread.currentThread().isInterrupted()).isTrue(); + } finally { + Thread.interrupted(); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/TestK8sHelper.java b/src/test/java/com/cloudogu/gitops/integration/TestK8sHelper.java new file mode 100644 index 000000000..98b109153 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/TestK8sHelper.java @@ -0,0 +1,637 @@ +package com.cloudogu.gitops.integration; + +import io.fabric8.kubernetes.api.model.ContainerStateTerminated; +import io.fabric8.kubernetes.api.model.ContainerStateWaiting; +import io.fabric8.kubernetes.api.model.ContainerStatus; +import io.fabric8.kubernetes.api.model.Namespace; +import io.fabric8.kubernetes.api.model.Pod; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import io.fabric8.kubernetes.client.KubernetesClientException; +import io.fabric8.kubernetes.client.dsl.ExecListener; +import io.fabric8.kubernetes.client.dsl.ExecWatch; +import lombok.extern.slf4j.Slf4j; + +import java.io.ByteArrayOutputStream; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Comparator; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Predicate; +import java.util.stream.Collectors; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.fail; + +@Slf4j +public class TestK8sHelper { + + public static final int DEFAULT_WAIT_MINUTES = 5; + public static final int DEFAULT_POLL_SECONDS = 5; + public static final String RUNNING = "Running"; + public static final String FAILED = "Failed"; + public static final String SUCCEEDED = "Succeeded"; + public static final String COMPLETED = "Completed"; + public static final Set FATAL_CONTAINER_WAITING_REASONS = Set.of( + "CrashLoopBackOff", + "CreateContainerConfigError", + "CreateContainerError", + "ErrImagePull", + "ImageInspectError", + "ImagePullBackOff", + "InvalidImageName", + "RunContainerError" + ); + + private TestK8sHelper() { + } + + /** + * This method logs Namespace and contining Pods to namespace. + */ + public static void dumpNamespacesAndPods() { + StringBuffer sb = new StringBuffer("##### K8s Dump ##### \n"); + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + List pods = client.pods().inAnyNamespace().list().getItems(); + + // sort: namespace, pod-name + pods.sort(Comparator + .comparing( + (Pod pod) -> pod.getMetadata() == null ? null : pod.getMetadata().getNamespace(), + Comparator.nullsFirst(Comparator.naturalOrder()) + ) + .thenComparing( + pod -> pod.getMetadata() == null ? null : pod.getMetadata().getName(), + Comparator.nullsFirst(Comparator.naturalOrder()) + )); + + // group by namespace + Map> podsByNs = pods.stream() + .collect(Collectors.groupingBy( + pod -> pod.getMetadata() == null || pod.getMetadata().getNamespace() == null + ? "" + : pod.getMetadata().getNamespace(), + LinkedHashMap::new, + Collectors.toList() + )); + + podsByNs.forEach((namespace, namespacePods) -> { + sb.append("\n=== Namespace: ") + .append(namespace) + .append(" (") + .append(namespacePods.size()) + .append(") ===\n"); + + for (Pod pod : namespacePods) { + String name = pod.getMetadata() == null ? null : pod.getMetadata().getName(); + String phase = pod.getStatus() == null ? null : pod.getStatus().getPhase(); + String node = pod.getSpec() == null || pod.getSpec().getNodeName() == null + ? "-" + : pod.getSpec().getNodeName(); + String startTime = pod.getStatus() == null || pod.getStatus().getStartTime() == null + ? "-" + : pod.getStatus().getStartTime(); + + int restarts = pod.getStatus() == null || pod.getStatus().getContainerStatuses() == null + ? 0 + : pod.getStatus().getContainerStatuses().stream() + .filter(Objects::nonNull) + .map(ContainerStatus::getRestartCount) + .filter(Objects::nonNull) + .mapToInt(Integer::intValue) + .sum(); + + sb.append(String.format( + " %-60s phase=%-10s restarts=%-3s node=%-25s start=%s", + name, + phase, + restarts, + node, + startTime + )); + sb.append("\n"); + } + }); + } + log.info(sb.toString()); + } + + /** + * Executes command on container and returns result. + * + * @param client Kubernetes client + * @param ns namespace + * @param pod pod name + * @param container container name + * @param cmd command + * @return stdout of the command + */ + public static String execAndGetStdout( + KubernetesClient client, + String ns, + String pod, + String container, + String... cmd + ) { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + ByteArrayOutputStream err = new ByteArrayOutputStream(); + + CountDownLatch finished = new CountDownLatch(1); + AtomicReference failure = new AtomicReference<>(); + + ExecListener listener = new ExecListener() { + + @Override + public void onClose(int code, String reason) { + finished.countDown(); + } + }; + + try (ExecWatch watch = client.pods() + .inNamespace(ns) + .withName(pod) + .inContainer(container) + .writingOutput(out) + .writingError(err) + .usingListener(listener) + .exec(cmd)) { + + Polling.until( + () -> finished.getCount() == 0, + Duration.ofMinutes(5), + Duration.ofMillis(500) + ); + } catch (Exception e) { + throw new RuntimeException("Exec failed/timeout for pod " + ns + "/" + pod, e); + } + + if (failure.get() != null) { + throw new RuntimeException("Exec failure", failure.get()); + } + + String stderr = err.toString(StandardCharsets.UTF_8); + if (!stderr.isBlank()) { + log.error(stderr); + throw new RuntimeException(stderr); + } + + return out.toString(StandardCharsets.UTF_8); + } + + /** + * Checks the current Kubernetes state once and verifies that every matching pod is running. + * Use a waitFor... variant when the tested resource may still be rolling out. + * + * @param namespace namespace + */ + public static boolean checkAllPodsRunningInNamespace(String namespace) { + return checkAllPodsRunningInNamespace(namespace, ""); + } + + /** + * Checks the current Kubernetes state once and verifies that every matching pod is running. + * Use a waitFor... variant when the tested resource may still be rolling out. + * + * @param namespace namespace + * @param podNameStartsWith optional pod name prefix. Empty string matches all pods in the namespace. + */ + public static boolean checkAllPodsRunningInNamespace(String namespace, String podNameStartsWith) { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + List actualPods = client.pods().inNamespace(namespace).list().getItems().stream() + .filter(pod -> pod.getMetadata().getName().startsWith(podNameStartsWith)) + .collect(Collectors.toList()); + + assertThat(actualPods) + .withFailMessage("No pods found in namespace: %s with name %s", namespace, podNameStartsWith) + .isNotEmpty(); + + failOnFatalPods(namespace, actualPods); + + List notRunningPods = actualPods.stream() + .filter(pod -> !isPodRunning(pod)) + .collect(Collectors.toList()); + + assertThat(notRunningPods) + .withFailMessage("These pods in %s are not yet running: %s", namespace, describePods(notRunningPods)) + .isEmpty(); + return true; + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + return false; + } + } + + public static boolean waitForAllPodsRunningInNamespace(String namespace) { + return waitForAllPodsRunningInNamespace(namespace, "", DEFAULT_WAIT_MINUTES, TimeUnit.MINUTES); + } + + public static boolean waitForAllPodsRunningInNamespace(String namespace, String podNameStartsWith) { + return waitForAllPodsRunningInNamespace( + namespace, + podNameStartsWith, + DEFAULT_WAIT_MINUTES, + TimeUnit.MINUTES + ); + } + + public static boolean waitForAllPodsRunningInNamespace(String namespace, String podNameStartsWith, int timeout) { + return waitForAllPodsRunningInNamespace(namespace, podNameStartsWith, timeout, TimeUnit.MINUTES); + } + + /** + * Waits until at least one matching pod exists and all matching pods are running. + */ + public static boolean waitForAllPodsRunningInNamespace( + String namespace, + String podNameStartsWith, + int timeout, + TimeUnit timeoutUnit + ) { + Polling.untilAsserted( + () -> checkAllPodsRunningInNamespace(namespace, podNameStartsWith), + Duration.of(timeout, timeoutUnit.toChronoUnit()), + Duration.ofSeconds(DEFAULT_POLL_SECONDS) + ); + return true; + } + + /** + * Checks the current Kubernetes state once and verifies one running pod for each expected name prefix. + * Extra pods in the namespace are ignored, which keeps the check stable during rollouts. + */ + public static boolean checkPodPrefixesRunningInNamespace(String namespace, List expectedPodPrefixes) { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + List actualPods = client.pods().inNamespace(namespace).list().getItems(); + + for (String prefix : expectedPodPrefixes) { + List matchingPods = actualPods.stream() + .filter(pod -> pod.getMetadata().getName().startsWith(prefix)) + .collect(Collectors.toList()); + failIfOnlyFatalPodsMatch(namespace, prefix, matchingPods); + } + + List missingPods = expectedPodPrefixes.stream() + .filter(prefix -> actualPods.stream() + .noneMatch(pod -> pod.getMetadata().getName().startsWith( + prefix))) + .collect(Collectors.toList()); + + assertThat(missingPods) + .withFailMessage("Missing these pods in %s: %s", namespace, missingPods) + .isEmpty(); + + List notRunningPodPrefixes = expectedPodPrefixes.stream() + .filter(prefix -> { + List matchingPods = actualPods.stream() + .filter(pod -> pod.getMetadata().getName().startsWith( + prefix)) + .collect( + Collectors.toList()); + return matchingPods.stream().noneMatch( + TestK8sHelper::isPodRunning); + }) + .collect(Collectors.toList()); + + assertThat(notRunningPodPrefixes) + .withFailMessage( + "No running pod found in %s for: %s. Current pods: %s", + namespace, + notRunningPodPrefixes, + describePods(actualPods) + ) + .isEmpty(); + return true; + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + return false; + } + } + + public static boolean waitForPodPrefixesRunningInNamespace( + String namespace, + List expectedPodPrefixes + ) { + return waitForPodPrefixesRunningInNamespace( + namespace, + expectedPodPrefixes, + DEFAULT_WAIT_MINUTES, + TimeUnit.MINUTES + ); + } + + public static boolean waitForPodPrefixesRunningInNamespace( + String namespace, + List expectedPodPrefixes, + int timeout + ) { + return waitForPodPrefixesRunningInNamespace(namespace, expectedPodPrefixes, timeout, TimeUnit.MINUTES); + } + + /** + * Waits until each expected pod name prefix has at least one running pod. + */ + public static boolean waitForPodPrefixesRunningInNamespace( + String namespace, + List expectedPodPrefixes, + int timeout, + TimeUnit timeoutUnit + ) { + Polling.untilAsserted( + () -> checkPodPrefixesRunningInNamespace(namespace, expectedPodPrefixes), + Duration.of(timeout, timeoutUnit.toChronoUnit()), + Duration.ofSeconds(DEFAULT_POLL_SECONDS) + ); + return true; + } + + /** + * Checks the current Kubernetes state once using named pod matchers. + * Use this when simple prefixes are ambiguous, for example when one pod name is a prefix of another. + */ + public static boolean checkPodsMatchingRunningInNamespace( + String namespace, + Map> expectedPods + ) { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + List actualPods = client.pods().inNamespace(namespace).list().getItems(); + + for (Map.Entry> entry : expectedPods.entrySet()) { + List matchingPods = actualPods.stream() + .filter(pod -> entry.getValue().test(pod.getMetadata().getName())) + .collect(Collectors.toList()); + failIfOnlyFatalPodsMatch(namespace, entry.getKey(), matchingPods); + } + + List missingPods = expectedPods.entrySet().stream() + .filter(entry -> actualPods.stream() + .noneMatch(pod -> entry.getValue().test( + pod.getMetadata().getName()))) + .map(Map.Entry::getKey) + .collect(Collectors.toList()); + + assertThat(missingPods) + .withFailMessage("Missing these pods in %s: %s", namespace, missingPods) + .isEmpty(); + + List notRunningPods = expectedPods.entrySet().stream() + .filter(entry -> { + List matchingPods = actualPods.stream() + .filter(pod -> entry.getValue().test( + pod.getMetadata().getName())) + .collect(Collectors.toList()); + return matchingPods.stream().noneMatch(TestK8sHelper::isPodRunning); + }) + .map(Map.Entry::getKey) + .collect(Collectors.toList()); + + assertThat(notRunningPods) + .withFailMessage( + "No running pod found in %s for: %s. Current pods: %s", + namespace, + notRunningPods, + describePods(actualPods) + ) + .isEmpty(); + return true; + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + return false; + } + } + + public static boolean waitForPodsMatchingRunningInNamespace( + String namespace, + Map> expectedPods + ) { + return waitForPodsMatchingRunningInNamespace( + namespace, + expectedPods, + DEFAULT_WAIT_MINUTES, + TimeUnit.MINUTES + ); + } + + public static boolean waitForPodsMatchingRunningInNamespace( + String namespace, + Map> expectedPods, + int timeout + ) { + return waitForPodsMatchingRunningInNamespace(namespace, expectedPods, timeout, TimeUnit.MINUTES); + } + + /** + * Waits until every named pod matcher resolves to at least one running pod. + */ + public static boolean waitForPodsMatchingRunningInNamespace( + String namespace, + Map> expectedPods, + int timeout, + TimeUnit timeoutUnit + ) { + Polling.untilAsserted( + () -> checkPodsMatchingRunningInNamespace(namespace, expectedPods), + Duration.of(timeout, timeoutUnit.toChronoUnit()), + Duration.ofSeconds(DEFAULT_POLL_SECONDS) + ); + return true; + } + + /** + * Checks the current Kubernetes state once and verifies that all expected namespaces exist. + */ + public static boolean checkNamespacesExist(List expectedNamespaces) { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + List currentNamespaces = client.namespaces().list().getItems(); + + List missingNamespaces = expectedNamespaces.stream() + .filter(expectedNamespace -> currentNamespaces.stream() + .noneMatch( + currentNamespace -> + currentNamespace.getMetadata().getName().equals( + expectedNamespace))) + .collect(Collectors.toList()); + + assertThat(missingNamespaces) + .withFailMessage("Missing these Namespaces: %s", missingNamespaces) + .isEmpty(); + return true; + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + return false; + } + } + + public static boolean waitForNamespaces(List expectedNamespaces) { + return waitForNamespaces(expectedNamespaces, DEFAULT_WAIT_MINUTES, TimeUnit.MINUTES); + } + + public static boolean waitForNamespaces(List expectedNamespaces, int timeout) { + return waitForNamespaces(expectedNamespaces, timeout, TimeUnit.MINUTES); + } + + /** + * Waits until all expected namespaces exist. + */ + public static boolean waitForNamespaces( + List expectedNamespaces, + int timeout, + TimeUnit timeoutUnit + ) { + Polling.untilAsserted( + () -> checkNamespacesExist(expectedNamespaces), + Duration.of(timeout, timeoutUnit.toChronoUnit()), + Duration.ofSeconds(DEFAULT_POLL_SECONDS) + ); + return true; + } + + private static void failOnFatalPods(String namespace, Collection pods) { + Collection fatalPods = pods.stream() + .filter(TestK8sHelper::isPodFatal) + .collect(Collectors.toList()); + + if (!fatalPods.isEmpty()) { + throw new IllegalStateException( + "Pods in " + namespace + " reached a terminal or unrecoverable state: " + describePods(fatalPods) + ); + } + } + + private static void failIfOnlyFatalPodsMatch( + String namespace, + String expectedPod, + Collection matchingPods + ) { + if (matchingPods.isEmpty() || matchingPods.stream().anyMatch(TestK8sHelper::isPodRunning)) { + return; + } + + if (matchingPods.stream().allMatch(TestK8sHelper::isPodFatal)) { + throw new IllegalStateException( + "No recoverable pod found in " + namespace + " for " + expectedPod + + ". Matching pods: " + describePods(matchingPods) + ); + } + } + + private static boolean isPodRunning(Pod pod) { + String phase = pod.getStatus() == null ? null : pod.getStatus().getPhase(); + return (RUNNING.equals(phase) || SUCCEEDED.equals(phase) || COMPLETED.equals(phase)) + && !hasFatalContainerState(pod); + } + + private static boolean isPodFatal(Pod pod) { + String phase = pod.getStatus() == null ? null : pod.getStatus().getPhase(); + return FAILED.equals(phase) || hasFatalContainerState(pod); + } + + private static boolean hasFatalContainerState(Pod pod) { + return containerStatusesFor(pod).stream().anyMatch(status -> { + ContainerStateWaiting waiting = status.getState() == null ? null : status.getState().getWaiting(); + ContainerStateTerminated terminated = status.getState() == null ? null : status.getState().getTerminated(); + + return (waiting != null && FATAL_CONTAINER_WAITING_REASONS.contains(waiting.getReason())) + || (terminated != null + && terminated.getExitCode() != null + && terminated.getExitCode() != 0); + }); + } + + private static List containerStatusesFor(Pod pod) { + List statuses = new ArrayList<>(); + if (pod.getStatus() == null) { + return statuses; + } + if (pod.getStatus().getInitContainerStatuses() != null) { + statuses.addAll(pod.getStatus().getInitContainerStatuses()); + } + if (pod.getStatus().getContainerStatuses() != null) { + statuses.addAll(pod.getStatus().getContainerStatuses()); + } + return statuses; + } + + private static String describePods(Collection pods) { + return pods.stream() + .map(pod -> { + String podName = pod.getMetadata().getName(); + String phase = pod.getStatus() == null || pod.getStatus().getPhase() == null + ? "" + : pod.getStatus().getPhase(); + List containerStatuses = pod.getStatus() == null + ? null + : pod.getStatus().getContainerStatuses(); + String readyContainers; + if (containerStatuses == null) { + readyContainers = "0/0"; + } else { + long readyCount = containerStatuses.stream() + .filter(status -> Boolean.TRUE.equals(status.getReady())) + .count(); + readyContainers = readyCount + "/" + containerStatuses.size(); + } + String details = podProblemDetails(pod); + return podName + ":" + phase + ":ready=" + readyContainers + + (details.isEmpty() ? "" : ":" + details); + }) + .collect(Collectors.joining(", ")); + } + + private static String podProblemDetails(Pod pod) { + List details = new ArrayList<>(); + + if (pod.getStatus() != null && pod.getStatus().getReason() != null + && !pod.getStatus().getReason().isEmpty()) { + details.add("reason=" + pod.getStatus().getReason()); + } + if (pod.getStatus() != null && pod.getStatus().getMessage() != null + && !pod.getStatus().getMessage().isEmpty()) { + details.add("message=" + shorten(pod.getStatus().getMessage())); + } + + for (ContainerStatus status : containerStatusesFor(pod)) { + String containerState = describeContainerState(status); + if (containerState != null && !containerState.isEmpty()) { + details.add(containerState); + } + } + + return details.isEmpty() ? "" : "details=[" + String.join("; ", details) + "]"; + } + + private static String describeContainerState(ContainerStatus status) { + ContainerStateWaiting waiting = status.getState() == null ? null : status.getState().getWaiting(); + if (waiting != null) { + String reason = waiting.getReason() == null || waiting.getReason().isEmpty() + ? "" + : waiting.getReason(); + String message = waiting.getMessage() == null || waiting.getMessage().isEmpty() + ? "" + : " message=" + shorten(waiting.getMessage()); + return "container=" + status.getName() + " waiting=" + reason + message; + } + + ContainerStateTerminated terminated = status.getState() == null ? null : status.getState().getTerminated(); + if (terminated != null) { + String reason = terminated.getReason() == null || terminated.getReason().isEmpty() + ? "" + : terminated.getReason(); + return "container=" + status.getName() + " terminated=" + reason + " exit=" + terminated.getExitCode(); + } + + return null; + } + + private static String shorten(String value) { + return value.length() <= 160 ? value : value.substring(0, 157) + "..."; + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/ArgoCDOperatorProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/ArgoCDOperatorProfileTestIT.java new file mode 100644 index 000000000..e9f317c5e --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/ArgoCDOperatorProfileTestIT.java @@ -0,0 +1,74 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.Polling; +import com.cloudogu.gitops.integration.TestK8sHelper; +import io.fabric8.kubernetes.api.model.Namespace; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import io.fabric8.kubernetes.client.KubernetesClientException; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.time.Duration; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.fail; + +/** + * This tests can only be successfull, if one of theses profiles used. + * + *

To run locally: add -Dmicronaut.environments=operator-full to your execute configuration + */ +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "operator-full|operator-minimal") +public class ArgoCDOperatorProfileTestIT extends ProfileTestSetup { + + static String namespaceOperator = "argocd-operator-system"; + static String namespaceArgocd = "argocd"; + + @BeforeAll + static void labelTest() { + System.out.println("###### Integration ArgoCD Operator test ######"); + try { + Polling.until( + () -> TestK8sHelper.checkAllPodsRunningInNamespace( + namespaceOperator, + "argocd-operator-controller" + ) && TestK8sHelper.checkAllPodsRunningInNamespace( + namespaceArgocd, + "argocd-server" + ), + Duration.ofMinutes(40), + Duration.ofSeconds(5) + ); + } catch (Polling.TimeoutException timeoutEx) { + TestK8sHelper.dumpNamespacesAndPods(); + fail("Cluster not ready, sth false."); + } + } + + @Test + void ensureNamespaceExists() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + Namespace argocdNamespace = client.namespaces().withName(namespaceOperator).get(); + + assertThat(argocdNamespace).isNotNull(); + assertThat(argocdNamespace.getMetadata().getName()).isEqualTo(namespaceOperator); + } catch (KubernetesClientException ex) { + // Handle exception + fail("not expected exception was thrown. ", ex); + } + } + + @Test + void ensureOperatorNamespaceExists() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + Namespace argocdNamespace = client.namespaces().withName(namespaceArgocd).get(); + + assertThat(argocdNamespace).isNotNull(); + } catch (KubernetesClientException ex) { + // Handle exception + fail("not expected exception was thrown. ", ex); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/ArgoCDProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/ArgoCDProfileTestIT.java new file mode 100644 index 000000000..71e7f00cc --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/ArgoCDProfileTestIT.java @@ -0,0 +1,50 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.TestK8sHelper; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.util.List; + +/** + * These tests can only be successful if one of these profiles is used. + * + *

To run locally: add -Dmicronaut.environments=full to your execution configuration + */ +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full|minimal|operator-full|content-examples|operator-minimal|operator-content-examples") +public class ArgoCDProfileTestIT extends ProfileTestSetup { + + String namespace = "argocd"; + + @BeforeAll + static void labelTest() { + System.out.println("###### Integration ArgoCD test ######"); + } + + @Test + void ensureNamespaceExists() { + TestK8sHelper.waitForNamespaces(List.of(namespace), 40); + } + + /** chechs that ArgoCD pods running */ + @Test + void ensureArgoCDIsOnlineAndPodsAreRunning() { + String expectedPod1 = "argocd-application-controller"; + String expectedPod2 = "argocd-applicationset-controller"; + // String expectedPod3 = "argocd-notifications-controller"; // not stable + String expectedPod4 = "argocd-redis"; + String expectedPod5 = "argocd-repo-server"; + String expectedPod6 = "argocd-server"; + + List expectedPods = List.of( + expectedPod1, + expectedPod2, + /* expectedPod3, */ expectedPod4, + expectedPod5, + expectedPod6 + ); + + TestK8sHelper.waitForPodPrefixesRunningInNamespace(namespace, expectedPods, 40); + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/FullProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/FullProfileTestIT.java new file mode 100644 index 000000000..510d82236 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/FullProfileTestIT.java @@ -0,0 +1,128 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.TestK8sHelper; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import java.util.function.Predicate; + +/** + * This test ensures all Pods and Namespaces are available, runnning at a startet GOP with - more or less - defaulöt values. + * + *

To run locally: add -Dmicronaut.environments=full to your execute configuration + */ +@Slf4j +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full") +public class FullProfileTestIT extends ProfileTestSetup { + + /** Gets path to kubeconfig. */ + static final String EXAMPLE_APPS_NAMESPACE = "example-apps-staging"; + + @BeforeAll + static void labelMyTest() { + log.info("########### K8S SMOKE TESTS PROFILE full ###########"); + } + + @Test + void ensureExampleAppsAreRunning() { + TestK8sHelper.waitForAllPodsRunningInNamespace(EXAMPLE_APPS_NAMESPACE, "", 40, TimeUnit.MINUTES); + } + + @Test + void ensureJenkinsPodIsStarted() { + TestK8sHelper.waitForAllPodsRunningInNamespace("jenkins", "jenkins"); + } + + @Test + void ensureArgoCDIsOnlineAndPodsAreRunning() { + String expectedPod1 = "argocd-application-controller"; + String expectedPod2 = "argocd-applicationset-controller"; + // String expectedPod3 = "argocd-notifications-controller"; // not stable + String expectedPod4 = "argocd-redis"; + String expectedPod5 = "argocd-repo-server"; + String expectedPod6 = "argocd-server"; + + List expectedPods = List.of( + expectedPod1, + expectedPod2, + /* expectedPod3, */ expectedPod4, + expectedPod5, + expectedPod6 + ); + TestK8sHelper.waitForPodPrefixesRunningInNamespace("argocd", expectedPods); + } + + @Test + void ensureScmmPodIsStarted() { + TestK8sHelper.waitForAllPodsRunningInNamespace("scm-manager"); + } + + @Test + void ensureNamespacesExists() { + List expectedNamespaces = List.of( + "argocd", + "cert-manager", + "jenkins", + "registry", + "scm-manager", + "default", + "example-apps-production", + "example-apps-staging", + "ingress", + "kube-node-lease", + "kube-public", + "kube-system", + "monitoring", + "secrets" + ); + TestK8sHelper.waitForNamespaces(expectedNamespaces); + } + + /** tests searches for ingress services and ensure ingress is used as loadbalancer */ + @Test + void ensureIngressIsOnline() { + TestK8sHelper.waitForAllPodsRunningInNamespace("ingress", "traefik"); + } + + @Test + void ensureCertManagerIsOnline() { + TestK8sHelper.waitForAllPodsRunningInNamespace("cert-manager"); + } + + @Test + void ensureVaultIsOnline() { + TestK8sHelper.waitForAllPodsRunningInNamespace("secrets", "vault-0"); + } + + @Test + void ensureRegistryIsOnline() { + TestK8sHelper.waitForAllPodsRunningInNamespace("registry", "docker-registry"); + } + + @Test + void ensureExternalSecretsPodsRunning() { + Map> expectedPods = new LinkedHashMap<>(); + expectedPods.put( + "external-secrets", + podName -> podName.startsWith("external-secrets-") + && !podName.startsWith("external-secrets-webhook") + && !podName.startsWith("external-secrets-cert-controller") + ); + expectedPods.put( + "external-secrets-webhook", + podName -> podName.startsWith("external-secrets-webhook") + ); + expectedPods.put( + "external-secrets-cert-controller", + podName -> podName.startsWith("external-secrets-cert-controller") + ); + + TestK8sHelper.waitForPodsMatchingRunningInNamespace("secrets", expectedPods); + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/FullSecretsProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/FullSecretsProfileTestIT.java new file mode 100644 index 000000000..6b9d66882 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/FullSecretsProfileTestIT.java @@ -0,0 +1,127 @@ +package com.cloudogu.gitops.integration.profiles; + +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.springframework.security.crypto.bcrypt.BCrypt; + +import java.nio.charset.StandardCharsets; +import java.util.Base64; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Verifies that the full-secrets profile resolves credentials from Kubernetes Secrets and passes them to consumers. + */ +@Slf4j +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full-secrets") +public class FullSecretsProfileTestIT extends ProfileTestSetup { + + private static final String SOURCE_NAMESPACE = "gop-job"; + + @BeforeAll + static void labelMyTest() { + log.info("########### K8S CREDENTIAL TESTS PROFILE full-secrets ###########"); + } + + @Test + void usesApplicationCredentialsFromSecret() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + Secret source = secret(client, SOURCE_NAMESPACE, "argocd-credentials"); + String expectedUsername = secretValue(source, "username"); + String expectedPassword = secretValue(source, "password"); + + Secret argocdSecret = secret(client, "argocd", "argocd-secret"); + assertThat(BCrypt.checkpw(expectedPassword, secretValue(argocdSecret, "admin.password"))).isTrue(); + + assertCredentials( + secret(client, "monitoring", "grafana-admin-credentials"), + "admin-user", + "admin-password", + expectedUsername, + expectedPassword + ); + assertCredentials( + secret(client, "secrets", "vault-user-credentials"), + "username", + "password", + expectedUsername, + expectedPassword + ); + } + } + + @Test + void usesJenkinsCredentialsFromSecret() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + Secret source = secret(client, SOURCE_NAMESPACE, "jenkins-credentials"); + assertCredentials( + secret(client, "jenkins", "jenkins-credentials"), + "jenkins-admin-user", + "jenkins-admin-password", + secretValue(source, "username"), + secretValue(source, "password") + ); + } + } + + @Test + void usesScmManagerCredentialsFromSecret() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + Secret source = secret(client, SOURCE_NAMESPACE, "scm-tenant-credentials"); + assertCredentials( + secret(client, "scm-manager", "scm-manager-credentials"), + "SCM_WEBAPP_INITIALUSER", + "SCM_WEBAPP_INITIALPASSWORD", + secretValue(source, "username"), + secretValue(source, "password") + ); + } + } + + @Test + void usesRegistryCredentialsFromSecret() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + Secret source = secret(client, SOURCE_NAMESPACE, "registry-credentials"); + String dockerConfig = secretValue(secret(client, "jenkins", "proxy-registry"), ".dockerconfigjson"); + + assertThat(dockerConfig) + .contains(secretValue(source, "username")) + .contains(secretValue(source, "password")); + } + } + + private static void assertCredentials( + Secret secret, + String usernameKey, + String passwordKey, + String expectedUsername, + String expectedPassword + ) { + assertThat(secretValue(secret, usernameKey)).isEqualTo(expectedUsername); + assertThat(secretValue(secret, passwordKey)).isEqualTo(expectedPassword); + } + + private static Secret secret(KubernetesClient client, String namespace, String name) { + Secret secret = client.secrets().inNamespace(namespace).withName(name).get(); + assertThat(secret) + .as("Secret %s/%s", namespace, name) + .isNotNull(); + return secret; + } + + private static String secretValue(Secret secret, String key) { + if (secret.getStringData() != null && secret.getStringData().containsKey(key)) { + return secret.getStringData().get(key); + } + + assertThat(secret.getData()) + .as("Secret %s/%s data", secret.getMetadata().getNamespace(), secret.getMetadata().getName()) + .containsKey(key); + return new String(Base64.getDecoder().decode(secret.getData().get(key)), StandardCharsets.UTF_8); + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/MandantProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/MandantProfileTestIT.java new file mode 100644 index 000000000..e4d80c99f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/MandantProfileTestIT.java @@ -0,0 +1,134 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.Polling; +import com.cloudogu.gitops.integration.TestK8sHelper; +import io.fabric8.kubernetes.api.model.Namespace; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import io.fabric8.kubernetes.client.KubernetesClientException; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.DisabledIfSystemProperty; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.time.Duration; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.fail; + +/** + * This test ensures all Pods and Namespaces are available, runnning at a startet GOP with - more or less - defaulöt values. + * + *

To run locally: add -Dmicronaut.environments=full to your execute configuration + */ +@Slf4j +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") +public class MandantProfileTestIT extends ProfileTestSetup { + + /** Gets path to kubeconfig. */ + static final String RUNNING = "Running"; + static final String TENANT_POD_FOR_CONDITION = "argocd-application-controller"; + static final String TENANT_NAMESPACE_ARGOCD = "tenant1-argocd"; + static final String TENANT_NAMESPACE_REGISTRY = "tenant1-registry"; + static final String TENANT_NAMESPACE_SCM = "tenant1-scm-manager"; + + @BeforeAll + static void labelMyTest() { + log.info("########### PROFILE Operator-Mandants ###########"); + waitUntilTenantIsReady(); + } + + private static void waitUntilTenantIsReady() { + // tenant is created very late after running GOP twice! + Polling.until( + () -> TestK8sHelper.checkAllPodsRunningInNamespace( + TENANT_NAMESPACE_REGISTRY, + "docker-registry" + ) && TestK8sHelper.checkAllPodsRunningInNamespace( + TENANT_NAMESPACE_SCM, + "scmm-" + ), + Duration.ofMinutes(40), + Duration.ofSeconds(5) + ); + } + + @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") + // just local + @Test + void ensureJenkinsPodIsStartedOnTenant() { + TestK8sHelper.waitForAllPodsRunningInNamespace("tenant1-jenkins", "jenkins"); + } + + @Test + void ensureRegistryPodIsStartedOnTenant() { + TestK8sHelper.waitForAllPodsRunningInNamespace("tenant1-registry", "docker-registry"); + } + + @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") + // just local + @Test + void ensureArgocdPodsAreStartedOnTenant() { + String argocdNamespace = TENANT_NAMESPACE_ARGOCD; + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-application-controller"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-applicationset-controller"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-redis"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-repo-server"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-server"); + } + + @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") + // just local + @Test + void ensureArgocdPodsAreStartedOnCentral() { + String argocdNamespace = "argocd"; + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-application-controller"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-applicationset-controller"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-redis"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-repo-server"); + TestK8sHelper.waitForAllPodsRunningInNamespace(argocdNamespace, "argocd-server"); + } + + @Test + void ensureScmmPodIsStarted() { + TestK8sHelper.waitForAllPodsRunningInNamespace("scm-manager"); + } + + @DisabledIfSystemProperty(named = "micronaut.environments", matches = "operator-mandants") + // just local + @Test + void ensureNamespacesExists() { + List expectedNamespaces = List.of( + "argocd", + "argocd-operator-system", + "scm-manager", + "default", + "tenant1-argocd", + "tenant1-jenkins", + "tenant1-registry", + "tenant1-example-apps-staging", + "tenant1-example-apps-staging", + "tenant1-scm-manager", + "kube-node-lease", + "kube-public", + "kube-system" + ); + + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + List currentNamespaces = client.namespaces().list().getItems(); + + // 1. Verify all expected pods are present + List missingNamespaces = expectedNamespaces.stream() + .filter(prefix -> currentNamespaces.stream() + .noneMatch(namespace -> namespace.getMetadata().getName().startsWith(prefix))) + .toList(); + assertThat(missingNamespaces) + .as("Missing these Namespace: %s", missingNamespaces) + .isEmpty(); + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/PetclinicProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/PetclinicProfileTestIT.java new file mode 100644 index 000000000..02bf5e991 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/PetclinicProfileTestIT.java @@ -0,0 +1,106 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.Polling; +import com.cloudogu.gitops.integration.TestK8sHelper; +import io.fabric8.kubernetes.api.model.Service; +import io.fabric8.kubernetes.api.model.networking.v1.Ingress; +import io.fabric8.kubernetes.api.model.networking.v1.IngressRule; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import io.fabric8.kubernetes.client.KubernetesClientException; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.DisabledIfSystemProperty; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.util.List; +import java.util.concurrent.TimeUnit; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.fail; + +/** + * This tests can only be successfull, if one of theses profiles used. + * + *

To run locally: add -Dmicronaut.environments=content-examples to your execute configuration + */ +@Slf4j +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full|operator-full|content-examples") +public class PetclinicProfileTestIT extends ProfileTestSetup { + + static String exampleStagingNs = "example-apps-staging"; + + @BeforeAll + static void labelTest() { + System.out.println("###### Testing Petclinic ######"); + // petclinic need most of time to run. If online, we can start all tests. + try { + waitForContentExamplePrerequisites(); + TestK8sHelper.waitForAllPodsRunningInNamespace(exampleStagingNs, "", 40, TimeUnit.MINUTES); + } catch (Polling.TimeoutException timeoutEx) { + TestK8sHelper.dumpNamespacesAndPods(); + fail("Cluster not ready, sth false.", timeoutEx); + } + } + + private static void waitForContentExamplePrerequisites() { + TestK8sHelper.waitForNamespaces(List.of("jenkins", "registry", exampleStagingNs)); + TestK8sHelper.waitForAllPodsRunningInNamespace("registry", "docker-registry", 40); + TestK8sHelper.waitForAllPodsRunningInNamespace("jenkins", "jenkins", 40); + } + + @Test + void ensurePetclinicIsRunningOnStages() { + TestK8sHelper.waitForAllPodsRunningInNamespace(exampleStagingNs); + } + + @DisabledIfSystemProperty(named = "micronaut.environments", matches = "full|operator-full|content-examples") + @Test + void ensurePetclinicIngressIsOnline() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + String nameOfServiceAndIngress = "spring-petclinic-plain"; + // check Ingress + Ingress ingress = client.network() + .v1() + .ingresses() + .inNamespace(exampleStagingNs) + .withName(nameOfServiceAndIngress) + .get(); + + assertThat(ingress) + .as("Ingress '%s' not found in '%s'", nameOfServiceAndIngress, exampleStagingNs) + .isNotNull(); + + List rules = ingress.getSpec() == null || ingress.getSpec().getRules() == null + ? List.of() + : ingress.getSpec().getRules(); + List hosts = rules.stream() + .map(rule -> rule == null ? null : rule.getHost()) + .filter(host -> host != null && !host.isEmpty()) + .toList(); + + // in this case, petclinic do not care about prefix + assertThat(hosts.get(0)).contains("petclinic"); + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + } + } + + @DisabledIfSystemProperty(named = "micronaut.environments", matches = "full|operator-full|content-examples") + @Test + void ensurePetclinicServidsdsdceIsOnline() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + // Check Service + String nameOfServiceAndIngress = "spring-petclinic-plain"; + Service service = client.services() + .inNamespace(exampleStagingNs) + .withName(nameOfServiceAndIngress) + .get(); + + assertThat(service).isNotNull(); + } catch (KubernetesClientException ex) { + fail("Unexpected Kubernetes exception", ex); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/PrefixProfileTestIT.java b/src/test/java/com/cloudogu/gitops/integration/profiles/PrefixProfileTestIT.java new file mode 100644 index 000000000..cfe7acef6 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/PrefixProfileTestIT.java @@ -0,0 +1,79 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.Polling; +import com.cloudogu.gitops.integration.TestK8sHelper; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.util.List; +import java.util.concurrent.TimeUnit; + +import static org.assertj.core.api.Assertions.fail; + +/** + * These tests can only be successful if one of these profiles is used. + * + *

To run locally: add -Dmicronaut.environments=full-prefix to your execution configuration + */ +@Slf4j +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full-prefix") +public class PrefixProfileTestIT extends ProfileTestSetup { + // is used for pre-condition + static String exampleStagingNs = "my-prefix-example-apps-staging"; + static String argocdNs = "my-prefix-argocd"; + String scmManagerNs = "my-prefix-scm-manager"; + String registryNs = "my-prefix-registry"; + String ingressNs = "my-prefix-ingress"; + /* Jenking can not start ingress*/ + static String certManagerNs = "my-prefix-cert-manager"; + String jenkinsNs = "my-prefix-jenkins"; + static String monitoringNs = "my-prefix-monitoring"; + String secretsNs = "my-prefix-secrets"; + String exampleProductionNs = "my-prefix-example-apps-production"; + + @BeforeAll + static void labelTest() { + log.info("###### Integration test for Prefix ######"); + + try { + TestK8sHelper.waitForAllPodsRunningInNamespace(certManagerNs, "", 40, TimeUnit.MINUTES); + } catch (Polling.TimeoutException timeoutEx) { + TestK8sHelper.dumpNamespacesAndPods(); + fail("Cluster not ready, sth false.", timeoutEx); + } + } + + @Test + void ensureNamespacesExistWithPrefix() { + List expectedNamespaces = List.of( + argocdNs, + scmManagerNs, + registryNs, + ingressNs, + certManagerNs, + jenkinsNs, + monitoringNs, + secretsNs, + exampleProductionNs, + exampleStagingNs + ); + + TestK8sHelper.waitForNamespaces(expectedNamespaces); + } + + @Test + void ensurePodsAreRunningInPrefixedNamespaces() { + List namespacesToCheck = List.of( + argocdNs, + scmManagerNs, + registryNs, + certManagerNs, + monitoringNs + ); + for (String namespace : namespacesToCheck) { + TestK8sHelper.waitForAllPodsRunningInNamespace(namespace); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/profiles/ProfileTestSetup.java b/src/test/java/com/cloudogu/gitops/integration/profiles/ProfileTestSetup.java new file mode 100644 index 000000000..a5e44ec35 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/profiles/ProfileTestSetup.java @@ -0,0 +1,34 @@ +package com.cloudogu.gitops.integration.profiles; + +import com.cloudogu.gitops.integration.TestK8sHelper; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.extension.ExtensionContext; +import org.junit.jupiter.api.extension.RegisterExtension; +import org.junit.jupiter.api.extension.TestWatcher; + +/** + * Common setup to dump K8s content after failing tests. + */ +@Slf4j +public class ProfileTestSetup implements TestWatcher { + + private static boolean anyTestFailed = false; + + @RegisterExtension + final TestWatcher watcher = this; + + @Override + public void testFailed(ExtensionContext context, Throwable cause) { + anyTestFailed = true; + } + + @AfterAll + static void afterAllOnlyOnFailure() { + // if one test fails, logging is necessary + if (anyTestFailed) { + log.info("############## K8s dump ##############"); + TestK8sHelper.dumpNamespacesAndPods(); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/tools/CertManagerTestIT.java b/src/test/java/com/cloudogu/gitops/integration/tools/CertManagerTestIT.java new file mode 100644 index 000000000..9e415193d --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/tools/CertManagerTestIT.java @@ -0,0 +1,66 @@ +package com.cloudogu.gitops.integration.tools; + +import com.cloudogu.gitops.integration.TestK8sHelper; +import lombok.extern.slf4j.Slf4j; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Predicate; + +/** + * This class checks if cert-manager is started well. + * Cert-Manager contains own namespace ('cert-manager') which owns and 3 Pods: + */ +@Slf4j +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full") +// TODO: why not in ArgoCD Operator? Clarify +public class CertManagerTestIT extends KubernetesApiTestSetup { + + String namespace = "cert-manager"; + + @Override + boolean isReadyToStartTests() { + try { + return TestK8sHelper.checkPodsMatchingRunningInNamespace(namespace, expectedCertManagerPods()); + } catch (AssertionError ignored) { + return false; + } + } + + @BeforeAll + static void labelTest() { + System.out.println("###### CERT-MANAGER ######"); + } + + @Test + void ensureNamespaceExists() { + TestK8sHelper.waitForNamespaces(List.of(namespace)); + } + + @Test + void ensureAllCertManagerPodsAreExist() { + TestK8sHelper.waitForPodsMatchingRunningInNamespace(namespace, expectedCertManagerPods()); + } + + @Test + void ensureExpectedCertManagerPodsAreRunning() { + TestK8sHelper.waitForPodsMatchingRunningInNamespace(namespace, expectedCertManagerPods()); + } + + private static Map> expectedCertManagerPods() { + Map> expectedPods = new LinkedHashMap<>(); + expectedPods.put( + "cert-manager", + podName -> podName.startsWith("cert-manager-") + && !podName.startsWith("cert-manager-cainjector") + && !podName.startsWith("cert-manager-webhook") + ); + expectedPods.put("cert-manager-cainjector", podName -> podName.startsWith("cert-manager-cainjector")); + expectedPods.put("cert-manager-webhook", podName -> podName.startsWith("cert-manager-webhook")); + return expectedPods; + } +} diff --git a/src/test/java/com/cloudogu/gitops/integration/tools/KubernetesApiTestSetup.java b/src/test/java/com/cloudogu/gitops/integration/tools/KubernetesApiTestSetup.java new file mode 100644 index 000000000..103b6fd0e --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/tools/KubernetesApiTestSetup.java @@ -0,0 +1,44 @@ +package com.cloudogu.gitops.integration.tools; + +import com.cloudogu.gitops.integration.Polling; +import org.junit.jupiter.api.BeforeEach; + +import java.time.Duration; + +public abstract class KubernetesApiTestSetup { + + int TIME_TO_WAIT = 12; + int RETRY_SECONDS = 30; + + /** + * Waits until the Kubernetes resources required by the integration test are ready. + */ + @BeforeEach + void waitUntilReady() { + Polling.until( + this::waitingCondition, + maxWaitTimeInMinutes(TIME_TO_WAIT), + pollIntervallSeconds(RETRY_SECONDS) + ); + } + + private Duration pollIntervallSeconds(int time) { + return Duration.ofSeconds(time); + } + + private Duration maxWaitTimeInMinutes(int time) { + return Duration.ofMinutes(time); + } + + boolean waitingCondition() { + System.out.println("waiting for pods"); + return isReadyToStartTests(); + } + + /** + * This condition is to override, if test has to wait, i.e. ArgoCD has to do its GitOps magic. + * + * @return whether the tests are ready to start + */ + abstract boolean isReadyToStartTests(); +} diff --git a/src/test/java/com/cloudogu/gitops/integration/tools/MonitoringTestIT.java b/src/test/java/com/cloudogu/gitops/integration/tools/MonitoringTestIT.java new file mode 100644 index 000000000..33f16d1c8 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/integration/tools/MonitoringTestIT.java @@ -0,0 +1,88 @@ +package com.cloudogu.gitops.integration.tools; + +import com.cloudogu.gitops.integration.TestK8sHelper; +import io.fabric8.kubernetes.api.model.Pod; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.KubernetesClientBuilder; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * This class checks if Prometheus is started well. + * Prometheus contains own namespace ('monitoring') which owns and 3 Pods: + * - Grafana + * - Operator + * - prometheus-stack + */ +@EnabledIfSystemProperty(named = "micronaut.environments", matches = "full") +public class MonitoringTestIT extends KubernetesApiTestSetup { + + String namespace = "monitoring"; + String grafanaPod = "kube-prometheus-stack-grafana"; + String operatorPod = "kube-prometheus-stack-operator"; + String prometheusPod = "prometheus-kube-prometheus-stack-prometheus"; + + @Override + boolean isReadyToStartTests() { + try { + return TestK8sHelper.checkAllPodsRunningInNamespace(namespace, grafanaPod); + } catch (AssertionError ignored) { + return false; + } + } + + @BeforeAll + static void labelTest() { + System.out.println("###### PROMETHEUS ######"); + } + + @Test + void ensureNamespaceExists() { + TestK8sHelper.waitForNamespaces(List.of(namespace)); + } + + @Test + void ensureGrafanaIsStarted() { + TestK8sHelper.waitForAllPodsRunningInNamespace(namespace, grafanaPod); + } + + @Test + void ensureOperatorIsStarted() { + TestK8sHelper.waitForAllPodsRunningInNamespace(namespace, operatorPod); + } + + @Disabled("not start on jenkins") + @Test + void ensureMonitoringIsStarted() { + List pods = listPods(); + assertThat(pods).isNotEmpty(); + + Pod prometheus = null; + for (Pod pod : pods) { + if (pod.getMetadata().getName().contains(prometheusPod)) { + prometheus = pod; + break; + } + } + assertThat(prometheus).isNotNull(); + assertThat(prometheus.getStatus().getPhase()).isEqualTo("Running"); + } + + @Disabled("jenkins got only 2") + @Test + void ensureNamespaceGot3Pods() { + assertThat(listPods()).hasSize(3); + } + + private List listPods() { + try (KubernetesClient client = new KubernetesClientBuilder().build()) { + return client.pods().inNamespace(namespace).list().getItems(); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/TestLogger.java b/src/test/java/com/cloudogu/gitops/testhelper/TestLogger.java new file mode 100644 index 000000000..71619a143 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/TestLogger.java @@ -0,0 +1,79 @@ +package com.cloudogu.gitops.testhelper; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.LoggerContext; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import lombok.Getter; +import org.slf4j.LoggerFactory; + +import java.util.Collections; +import java.util.List; +import java.util.stream.Collectors; + +public class TestLogger { + + private final Class loggerInClass; + + @Getter + private final MemoryAppender logs; + + public TestLogger(Class clazz) { + this(clazz, Level.DEBUG); + } + + public TestLogger(Class clazz, Level logLevel) { + this.loggerInClass = clazz; + this.logs = new MemoryAppender(); + + Logger logger = (Logger) LoggerFactory.getLogger(loggerInClass); + logs.setContext((LoggerContext) LoggerFactory.getILoggerFactory()); + logger.setLevel(logLevel); + logger.addAppender(logs); + logs.start(); + } + + public void changeLogLevel(Level logLevel) { + Logger logger = (Logger) LoggerFactory.getLogger(loggerInClass); + logger.setLevel(logLevel); + } + + public static class MemoryAppender extends ListAppender { + + public void reset() { + list.clear(); + } + + public boolean contains(String string, Level level) { + return list.stream() + .anyMatch(event -> event.toString().contains(string) && event.getLevel().equals(level)); + } + + public int countEventsForLogger(String loggerName) { + return (int) list.stream() + .filter(event -> event.getLoggerName().contains(loggerName)) + .count(); + } + + public List search(String string) { + return list.stream() + .filter(event -> event.toString().contains(string)) + .collect(Collectors.toList()); + } + + public List search(String string, Level level) { + return list.stream() + .filter(event -> event.toString().contains(string) && event.getLevel().equals(level)) + .collect(Collectors.toList()); + } + + public int getSize() { + return list.size(); + } + + public List getLoggedEvents() { + return Collections.unmodifiableList(list); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/git/GitHandlerForTests.java b/src/test/java/com/cloudogu/gitops/testhelper/git/GitHandlerForTests.java new file mode 100644 index 000000000..e49306000 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/git/GitHandlerForTests.java @@ -0,0 +1,45 @@ +package com.cloudogu.gitops.testhelper.git; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.utils.K8sClientForTest; +import com.cloudogu.gitops.utils.NetworkingUtils; + +public class GitHandlerForTests extends GitHandler { + + private final GitProvider tenantProvider; + private final GitProvider centralProvider; + + public GitHandlerForTests(GitProvider tenantProvider) { + this(tenantProvider, null); + } + + public GitHandlerForTests(GitProvider tenantProvider, GitProvider centralProvider) { + this(tenantProvider, centralProvider, new K8sClientForTest()); + } + + private GitHandlerForTests( + GitProvider tenantProvider, + GitProvider centralProvider, + K8sClientForTest k8sClient) { + super(k8sClient, new NetworkingUtils(), new Config(), new CredentialsResolver(k8sClient)); + this.tenantProvider = tenantProvider; + this.centralProvider = centralProvider; + setTenant(tenantProvider); + setCentral(centralProvider); + } + + @Override + public void prepareProviders(DeploymentContext context) { + // Inject the test providers into the base class before running the real logic + setTenant(tenantProvider); + setCentral(context.isMultiTenant() ? centralProvider : null); + } + + @Override + public void validate() { + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/git/GitlabMock.java b/src/test/java/com/cloudogu/gitops/testhelper/git/GitlabMock.java new file mode 100644 index 000000000..496225cda --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/git/GitlabMock.java @@ -0,0 +1,96 @@ +package com.cloudogu.gitops.testhelper.git; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import lombok.Getter; +import lombok.Setter; + +import java.net.URI; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +public class GitlabMock implements GitProvider { + + @Getter + @Setter + private URI base = URI.create("https://example.com/group"); + + @Getter + @Setter + private String namePrefix = ""; + + @Getter + private final List createdRepos = new ArrayList<>(); + + @Getter + private final List> permissionCalls = new ArrayList<>(); + + @Override + public boolean createRepository(String repoTarget, String description, boolean initialize) { + createdRepos.add(repoTarget); + return true; + } + + @Override + public boolean createRepository(String repoTarget, String description) { + return createRepository(repoTarget, description, true); + } + + @Override + public void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { + Map permissionCall = new LinkedHashMap<>(); + permissionCall.put("repoTarget", repoTarget); + permissionCall.put("principal", principal); + permissionCall.put("role", role); + permissionCall.put("scope", scope); + permissionCalls.add(permissionCall); + } + + @Override + public String repoUrl(String repoTarget, RepoUrlScope scope) { + String cleaned = base.toString().replaceAll("/+$", ""); + return cleaned + "/" + repoTarget + ".git"; + } + + @Override + public String repoPrefix() { + String cleaned = base.toString().replaceAll("/+$", ""); + String prefix = namePrefix == null ? "" : namePrefix; + return cleaned + "/" + prefix; + } + + @Override + public URI prometheusMetricsEndpoint() { + return base; + } + + @Override + public Credentials getCredentials() { + return new Credentials("gitops", "gitops"); + } + + @Override + public String getUrl() { + return base.toString(); + } + + @Override + public String getProtocol() { + return base.getScheme(); + } + + @Override + public String getHost() { + return base.getHost(); + } + + @Override + public String getGitOpsUsername() { + return "gitops"; + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/git/ScmManagerProviderMock.java b/src/test/java/com/cloudogu/gitops/testhelper/git/ScmManagerProviderMock.java new file mode 100644 index 000000000..d7021807c --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/git/ScmManagerProviderMock.java @@ -0,0 +1,149 @@ +package com.cloudogu.gitops.testhelper.git; + +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.git.providers.AccessRole; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.RepoUrlScope; +import com.cloudogu.gitops.infrastructure.git.providers.Scope; +import lombok.Getter; +import lombok.Setter; + +import java.net.URI; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Lightweight test double for SCM-Manager via the GitProvider interface. + *

+ * Models the SCM-Manager specific GitProvider behavior that is relevant for tests: + * - configurable in-cluster and client base URLs + * - optional namePrefix to model tenant behavior + * - repository URL/prefix generation + * - createRepository/setRepositoryPermission call recording + */ +public class ScmManagerProviderMock implements GitProvider { + + private final Set initOnceRepos = new HashSet<>(); + private final Map createCalls = new HashMap<>(); + + @Getter + @Setter + private URI inClusterBase = URI.create("http://scmm.scm-manager.svc.cluster.local/scm"); + + @Getter + @Setter + private URI clientBase = URI.create("http://localhost:8080/scm"); + + @Getter + @Setter + private String namePrefix = ""; + + @Setter + private Credentials credentials = new Credentials("gitops", "gitops"); + + @Setter + private String gitOpsUsername = "gitops"; + + @Getter + @Setter + private URI prometheus = URI.create("http://localhost:8080/scm/api/v2/metrics/prometheus"); + + @Getter + private final List createdRepos = new ArrayList<>(); + + @Getter + private final List> permissionCalls = new ArrayList<>(); + + /** + * Optional sequence to control createRepository() return values per call. + *

+ * Empty list means: return true by default. + */ + @Getter + @Setter + private List nextCreateResults = new ArrayList<>(); + + public void initOnceRepo(String fullName) { + initOnceRepos.add(fullName); + } + + public void clearInitOnce() { + initOnceRepos.clear(); + createCalls.clear(); + } + + @Override + public boolean createRepository(String repoTarget, String description, boolean initialize) { + createdRepos.add(repoTarget); + + if (initOnceRepos.contains(repoTarget)) { + return createCalls.merge(repoTarget, 1, Integer::sum) == 1; + } + + return nextCreateResults == null || nextCreateResults.isEmpty() ? true : nextCreateResults.remove(0); + } + + @Override + public void setRepositoryPermission(String repoTarget, String principal, AccessRole role, Scope scope) { + Map permissionCall = new LinkedHashMap<>(); + permissionCall.put("repoTarget", repoTarget); + permissionCall.put("principal", principal); + permissionCall.put("role", role); + permissionCall.put("scope", scope); + permissionCalls.add(permissionCall); + } + + @Override + public String repoUrl(String repoTarget, RepoUrlScope scope) { + URI base = scope == RepoUrlScope.CLIENT ? clientBase : inClusterBase; + String cleanedBase = withoutTrailingSlash(base).toString(); + return cleanedBase + "/repo/" + repoTarget; + } + + @Override + public String repoPrefix() { + String base = withoutTrailingSlash(inClusterBase).toString(); + String prefix = namePrefix == null ? "" : namePrefix; + return base + "/repo/" + prefix; + } + + @Override + public Credentials getCredentials() { + return credentials; + } + + @Override + public URI prometheusMetricsEndpoint() { + return prometheus; + } + + @Override + public String getUrl() { + return withoutTrailingSlash(inClusterBase).toString(); + } + + @Override + public String getProtocol() { + return inClusterBase.getScheme(); + } + + @Override + public String getHost() { + return inClusterBase.getHost(); + } + + @Override + public String getGitOpsUsername() { + return gitOpsUsername; + } + + private static URI withoutTrailingSlash(URI uri) { + String value = uri.toString(); + return URI.create(value.endsWith("/") ? value.substring(0, value.length() - 1) : value); + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/git/TestGitProvider.java b/src/test/java/com/cloudogu/gitops/testhelper/git/TestGitProvider.java new file mode 100644 index 000000000..25f4ea3fb --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/git/TestGitProvider.java @@ -0,0 +1,61 @@ +package com.cloudogu.gitops.testhelper.git; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; + +import java.net.URI; +import java.util.LinkedHashMap; +import java.util.Map; + +public final class TestGitProvider { + + private TestGitProvider() { + } + + public static Map buildProviders(Config config) { + boolean dedicatedInstance = Boolean.TRUE.equals(config.getMultiTenant().getUseDedicatedInstance()); + + if (config.getScm().getScmProviderType() == ScmProviderType.GITLAB) { + GitlabMock gitlab = new GitlabMock(); + gitlab.setBase(URI.create(config.getScm().getGitlab().getUrl())); + gitlab.setNamePrefix(config.getApplication().getNamePrefix()); + return providers(gitlab, dedicatedInstance ? gitlab : null); + } + + String namePrefix = config.getApplication().getNamePrefix(); + String serviceDns = "http://scmm." + namePrefix + "scm-manager.svc.cluster.local/scm"; + String tenantInCluster = valueOrDefault( + config.getScm().getScmManager() == null ? null : config.getScm().getScmManager().getUrl(), + serviceDns + ); + String centralInCluster = valueOrDefault( + config.getMultiTenant().getScmManager() == null ? null : config.getMultiTenant().getScmManager().getUrl(), + tenantInCluster + ); + + ScmManagerProviderMock tenant = scmManagerProvider(tenantInCluster, namePrefix); + ScmManagerProviderMock central = dedicatedInstance + ? scmManagerProvider(centralInCluster, namePrefix) + : null; + return providers(tenant, central); + } + + private static ScmManagerProviderMock scmManagerProvider(String inClusterBase, String namePrefix) { + ScmManagerProviderMock provider = new ScmManagerProviderMock(); + provider.setInClusterBase(URI.create(inClusterBase)); + provider.setNamePrefix(namePrefix); + return provider; + } + + private static Map providers(GitProvider tenant, GitProvider central) { + Map providers = new LinkedHashMap<>(); + providers.put("tenant", tenant); + providers.put("central", central); + return providers; + } + + private static String valueOrDefault(String value, String defaultValue) { + return value == null || value.isEmpty() ? defaultValue : value; + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/git/TestGitRepoFactory.java b/src/test/java/com/cloudogu/gitops/testhelper/git/TestGitRepoFactory.java new file mode 100644 index 000000000..3ffe45c4f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/git/TestGitRepoFactory.java @@ -0,0 +1,85 @@ +package com.cloudogu.gitops.testhelper.git; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.GitRepoFactory; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.utils.FileSystemUtils; +import lombok.Getter; +import org.apache.commons.io.FileUtils; +import org.eclipse.jgit.api.errors.GitAPIException; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.util.HashMap; +import java.util.Map; + +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.spy; + +public class TestGitRepoFactory extends GitRepoFactory { + + @Getter + private final Map repos = new HashMap<>(); + + public TestGitRepoFactory(Config config, FileSystemUtils fileSystemUtils) { + super(config, fileSystemUtils); + } + + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + if (gitProvider == null) { + throw new IllegalStateException( + "No GitProvider provided for repo '" + repoTarget + "'." + ); + } + + GitRepo existingRepo = repos.get(repoTarget); + if (existingRepo != null) { + return existingRepo; + } + + String prefixedRepoTarget = config.getApplication().getNamePrefix() + repoTarget; + GitRepo repoNew = new GitRepo(config, gitProvider, prefixedRepoTarget, fileSystemUtils) { + private String remoteGitRepoUrl = ""; + + @Override + public String getGitRepositoryUrl() { + if (remoteGitRepoUrl.isEmpty()) { + try { + File tempDir = Files.createTempDirectory("gitops-playground-repocopy").toFile(); + tempDir.deleteOnExit(); + String originalRepo = System.getProperty("user.dir") + + "/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/"; + + FileUtils.copyDirectory(new File(originalRepo), tempDir); + remoteGitRepoUrl = "file://" + tempDir.getAbsolutePath(); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + return remoteGitRepoUrl; + } + }; + + GitRepo spyRepo = spy(repoNew); + + // Test-only: remove local clone target before cloning to avoid "not empty" errors + try { + doAnswer(invocation -> { + File target = new File(spyRepo.getAbsoluteLocalRepoTmpDir()); + if (target.exists()) { + FileUtils.deleteDirectory(target); + } + return invocation.callRealMethod(); + }).when(spyRepo).cloneRepo(); + } catch (GitAPIException e) { + throw new IllegalStateException("Failed to configure GitRepo test spy", e); + } + + repos.put(repoTarget, spyRepo); + return spyRepo; + } +} diff --git a/src/test/java/com/cloudogu/gitops/testhelper/git/TestScmManagerApiClient.java b/src/test/java/com/cloudogu/gitops/testhelper/git/TestScmManagerApiClient.java new file mode 100644 index 000000000..9537532d0 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/testhelper/git/TestScmManagerApiClient.java @@ -0,0 +1,105 @@ +package com.cloudogu.gitops.testhelper.git; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.Permission; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.RepositoryApi; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient; +import lombok.Getter; +import okhttp3.internal.http.RealResponseBody; +import okio.BufferedSource; +import org.mockito.ArgumentMatchers; +import retrofit2.Call; +import retrofit2.Response; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.HashSet; +import java.util.Set; + +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class TestScmManagerApiClient extends ScmManagerApiClient { + + @Getter + private final RepositoryApi repositoryApi = mock(RepositoryApi.class); + private final Set createdRepos = new HashSet<>(); + private final Set createdPermissions = new HashSet<>(); + + public TestScmManagerApiClient(Config config) { + super( + config.getScm().getScmManager().getUrl(), + new Credentials( + config.getScm().getScmManager().getUsername(), + config.getScm().getScmManager().getPassword() + ), + null + ); + } + + @Override + public RepositoryApi repositoryApi() { + return repositoryApi; + } + + /** + * Make all repo API calls return created on the first call and exists on subsequent calls for each repo. + */ + public void mockRepoApiBehaviour() { + Call responseCreated = mockSuccessfulResponse(201); + Call responseExists = mockErrorResponse(409); + + when(repositoryApi.create(ArgumentMatchers.any(Repository.class), anyBoolean())) + .thenAnswer(invocation -> { + Repository repo = invocation.getArgument(0); + if (createdRepos.contains(repo.getFullRepoName())) { + return responseExists; + } + createdRepos.add(repo.getFullRepoName()); + return responseCreated; + }); + + when(repositoryApi.createPermission(anyString(), anyString(), ArgumentMatchers.any(Permission.class))) + .thenAnswer(invocation -> { + String namespace = invocation.getArgument(0); + String name = invocation.getArgument(1); + String repository = namespace + "/" + name; + if (createdPermissions.contains(repository)) { + return responseExists; + } + createdPermissions.add(repository); + return responseCreated; + }); + } + + @SuppressWarnings("unchecked") + public static Call mockSuccessfulResponse(int expectedReturnCode) { + Call expectedCall = mock(Call.class); + try { + when(expectedCall.execute()).thenReturn(Response.success(expectedReturnCode, null)); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + return expectedCall; + } + + @SuppressWarnings("unchecked") + public static Call mockErrorResponse(int expectedReturnCode) { + Call expectedCall = mock(Call.class); + // Response is a final class that cannot be mocked. + Response errorResponse = Response.error( + expectedReturnCode, + new RealResponseBody("dontcare", 0, mock(BufferedSource.class)) + ); + try { + when(expectedCall.execute()).thenReturn(errorResponse); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + return expectedCall; + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/CertManagerTest.java b/src/test/java/com/cloudogu/gitops/tools/CertManagerTest.java new file mode 100644 index 000000000..77895df5e --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/CertManagerTest.java @@ -0,0 +1,284 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; +import java.util.Objects; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class CertManagerTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + private final String chartVersion = "1.19.4"; + private final Config config = Config.fromMap(Map.of( + "features", Map.of( + "certManager", Map.of( + "active", true, + "helm", Map.of( + "chart", "cert-manager", + "repoURL", "https://charts.jetstack.io", + "version", chartVersion + ) + ) + ) + )); + + private Path temporaryYamlFile; + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + private File clusterResourcesRepoDir; + private RepositoryWorkspace repositoryWorkspace; + private DeploymentContext deploymentContext; + + private final ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + + @Mock + private Deployer deploymentStrategy; + @Mock + private AirGappedUtils airGappedUtils; + @Mock + private GitHandler gitHandler; + @Mock + private GitProvider gitProvider; + @Mock + private ImagePullSecretCreator imagePullSecretCreator; + + @Test + void helmReleaseIsInstalled() throws GitAPIException { + install(createCertManager()); + + verify(deploymentStrategy).deployFeature( + "https://charts.jetstack.io", + "cert-manager", + "cert-manager", + chartVersion, + "cert-manager", + "cert-manager", + temporaryYamlFile, + RepoType.HELM, + false, + deploymentContext, + repositoryWorkspace + ); + } + + @Test + void preparesCertManagerAppContentInClusterResourcesWorkspaceWithoutCopyingTemplates() throws GitAPIException { + install(createCertManager()); + + assertThat(new File(clusterResourcesRepoDir, "apps/cert-manager")).exists(); + assertThat(new File(clusterResourcesRepoDir, "apps/cert-manager/templates")).doesNotExist(); + } + + @Test + void setsPodResourceLimitsAndRequests() throws GitAPIException, IOException { + config.getApplication().setPodResources(true); + + install(createCertManager()); + + assertThat((Map) parseActualYaml().get("resources")).containsKeys("limits", "requests"); + assertThat((Map) ((Map) parseActualYaml().get("cainjector")).get("resources")) + .containsKeys("limits", "requests"); + assertThat((Map) ((Map) parseActualYaml().get("webhook")).get("resources")) + .containsKeys("limits", "requests"); + } + + @Test + void isDisabledViaActiveFlag() throws GitAPIException { + config.getFeatures().getCertManager().setActive(false); + + assertFalse(createCertManager().isEnabled(new ContextBuilder(config).build())); + } + + @Test + void helmReleaseIsInstalledInAirGappedMode() throws GitAPIException, IOException { + when(gitHandler.getResourcesScm()).thenReturn(gitProvider); + when(gitProvider.repoUrl(any())).thenReturn("http://scmm.scm-manager.svc.cluster.local/scm/repo/a/b"); + + config.getApplication().setMirrorRepos(true); + when(airGappedUtils.mirrorHelmRepoToGit(any(HelmChartConfig.class))).thenReturn("a/b"); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path sourceChart = rootChartsFolder.resolve("cert-manager"); + Files.createDirectories(sourceChart); + + Map chartYaml = Map.of("version", chartVersion); + fileSystemUtils.writeYaml(chartYaml, sourceChart.resolve("Chart.yaml").toFile()); + + install(createCertManager()); + + ArgumentCaptor helmConfig = ArgumentCaptor.forClass(HelmChartConfig.class); + verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()); + assertThat(helmConfig.getValue().chart()).isEqualTo("cert-manager"); + // check existing value, but its not used in deploy. + assertThat(helmConfig.getValue().repoURL()).isEqualTo("https://charts.jetstack.io"); + assertThat(helmConfig.getValue().version()).isEqualTo(chartVersion); + // important check: scmmRepoUrl is overridden with our values. + verify(deploymentStrategy).deployFeature( + "http://scmm.scm-manager.svc.cluster.local/scm/repo/a/b", + "cert-manager", + ".", + chartVersion, + "cert-manager", + "cert-manager", + temporaryYamlFile, + RepoType.GIT, + false, + deploymentContext, + repositoryWorkspace + ); + } + + @Test + void checkImagesAreOverriddes() throws GitAPIException, IOException { + when(gitHandler.getResourcesScm()).thenReturn(gitProvider); + when(gitProvider.repoUrl(any())).thenReturn("http://test"); + + // Prep + config.getApplication().setMirrorRepos(true); + // test values + config.getFeatures().getCertManager().getHelm() + .setImage("this.is.my.registry:30000/this.is.my.repository/myImage:1"); + config.getFeatures().getCertManager().getHelm() + .setWebhookImage("this.is.my.registry:30000/this.is.my.repository/myWebhook:2"); + config.getFeatures().getCertManager().getHelm() + .setCainjectorImage("this.is.my.registry:30000/this.is.my.repository/myCainjectorImage:3"); + config.getFeatures().getCertManager().getHelm() + .setAcmeSolverImage("this.is.my.registry:30000/this.is.my.repository/myAcmeSolverImage:4"); + config.getFeatures().getCertManager().getHelm() + .setStartupAPICheckImage("this.is.my.registry:30000/this.is.my.repository/myStartupAPICheckImage:5"); + + when(airGappedUtils.mirrorHelmRepoToGit(any(HelmChartConfig.class))).thenReturn("a/b"); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path sourceChart = rootChartsFolder.resolve("cert-manager"); + Files.createDirectories(sourceChart); + + Map chartYaml = Map.of("version", chartVersion); + fileSystemUtils.writeYaml(chartYaml, sourceChart.resolve("Chart.yaml").toFile()); + + install(createCertManager()); + + // Cert-Manager + Map image = (Map) parseActualYaml().get("image"); + assertThat(Objects.toString(image.get("repository"), null)) + .isEqualTo("this.is.my.registry:30000/this.is.my.repository/myImage"); + assertThat(Objects.toString(image.get("tag"), null)).isEqualTo("1"); + // webhook + Map webhookImage = (Map) ((Map) parseActualYaml().get("webhook")).get( + "image"); + assertThat(Objects.toString(webhookImage.get("repository"), null)) + .isEqualTo("this.is.my.registry:30000/this.is.my.repository/myWebhook"); + assertThat(Objects.toString(webhookImage.get("tag"), null)).isEqualTo("2"); + // cainjector + Map cainjectorImage = (Map) ((Map) parseActualYaml().get( + "cainjector")).get("image"); + assertThat(Objects.toString(cainjectorImage.get("repository"), null)) + .isEqualTo("this.is.my.registry:30000/this.is.my.repository/myCainjectorImage"); + assertThat(Objects.toString(cainjectorImage.get("tag"), null)).isEqualTo("3"); + // acmesolver + Map acmeSolverImage = (Map) ((Map) parseActualYaml().get( + "acmesolver")).get("image"); + assertThat(Objects.toString(acmeSolverImage.get("repository"), null)) + .isEqualTo("this.is.my.registry:30000/this.is.my.repository/myAcmeSolverImage"); + assertThat(Objects.toString(acmeSolverImage.get("tag"), null)).isEqualTo("4"); + // startupapicheck + Map startupApiCheckImage = (Map) ((Map) parseActualYaml().get( + "startupapicheck")).get("image"); + assertThat(Objects.toString(startupApiCheckImage.get("repository"), null)) + .isEqualTo("this.is.my.registry:30000/this.is.my.repository/myStartupAPICheckImage"); + assertThat(Objects.toString(startupApiCheckImage.get("tag"), null)).isEqualTo("5"); + } + + private CertManager createCertManager() throws GitAPIException { + // We use the real FileSystemUtils and not a mock to make sure file editing works as expected + FileSystemUtils testFileSystemUtils = new FileSystemUtils() { + @Override + public Path writeTempFile(Map mapValues) { + Path ret = super.writeTempFile(mapValues); + temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")); + return ret; + } + }; + + TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, testFileSystemUtils) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, gitProvider); + clusterResourcesRepoDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoProvider.create( + "argocd/cluster-resources", + scmManagerMock + ); + + repositoryWorkspace = spy(new RepositoryWorkspace(clusterResourcesRepo)); + doNothing().when(repositoryWorkspace).commitAndPushClusterResourcesChanges(anyString()); + + return new CertManager( + testFileSystemUtils, + deploymentStrategy, + airGappedUtils, + gitHandler, + imagePullSecretCreator, + new CertManagerToolConfigMapper(config) + ); + } + + private boolean install(CertManager certManager) { + deploymentContext = new ContextBuilder(config).build(); + return certManager.execute(deploymentContext, repositoryWorkspace); + } + + private Map parseActualYaml() throws IOException { + return YAML_MAPPER.readValue(temporaryYamlFile.toFile(), YAML_MAP_TYPE); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/CertManagerToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/CertManagerToolConfigMapperTest.java new file mode 100644 index 000000000..39378a030 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/CertManagerToolConfigMapperTest.java @@ -0,0 +1,101 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class CertManagerToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getApplication().setPodResources(true); + config.getApplication().setSkipCrds(true); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setPassword("registry-password"); + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setNamespace("certificates"); + config.getFeatures().getCertManager().setIssuer("production-issuer"); + config.getFeatures().getCertManager().getHelm().setRepoURL("https://cert.example.org"); + config.getFeatures().getCertManager().getHelm().setChart("cert-chart"); + config.getFeatures().getCertManager().getHelm().setVersion("1.2.3"); + config.getFeatures().getCertManager().getHelm().setValues(Map.of("replicas", 2)); + config.getFeatures().getCertManager().getHelm().setImage("cert-image"); + config.getFeatures().getCertManager().getHelm().setWebhookImage("webhook-image"); + config.getFeatures().getCertManager().getHelm().setCainjectorImage("cainjector-image"); + config.getFeatures().getCertManager().getHelm().setAcmeSolverImage("solver-image"); + config.getFeatures().getCertManager().getHelm().setStartupAPICheckImage("startup-image"); + + CertManagerToolConfig actual = new CertManagerToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(CertManagerToolConfig.builder() + .active(true) + .namespace("test-certificates") + .helm(HelmChartConfig.builder() + .repoURL("https://cert.example.org") + .chart("cert-chart") + .version("1.2.3") + .values(Map.of("replicas", 2)) + .localHelmChartFolder("/charts") + .build()) + .imagePullSecret(imagePullSecret()) + .templateConfig(Map.of( + "application", + Map.of("podResources", true, "skipCrds", true), + "features", + Map.of( + "certManager", Map.of( + "issuer", "production-issuer", + "helm", Map.of( + "image", "cert-image", + "webhookImage", "webhook-image", + "cainjectorImage", "cainjector-image", + "acmeSolverImage", "solver-image", + "startupAPICheckImage", "startup-image" + ) + ) + ), + "registry", + Map.of("createImagePullSecrets", true) + )) + .build()); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.SINGLE_TENANT, + DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + false, + DeploymentContext.ClusterDistribution.KUBERNETES + ); + } + + private static ImagePullSecretConfig imagePullSecret() { + return ImagePullSecretConfig.builder() + .create(true) + .proxyUrl("proxy.example.org") + .url("registry.example.org") + .proxyUsername("proxy-user") + .readOnlyUsername("read-only-user") + .username("registry-user") + .proxyPassword("proxy-password") + .readOnlyPassword("read-only-password") + .password("registry-password") + .build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorTest.java b/src/test/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorTest.java new file mode 100644 index 000000000..5006bf6d5 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorTest.java @@ -0,0 +1,287 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.CommandExecutorForTest; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +@EnableKubernetesMockClient(crud = true) +class ExternalSecretsOperatorTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + private final Config config = Config.fromMap(Map.of( + "application", Map.of("namePrefix", "foo-"), + "registry", Map.of(), + "features", Map.of( + "secrets", Map.of("active", true) + ) + )); + + private final CommandExecutorForTest commandExecutor = new CommandExecutorForTest(); + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + private Path temporaryYamlFile; + private File clusterResourcesRepoDir; + private RepositoryWorkspace repositoryWorkspace; + private DeploymentContext deploymentContext; + + private final ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + + @Mock + private Deployer deployer; + @Mock + private AirGappedUtils airGappedUtils; + @Mock + private GitHandler gitHandler; + @Mock + private GitProvider gitProvider; + @Mock + private ImagePullSecretCreator imagePullSecretCreator; + + KubernetesClient client; + + @Test + void isDisabledViaActiveFlag() throws GitAPIException { + config.getFeatures().getSecrets().setActive(false); + + assertFalse(createExternalSecretsOperator().isEnabled(new ContextBuilder(config).build())); + } + + @Test + void helmReleaseIsInstalled() throws GitAPIException, IOException { + install(createExternalSecretsOperator()); + + verify(deployer).deployFeature( + "https://charts.external-secrets.io", + "external-secrets", + "external-secrets", + "0.9.16", + "foo-secrets", + "external-secrets", + temporaryYamlFile, + RepoType.HELM, + false, + deploymentContext, + repositoryWorkspace + ); + + assertThat(parseActualYaml()).doesNotContainKeys("resources"); + assertThat(parseActualYaml()).doesNotContainKey("imagePullSecrets"); + assertThat(parseActualYaml()).doesNotContainKey("certController"); + assertThat(parseActualYaml()).doesNotContainKey("webhook"); + + assertThat(parseActualYaml().get("installCRDs")).isNull(); + } + + @Test + void preparesExternalSecretsAppContentInClusterResourcesWorkspaceWithoutCopyingTemplates() throws GitAPIException { + install(createExternalSecretsOperator()); + + assertThat(new File(clusterResourcesRepoDir, "apps/external-secrets")).exists(); + assertThat(new File(clusterResourcesRepoDir, "apps/external-secrets/templates")).doesNotExist(); + } + + @Test + void skipsCrds() throws GitAPIException, IOException { + config.getApplication().setSkipCrds(true); + + install(createExternalSecretsOperator()); + + assertThat(parseActualYaml().get("installCRDs")).isEqualTo(false); + } + + @Test + void helmReleaseIsInstalledWithCustomImages() throws GitAPIException, IOException { + Config.SecretsSchema.ESOSchema.ESOHelmSchema helm = new Config.SecretsSchema.ESOSchema.ESOHelmSchema(); + helm.setImage("localhost:5000/external-secrets/external-secrets:v0.6.1"); + helm.setCertControllerImage("localhost:5000/external-secrets/external-secrets-certcontroller:v0.6.1"); + helm.setWebhookImage("localhost:5000/external-secrets/external-secrets-webhook:v0.6.1"); + config.getFeatures().getSecrets().getExternalSecrets().setHelm(helm); + + install(createExternalSecretsOperator()); + + Map valuesYaml = parseActualYaml(); + Map image = (Map) valuesYaml.get("image"); + assertThat(image.get("repository")).isEqualTo("localhost:5000/external-secrets/external-secrets"); + assertThat(image.get("tag")).isEqualTo("v0.6.1"); + + Map certController = (Map) valuesYaml.get("certController"); + Map certControllerImage = (Map) certController.get("image"); + assertThat(certControllerImage.get("repository")) + .isEqualTo("localhost:5000/external-secrets/external-secrets-certcontroller"); + assertThat(certControllerImage.get("tag")).isEqualTo("v0.6.1"); + + Map webhook = (Map) valuesYaml.get("webhook"); + Map webhookImage = (Map) webhook.get("image"); + assertThat(webhookImage.get("repository")) + .isEqualTo("localhost:5000/external-secrets/external-secrets-webhook"); + assertThat(webhookImage.get("tag")).isEqualTo("v0.6.1"); + } + + @Test + void setsPodResourceLimitsAndRequests() throws GitAPIException, IOException { + config.getApplication().setPodResources(true); + + install(createExternalSecretsOperator()); + + assertThat((Map) parseActualYaml().get("resources")).containsKeys("limits", "requests"); + assertThat((Map) ((Map) parseActualYaml().get("webhook")).get("resources")) + .containsKeys("limits", "requests"); + assertThat((Map) ((Map) parseActualYaml().get("certController")).get("resources")) + .containsKeys("limits", "requests"); + } + + @Test + void helmReleaseIsInstalledInAirGappedMode() throws GitAPIException, IOException { + when(gitHandler.getResourcesScm()).thenReturn(gitProvider); + when(gitProvider.repoUrl(any())).thenReturn("http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b"); + when(airGappedUtils.mirrorHelmRepoToGit(any(HelmChartConfig.class))).thenReturn("a/b"); + + config.getApplication().setMirrorRepos(true); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path sourceChart = rootChartsFolder.resolve("external-secrets"); + Files.createDirectories(sourceChart); + + Map chartYaml = Map.of("version", "1.2.3"); + fileSystemUtils.writeYaml(chartYaml, sourceChart.resolve("Chart.yaml").toFile()); + + install(createExternalSecretsOperator()); + + ArgumentCaptor helmConfig = ArgumentCaptor.forClass(HelmChartConfig.class); + verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()); + assertThat(helmConfig.getValue().chart()).isEqualTo("external-secrets"); + assertThat(helmConfig.getValue().repoURL()).isEqualTo("https://charts.external-secrets.io"); + assertThat(helmConfig.getValue().version()).isEqualTo("0.9.16"); + + verify(deployer).deployFeature( + eq("http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b"), + eq("external-secrets"), + eq("."), + eq("1.2.3"), + eq("foo-secrets"), + eq("external-secrets"), + eq(temporaryYamlFile), + eq(RepoType.GIT), + eq(false), + eq(deploymentContext), + eq(repositoryWorkspace) + ); + } + + @Test + void deploysImagePullSecretsForProxyRegistry() throws GitAPIException, IOException { + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-pw"); + + Config.SecretsSchema.ESOSchema.ESOHelmSchema helm = new Config.SecretsSchema.ESOSchema.ESOHelmSchema(); + helm.setCertControllerImage("some:thing"); + helm.setWebhookImage("some:thing"); + config.getFeatures().getSecrets().getExternalSecrets().setHelm(helm); + + install(createExternalSecretsOperator()); + + List> expectedImagePullSecrets = List.of(Map.of("name", "proxy-registry")); + assertThat(parseActualYaml().get("imagePullSecrets")).isEqualTo(expectedImagePullSecrets); + assertThat(((Map) parseActualYaml().get("certController")).get("imagePullSecrets")) + .isEqualTo(expectedImagePullSecrets); + assertThat(((Map) parseActualYaml().get("webhook")).get("imagePullSecrets")) + .isEqualTo(expectedImagePullSecrets); + } + + private ExternalSecretsOperator createExternalSecretsOperator() throws GitAPIException { + FileSystemUtils fileSystemUtils = new FileSystemUtils() { + @Override + public Path writeTempFile(Map mergeMap) { + Path ret = super.writeTempFile(mergeMap); + temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")); + // Path after template invocation + return ret; + } + }; + + TestGitRepoFactory repoFactory = new TestGitRepoFactory(config, new FileSystemUtils()) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, gitProvider); + clusterResourcesRepoDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoFactory.create( + "argocd/cluster-resources", + scmManagerMock + ); + + repositoryWorkspace = spy(new RepositoryWorkspace(clusterResourcesRepo)); + doNothing().when(repositoryWorkspace).commitAndPushClusterResourcesChanges(anyString()); + + return new ExternalSecretsOperator( + fileSystemUtils, + deployer, + airGappedUtils, + gitHandler, + imagePullSecretCreator, + new ExternalSecretsOperatorToolConfigMapper(config) + ); + } + + private boolean install(ExternalSecretsOperator operator) { + deploymentContext = new ContextBuilder(config).build(); + return operator.execute(deploymentContext, repositoryWorkspace); + } + + private Map parseActualYaml() throws IOException { + return YAML_MAPPER.readValue(temporaryYamlFile.toFile(), YAML_MAP_TYPE); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfigMapperTest.java new file mode 100644 index 000000000..e67c696a9 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/ExternalSecretsOperatorToolConfigMapperTest.java @@ -0,0 +1,110 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class ExternalSecretsOperatorToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getApplication().setPodResources(true); + config.getApplication().setSkipCrds(true); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setPassword("registry-password"); + config.getFeatures().getSecrets().setActive(true); + config.getFeatures().getSecrets().setNamespace("external-secrets"); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setRepoURL("https://eso.example.org"); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setChart("eso-chart"); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setVersion("2.3.4"); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setValues(Map.of("replicas", 3)); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setImage("eso-image"); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setCertControllerImage("cert-controller-image"); + config.getFeatures().getSecrets().getExternalSecrets().getHelm().setWebhookImage("webhook-image"); + + ExternalSecretsOperatorToolConfig actual = new ExternalSecretsOperatorToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(ExternalSecretsOperatorToolConfig.builder() + .active(true) + .namespace("test-external-secrets") + .helm(HelmChartConfig.builder() + .repoURL( + "https://eso.example.org") + .chart("eso-chart") + .version("2.3.4") + .values(Map.of( + "replicas", + 3 + )) + .localHelmChartFolder( + "/charts") + .build()) + .imagePullSecret(imagePullSecret()) + .templateConfig(Map.of( + "application", + Map.of( + "podResources", + true, + "skipCrds", + true + ), + "features", + Map.of( + "secrets", Map.of( + "externalSecrets", Map.of( + "helm", Map.of( + "image", + "eso-image", + "certControllerImage", + "cert-controller-image", + "webhookImage", + "webhook-image" + ) + ) + ) + ), + "registry", + Map.of("createImagePullSecrets", true) + )) + .build()); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.SINGLE_TENANT, + DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + false, + DeploymentContext.ClusterDistribution.KUBERNETES + ); + } + + private static ImagePullSecretConfig imagePullSecret() { + return ImagePullSecretConfig.builder() + .create(true) + .proxyUrl("proxy.example.org") + .url("registry.example.org") + .proxyUsername("proxy-user") + .readOnlyUsername("read-only-user") + .username("registry-user") + .proxyPassword("proxy-password") + .readOnlyPassword("read-only-password") + .password("registry-password") + .build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/IngressTest.java b/src/test/java/com/cloudogu/gitops/tools/IngressTest.java new file mode 100644 index 000000000..590564cd6 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/IngressTest.java @@ -0,0 +1,307 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +@EnableKubernetesMockClient(crud = true) +class IngressTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + // setting default config values with ingress active + private final Config config = new Config(); + + private Path temporaryYamlFile; + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + private File clusterResourcesRepoDir; + private RepositoryWorkspace repositoryWorkspace; + private DeploymentContext deploymentContext; + + private final ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + + @Mock + private Deployer deployer; + @Mock + private AirGappedUtils airGappedUtils; + @Mock + private GitHandler gitHandler; + @Mock + private GitProvider gitProvider; + @Mock + private ImagePullSecretCreator imagePullSecretCreator; + + KubernetesClient client; + + IngressTest() { + config.getApplication().setNamePrefix("foo-"); + config.getFeatures().getIngress().setActive(true); + } + + @Test + void helmReleaseIsInstalled() throws GitAPIException, IOException { + install(createIngress()); + + /* Assert one default value */ + Map actual = parseActualYaml(); + Map deployment = (Map) actual.get("deployment"); + assertThat(deployment.get("replicaCount")).isEqualTo(2); + + verify(deployer).deployFeature( + config.getFeatures().getIngress().getHelm().getRepoURL(), + "traefik", + config.getFeatures().getIngress().getHelm().getChart(), + config.getFeatures().getIngress().getHelm().getVersion(), + "foo-" + config.getFeatures().getIngress().getIngressNamespace(), + "traefik", + temporaryYamlFile, + RepoType.HELM, + false, + deploymentContext, + repositoryWorkspace + ); + + Map actualDeployment = (Map) parseActualYaml().get("deployment"); + assertThat(actualDeployment.get("metrics")).isNull(); + assertThat(actualDeployment.get("networkPolicy")).isNull(); + assertThat(parseActualYaml()).doesNotContainKey("imagePullSecrets"); + } + + @Test + void preparesTraefikAppContentInClusterResourcesWorkspaceWithoutCopyingTemplates() throws GitAPIException { + install(createIngress()); + + assertThat(new File(clusterResourcesRepoDir, "apps/traefik")).exists(); + assertThat(new File(clusterResourcesRepoDir, "apps/traefik/templates")).doesNotExist(); + } + + @Test + void setsPodResourceLimitsAndRequests() throws GitAPIException, IOException { + config.getApplication().setPodResources(true); + + install(createIngress()); + + Map deployment = (Map) parseActualYaml().get("deployment"); + assertThat((Map) deployment.get("resources")).containsKeys("limits", "requests"); + } + + @Test + void whenIngressIsNotEnabledIngressHelmValuesYamlHasNoContent() throws GitAPIException { + config.getFeatures().getIngress().setActive(false); + + assertFalse(createIngress().isEnabled(new ContextBuilder(config).build())); + } + + @Test + void additionalHelmValuesMergedWithDefaultValues() throws GitAPIException, IOException { + Map controllerValues = new LinkedHashMap<>(); + controllerValues.put("replicaCount", 42); + controllerValues.put("span", "7,5"); + Map values = new LinkedHashMap<>(); + values.put("controller", controllerValues); + config.getFeatures().getIngress().getHelm().setValues(values); + + install(createIngress()); + Map actual = parseActualYaml(); + Map controller = (Map) actual.get("controller"); + + assertThat(controller.get("replicaCount")).isEqualTo(42); + assertThat(controller.get("span")).isEqualTo("7,5"); + } + + @Test + void helmReleaseIsInstalledInAirGappedMode() throws GitAPIException, IOException { + when(gitHandler.getResourcesScm()).thenReturn(gitProvider); + when(gitProvider.repoUrl(any())).thenReturn("http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b"); + when(airGappedUtils.mirrorHelmRepoToGit(any(HelmChartConfig.class))).thenReturn("a/b"); + + config.getApplication().setMirrorRepos(true); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path sourceChart = rootChartsFolder.resolve("traefik"); + Files.createDirectories(sourceChart); + + Map chartYaml = Map.of("version", "1.2.3"); + fileSystemUtils.writeYaml(chartYaml, sourceChart.resolve("Chart.yaml").toFile()); + + install(createIngress()); + + ArgumentCaptor helmConfig = ArgumentCaptor.forClass(HelmChartConfig.class); + verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()); + assertThat(helmConfig.getValue().chart()).isEqualTo("traefik"); + assertThat(helmConfig.getValue().repoURL()).isEqualTo("https://traefik.github.io/charts"); + assertThat(helmConfig.getValue().version()).isEqualTo("39.0.9"); + + verify(deployer).deployFeature( + "http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b", + "traefik", + ".", + "1.2.3", + "foo-" + config.getFeatures().getIngress().getIngressNamespace(), + "traefik", + temporaryYamlFile, + RepoType.GIT, + false, + deploymentContext, + repositoryWorkspace + ); + } + + @Test + void whenMonitoringIsEnabledMetricsAreEnabled() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().setActive(true); + config.getApplication().setNamePrefix("heliosphere"); + + install(createIngress()); + + Map actual = parseActualYaml(); + Map metrics = (Map) actual.get("metrics"); + Map prometheus = (Map) metrics.get("prometheus"); + Map serviceMonitor = (Map) prometheus.get("serviceMonitor"); + + assertThat(metrics.get("enabled")).isEqualTo(true); + assertThat(serviceMonitor.get("enabled")).isEqualTo(true); + assertThat(serviceMonitor.get("namespace")).isEqualTo("heliospheremonitoring"); + } + + @Test + void activatesNetworkPolicies() throws GitAPIException, IOException { + config.getApplication().setNetpols(true); + + install(createIngress()); + + Map actual = parseActualYaml(); + Map deployment = (Map) actual.get("deployment"); + Map networkPolicy = (Map) deployment.get("networkPolicy"); + + assertThat(networkPolicy.get("enabled")).isEqualTo(true); + } + + @Test + void deploysImagePullSecretsForProxyRegistry() throws GitAPIException, IOException { + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-pw"); + + install(createIngress()); + + Map deployment = (Map) parseActualYaml().get("deployment"); + assertThat(deployment.get("imagePullSecrets")).isEqualTo(List.of(Map.of("name", "proxy-registry"))); + } + + @Test + void allowsOverridingTheImage() throws GitAPIException, IOException { + config.getFeatures().getIngress().getHelm().setImage("localhost/abc:v42"); + + install(createIngress()); + + Map yaml = parseActualYaml(); + Map image = (Map) yaml.get("image"); + assertThat(image.get("repository")).isEqualTo("localhost/abc"); + assertThat(image.get("tag")).isEqualTo("v42"); + assertThat(image.get("digest")).isNull(); + } + + @Test + void getNamespaceFromFeature() throws GitAPIException { + assertThat(createIngress().getActiveNamespaceFromFeature(new ContextBuilder(config).build())) + .isEqualTo("foo-" + config.getFeatures().getIngress().getIngressNamespace()); + + config.getFeatures().getIngress().setActive(false); + + assertThat(createIngress().getActiveNamespaceFromFeature(new ContextBuilder(config).build())).isEqualTo(null); + } + + private Ingress createIngress() throws GitAPIException { + // We use the real FileSystemUtils and not a mock to make sure file editing works as expected + FileSystemUtils testFileSystemUtils = new FileSystemUtils() { + @Override + public Path writeTempFile(Map mergeMap) { + Path ret = super.writeTempFile(mergeMap); + temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")); + // Path after template invocation + return ret; + } + }; + + TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, testFileSystemUtils) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, gitProvider); + clusterResourcesRepoDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoProvider.create("argocd/cluster-resources", scmManagerMock); + + repositoryWorkspace = spy(new RepositoryWorkspace(clusterResourcesRepo)); + doNothing().when(repositoryWorkspace).commitAndPushClusterResourcesChanges(anyString()); + + return new Ingress( + testFileSystemUtils, + deployer, + airGappedUtils, + gitHandler, + imagePullSecretCreator, + new IngressToolConfigMapper(config) + ); + } + + private boolean install(Ingress ingress) { + deploymentContext = new ContextBuilder(config).build(); + return ingress.execute(deploymentContext, repositoryWorkspace); + } + + private Map parseActualYaml() throws IOException { + return YAML_MAPPER.readValue(temporaryYamlFile.toFile(), YAML_MAP_TYPE); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/IngressToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/IngressToolConfigMapperTest.java new file mode 100644 index 000000000..9f56e139b --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/IngressToolConfigMapperTest.java @@ -0,0 +1,88 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class IngressToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getApplication().setNetpols(true); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setPassword("registry-password"); + config.getFeatures().getIngress().setActive(true); + config.getFeatures().getIngress().setIngressNamespace("gateway"); + config.getFeatures().getIngress().getHelm().setRepoURL("https://ingress.example.org"); + config.getFeatures().getIngress().getHelm().setChart("ingress-chart"); + config.getFeatures().getIngress().getHelm().setVersion("3.4.5"); + config.getFeatures().getIngress().getHelm().setValues(Map.of("replicas", 4)); + config.getFeatures().getIngress().getHelm().setImage("ingress-image"); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getMonitoring().setNamespace("observability"); + + IngressToolConfig actual = new IngressToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(IngressToolConfig.builder() + .active(true) + .namespace("test-gateway") + .helm(HelmChartConfig.builder() + .repoURL("https://ingress.example.org") + .chart("ingress-chart") + .version("3.4.5") + .values(Map.of("replicas", 4)) + .localHelmChartFolder("/charts") + .build()) + .imagePullSecret(imagePullSecret()) + .templateConfig(Map.of( + "application", Map.of("namePrefix", "test-", "netpols", true), + "features", Map.of( + "ingress", + Map.of("helm", Map.of("image", "ingress-image")), + "monitoring", + Map.of("active", true, "namespace", "observability") + ), + "registry", Map.of("createImagePullSecrets", true) + )) + .build()); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.SINGLE_TENANT, + DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + false, + DeploymentContext.ClusterDistribution.KUBERNETES + ); + } + + private static ImagePullSecretConfig imagePullSecret() { + return ImagePullSecretConfig.builder() + .create(true) + .proxyUrl("proxy.example.org") + .url("registry.example.org") + .proxyUsername("proxy-user") + .readOnlyUsername("read-only-user") + .username("registry-user") + .proxyPassword("proxy-password") + .readOnlyPassword("read-only-password") + .password("registry-password") + .build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/MonitoringTest.java b/src/test/java/com/cloudogu/gitops/tools/MonitoringTest.java new file mode 100644 index 000000000..a0366004e --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/MonitoringTest.java @@ -0,0 +1,1100 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.Tuple; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; + +import java.io.File; +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.argThat; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@EnableKubernetesMockClient(crud = true) +@SuppressWarnings("unchecked") +class MonitoringTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + + private final Config config = Config.fromMap(Map.of( + "registry", Map.of( + "internal", true, + "createImagePullSecrets", false + ), + "scm", Map.of( + "scmManager", Map.of("internal", true) + ), + "jenkins", Map.of( + "internal", true, + "active", true, + "metricsUsername", "metrics", + "metricsPassword", "metrics" + ), + "application", Map.ofEntries( + Map.entry("username", "abc"), + Map.entry("password", "123"), + Map.entry("openshift", false), + Map.entry("namePrefix", "foo-"), + Map.entry("mirrorRepos", false), + Map.entry("podResources", false), + Map.entry("skipCrds", false), + Map.entry("namespaceIsolation", false), + Map.entry("gitName", "Cloudogu"), + Map.entry("gitEmail", "hello@cloudogu.com"), + Map.entry("netpols", false), + Map.entry( + "namespaces", Map.of( + "dedicatedNamespaces", new LinkedHashSet<>(List.of( + "test1-default", + "test1-argocd", + "test1-monitoring", + "test1-secrets" + )), + "tenantNamespaces", new LinkedHashSet<>(List.of( + "test1-example-apps-staging", + "test1-example-apps-production" + )) + ) + ) + ), + "features", Map.of( + "argocd", Map.of("active", true), + "monitoring", Map.of( + "active", true, + "grafanaUrl", "", + "grafanaEmailFrom", "grafana@example.org", + "grafanaEmailTo", "infra@example.org", + "helm", Map.of( + "chart", "kube-prometheus-stack", + "repoURL", "https://prom", + "version", "19.2.2" + ) + ), + "secrets", Map.of("active", true), + "ingress", Map.of("active", true) + ) + )); + + private K8sClient k8sClient; + private final Deployer deployer = mock(Deployer.class); + private final AirGappedUtils airGappedUtils = mock(AirGappedUtils.class); + private Path temporaryYamlFilePrometheus; + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + private File clusterResourcesRepoDir; + + private final GitHandler gitHandler = mock(GitHandler.class); + private RepositoryWorkspace repositoryWorkspace; + private DeploymentContext deploymentContext; + private ScmManagerProviderMock scmManagerMock; + private final ImagePullSecretCreator imagePullSecretCreator = mock(ImagePullSecretCreator.class); + + KubernetesClient client; + KubernetesMockServer server; + + @BeforeEach + void setup() { + scmManagerMock = new ScmManagerProviderMock(); + k8sClient = mock(K8sClient.class); + k8sClient.setClient(client); + } + + @Test + void isDisabledViaActiveFlag() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(false); + assertFalse(createStack(scmManagerMock).isEnabled(new ContextBuilder(config).build())); + } + + @Test + void whenMailServerDisabledDoesNotIncludeMailConfigurationsIntoClusterResources() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(null); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + assertThat(grafana.get("notifiers")).isNull(); + } + + @Test + void whenMailServerEnabledIncludesMailConfigurationsIntoClusterResources() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + assertThat(grafana.get("notifiers")).isNotNull(); + } + + @Test + void whenEmailAddressesIsSet() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMonitoring().setGrafanaEmailFrom("grafana@example.com"); + config.getFeatures().getMonitoring().setGrafanaEmailTo("infra@example.com"); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map notifiers = (Map) grafana.get("notifiers"); + Map notifiersYaml = (Map) notifiers.get("notifiers.yaml"); + List> notifierList = (List>) notifiersYaml.get("notifiers"); + Map settings = (Map) notifierList.get(0).get("settings"); + + assertThat(settings.get("addresses")).isEqualTo("infra@example.com"); + + Map env = (Map) grafana.get("env"); + assertThat(env.get("GF_SMTP_FROM_ADDRESS")).isEqualTo("grafana@example.com"); + } + + @Test + void whenEmailAddressesIsNotSet() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map notifiers = (Map) grafana.get("notifiers"); + Map notifiersYaml = (Map) notifiers.get("notifiers.yaml"); + List> notifierList = (List>) notifiersYaml.get("notifiers"); + Map settings = (Map) notifierList.get(0).get("settings"); + + assertThat(settings.get("addresses")).isEqualTo("infra@example.org"); + + Map env = (Map) grafana.get("env"); + assertThat(env.get("GF_SMTP_FROM_ADDRESS")).isEqualTo("grafana@example.org"); + } + + @Test + void whenExternalMailserverIsSet() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpPort(1010110); + config.getFeatures().getMonitoring().setGrafanaEmailTo("grafana@example.com"); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map alerting = (Map) grafana.get("alerting"); + + Map expectedContactPoints = YAML_MAPPER.readValue( + """ + apiVersion: 1 + contactPoints: + - orgId: 1 + name: email + is_default: true + receivers: + - uid: email1 + type: email + settings: + addresses: grafana@example.com + """, YAML_MAP_TYPE + ); + assertThat(alerting.get("contactpoints.yaml")).isEqualTo(expectedContactPoints); + + Map expectedNotificationPolicies = YAML_MAPPER.readValue( + """ + apiVersion: 1 + policies: + - orgId: 1 + is_default: true + receiver: email + routes: + - receiver: email + group_by: ["grafana_folder", "alertname"] + """, YAML_MAP_TYPE + ); + assertThat(alerting.get("notification-policies.yaml")).isEqualTo(expectedNotificationPolicies); + + Map env = (Map) grafana.get("env"); + assertThat(env.get("GF_SMTP_HOST")).isEqualTo("smtp.example.com:1010110"); + } + + @Test + void whenExternalMailserverIsSetWithUser() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpUser("mailserver@example.com"); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map smtp = (Map) grafana.get("smtp"); + assertThat(smtp.get("existingSecret")).isEqualTo("grafana-email-secret"); + } + + @Test + void whenExternalMailserverUserContainsOnlyWhitespaceItIsStillTreatedAsConfigured() throws GitAPIException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpUser(" "); + + install(createStack(scmManagerMock)); + + verify(k8sClient).createSecret( + "generic", + "grafana-email-secret", + "foo-monitoring", + new Tuple<>("user", " "), + new Tuple<>("password", "") + ); + } + + @Test + void whenExternalMailserverIsSetWithPassword() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpPassword("1101ABCabc&/+*~"); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map smtp = (Map) grafana.get("smtp"); + assertThat(smtp.get("existingSecret")).isEqualTo("grafana-email-secret"); + } + + @Test + void whenExternalMailserverIsSetWithoutUserAndPassword() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + assertThat(grafana.get("valuesFrom")).isNull(); + assertThat(grafana.get("smtp")).isNull(); + } + + @Test + void checkIfKubernetesSecretWillBeCreatedWhenExternalEmailserversCredentialIsSet() throws GitAPIException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpUser("grafana@example.com"); + config.getFeatures().getMail().setSmtpPassword("1101ABCabc&/+*~"); + + install(createStack(scmManagerMock)); + } + + @Test + void resolvesExternalMailserverCredentialsFromSecretWithoutRenderingThem() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpUser("fallback-user"); + config.getFeatures().getMail().setSmtpPassword("fallback-password"); + config.getFeatures().getMail().setCredentials( + new Credentials(null, null, "smtp-credentials", "gop-job", "username", "password") + ); + when(k8sClient.getCredentialsFromSecret(argThat((Credentials credentials) -> + "smtp-credentials".equals(credentials.getSecretName()) + && "gop-job".equals(credentials.getSecretNamespace()) + ))).thenReturn(new Credentials("secret-smtp-user", "secret-smtp-password")); + + install(createStack(scmManagerMock)); + + verify(k8sClient).createSecret( + "generic", + "grafana-email-secret", + "foo-monitoring", + new Tuple<>("user", "secret-smtp-user"), + new Tuple<>("password", "secret-smtp-password") + ); + Map grafana = (Map) parseActualYaml().get("grafana"); + Map smtp = (Map) grafana.get("smtp"); + assertThat(smtp.get("existingSecret")).isEqualTo("grafana-email-secret"); + assertThat(Files.readString(temporaryYamlFilePrometheus)) + .doesNotContain("secret-smtp-user", "secret-smtp-password"); + assertThat(config.getFeatures().getMail().getSmtpUser()).isEqualTo("fallback-user"); + assertThat(config.getFeatures().getMail().getSmtpPassword()).isEqualTo("fallback-password"); + } + + @Test + void whenExternalMailserverIsSetWithoutPort() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map env = (Map) grafana.get("env"); + assertThat(env.get("GF_SMTP_HOST")).isEqualTo("smtp.example.com"); + } + + @Test + void whenExternalMailserverIsNotSet() throws GitAPIException, IOException { + config.getFeatures().getMail().setActive(null); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + assertThat(grafana.get("alerting")).isNull(); + } + + @Test + void configuresAdminUserIfRequested() throws GitAPIException, IOException { + config.getApplication().setUsername("my-user"); + config.getApplication().setPassword("hunter2"); + install(createStack(scmManagerMock)); + + verify(k8sClient).createSecret( + "generic", + "grafana-admin-credentials", + "foo-monitoring", + new Tuple<>("admin-user", "my-user"), + new Tuple<>("admin-password", "hunter2") + ); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map admin = (Map) grafana.get("admin"); + assertThat(admin.get("existingSecret")).isEqualTo("grafana-admin-credentials"); + assertThat(admin.get("userKey")).isEqualTo("admin-user"); + assertThat(admin.get("passwordKey")).isEqualTo("admin-password"); + assertThat(grafana).doesNotContainKeys("adminUser", "adminPassword"); + assertThat(Files.readString(temporaryYamlFilePrometheus)).doesNotContain("hunter2"); + } + + @Test + void resolvesApplicationCredentialsForGrafanaAdminSecretWithoutMutatingConfig() throws GitAPIException, IOException { + config.getApplication().setUsername("fallback-user"); + config.getApplication().setPassword("fallback-password"); + config.getApplication().setCredentials( + new Credentials(null, null, "application-credentials", "gop-job") + ); + when(k8sClient.getCredentialsFromSecret(argThat(credentials -> + "application-credentials".equals(credentials.getSecretName()) + && "gop-job".equals(credentials.getSecretNamespace()) + && "fallback-user".equals(credentials.getUsername()) + ))).thenReturn(new Credentials("secret-admin", "grafana-secret-password")); + + install(createStack(scmManagerMock)); + + verify(k8sClient).createSecret( + "generic", + "grafana-admin-credentials", + "foo-monitoring", + new Tuple<>("admin-user", "secret-admin"), + new Tuple<>("admin-password", "grafana-secret-password") + ); + assertThat(config.getApplication().getUsername()).isEqualTo("fallback-user"); + assertThat(config.getApplication().getPassword()).isEqualTo("fallback-password"); + assertThat(Files.readString(temporaryYamlFilePrometheus)).doesNotContain("grafana-secret-password"); + } + + @Test + void resolvesJenkinsMetricsCredentialsForPrometheus() throws GitAPIException, IOException { + config.getJenkins().setMetricsUsername("fallback-metrics-user"); + config.getJenkins().setMetricsPassword("fallback-metrics-password"); + config.getJenkins().setMetricsCredentials( + new Credentials(null, null, "jenkins-metrics-credentials", "gop-job") + ); + when(k8sClient.getCredentialsFromSecret(argThat(credentials -> + "jenkins-metrics-credentials".equals(credentials.getSecretName()) + && "gop-job".equals(credentials.getSecretNamespace()) + && "fallback-metrics-user".equals(credentials.getUsername()) + ))).thenReturn(new Credentials("secret-metrics-user", "secret-metrics-password")); + + install(createStack(scmManagerMock)); + + verify(k8sClient).createSecret( + "generic", + "prometheus-metrics-creds-jenkins", + "foo-monitoring", + new Tuple<>("password", "secret-metrics-password") + ); + Map prometheus = (Map) parseActualYaml().get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + List> additionalScrapeConfigs = + (List>) prometheusSpec.get("additionalScrapeConfigs"); + Map basicAuth = (Map) additionalScrapeConfigs.get(1).get("basic_auth"); + assertThat(basicAuth.get("username")).isEqualTo("secret-metrics-user"); + assertThat(config.getJenkins().getMetricsPassword()).isEqualTo("fallback-metrics-password"); + assertThat(Files.readString(temporaryYamlFilePrometheus)).doesNotContain("secret-metrics-password"); + } + + @Test + void usesRuntimeScmCredentialsForPrometheusSecret() throws GitAPIException { + scmManagerMock.setCredentials(new Credentials("scm-admin", "scm-runtime-password")); + + install(createStack(scmManagerMock)); + + verify(k8sClient).createSecret( + "generic", + "prometheus-metrics-creds-scmm", + "foo-monitoring", + new Tuple<>("password", "scm-runtime-password") + ); + } + + @Test + void configuresGrafanaOidcFromStructuredConfig() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().setGrafanaUrl("http://grafana.localhost"); + + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setClientId("grafana"); + oidc.setClientSecret("grafana-secret"); + oidc.setAdminGroupName("gop-admins"); + config.getFeatures().getMonitoring().setOidc(oidc); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map grafanaIni = (Map) grafana.get("grafana.ini"); + Map oauth = (Map) grafanaIni.get("auth.generic_oauth"); + + assertThat(oauth.get("enabled")).isEqualTo(true); + assertThat(oauth.get("client_id")).isEqualTo("grafana"); + assertThat(oauth.get("auth_url")).isEqualTo("http://keycloak.local.gd/realms/gop/protocol/openid-connect/auth"); + assertThat(oauth.get("role_attribute_path")).isEqualTo("contains(groups[*], 'gop-admins') && 'Admin' || 'None'"); + assertThat(oauth.get("role_attribute_strict")).isEqualTo(true); + } + + @Test + void doesNotConfigureGrafanaOidcWhenOidcConfigIsNull() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().setOidc(null); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map grafanaIni = (Map) grafana.get("grafana.ini"); + assertThat(grafanaIni).doesNotContainKey("auth.generic_oauth"); + } + + @Test + void usesDefaultGrafanaOidcScopesWhenScopesAreNull() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().setGrafanaUrl("http://grafana.localhost"); + + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setClientId("grafana"); + oidc.setClientSecret("grafana-secret"); + oidc.setScopes(null); + config.getFeatures().getMonitoring().setOidc(oidc); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map grafanaIni = (Map) grafana.get("grafana.ini"); + Map oauth = (Map) grafanaIni.get("auth.generic_oauth"); + assertThat(oauth.get("scopes")).isEqualTo("openid profile email"); + } + + @Test + void usesIngressIfEnabled() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().setGrafanaUrl("http://grafana.local"); + + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map serviceYaml = (Map) grafana.get("ingress"); + assertThat(serviceYaml.get("enabled")).isEqualTo(true); + assertThat(((List) serviceYaml.get("hosts")).get(0)).isEqualTo("grafana.local"); + } + + @Test + void doesNotUseIngressByDefault() throws GitAPIException, IOException { + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + assertThat(grafana).doesNotContainKey("ingress"); + } + + @Test + void preparesMonitoringAppContentInClusterResourcesWorkspaceWithoutCopyingTemplates() throws GitAPIException { + install(createStack(scmManagerMock)); + + assertThat(new File(clusterResourcesRepoDir, "apps/monitoring")).exists(); + assertThat(new File(clusterResourcesRepoDir, "apps/monitoring/templates")).doesNotExist(); + assertThat(new File(clusterResourcesRepoDir, "apps/monitoring/misc/dashboard")).exists(); + } + + @Test + void cleanupUnusedDashboardsRemovesAllDashboardsForDisabledFeatures() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getIngress().setActive(false); + config.getJenkins().setActive(false); + scmManagerMock.setPrometheus(null); + + install(createStack(scmManagerMock)); + + File dashboardDir = new File(clusterResourcesRepoDir, "apps/monitoring/misc/dashboard"); + + assertThat(new File(dashboardDir, "traefik-dashboard.yaml")).doesNotExist(); + assertThat(new File(dashboardDir, "traefik-dashboard-requests-handling.yaml")).doesNotExist(); + assertThat(new File(dashboardDir, "jenkins-dashboard.yaml")).doesNotExist(); + assertThat(new File(dashboardDir, "scmm-dashboard.yaml")).doesNotExist(); + } + + @Test + void cleanupUnusedDashboardsKeepsScmmDashboardWhenInternalScmMetricsEndpointExists() throws GitAPIException, URISyntaxException { + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getIngress().setActive(false); + config.getJenkins().setActive(false); + config.getScm().getScmManager().setUrl(null); + scmManagerMock.setPrometheus(new URI("http://localhost:8080/scm/api/v2/metrics/prometheus")); + + install(createStack(scmManagerMock)); + + File dashboardDir = new File(clusterResourcesRepoDir, "apps/monitoring/misc/dashboard"); + + assertThat(new File(dashboardDir, "traefik-dashboard.yaml")).doesNotExist(); + assertThat(new File(dashboardDir, "traefik-dashboard-requests-handling.yaml")).doesNotExist(); + assertThat(new File(dashboardDir, "jenkins-dashboard.yaml")).doesNotExist(); + assertThat(new File(dashboardDir, "scmm-dashboard.yaml")).exists(); + } + + @Test + void appliesPrometheusServiceMonitorCrdFromFileBeforeInstallingAirGappedMode() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().setActive(true); + config.getApplication().setMirrorRepos(true); + config.getApplication().setSkipCrds(false); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path crdFile = rootChartsFolder.resolve( + config.getFeatures().getMonitoring().getHelm().getChart() + "/charts/crds/crds/crd-servicemonitors.yaml" + ); + Files.createDirectories(crdFile.getParent()); + Files.writeString(crdFile, "dummy"); + + Path chartYaml = rootChartsFolder.resolve(config.getFeatures().getMonitoring().getHelm().getChart() + "/Chart.yaml"); + Files.createDirectories(chartYaml.getParent()); + Files.writeString(chartYaml, "apiVersion: v2\nname: kube-prometheus-stack\nversion: 42.0.3\n"); + + install(createStack(scmManagerMock)); + } + + @Test + void appliesPrometheusServiceMonitorCrdFromGithubBeforeInstalling() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(true); + config.getApplication().setMirrorRepos(false); + config.getApplication().setSkipCrds(false); + + install(createStack(scmManagerMock)); + } + + @Test + void doesNotApplyServiceMonitorCrdWhenMonitoringIsDisabled() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(false); + config.getApplication().setSkipCrds(false); + config.getApplication().setMirrorRepos(false); + + install(createStack(scmManagerMock)); + } + + @Test + void usesRemoteScmmUrlIfRequested() throws GitAPIException, IOException { + install(createStack(scmManagerMock)); + + Map prometheus = (Map) parseActualYaml().get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + List> additionalScrapeConfigs = + (List>) prometheusSpec.get("additionalScrapeConfigs"); + + List> staticConfigs0 = (List>) additionalScrapeConfigs.get(0).get( + "static_configs"); + List targets0 = (List) staticConfigs0.get(0).get("targets"); + assertThat(targets0.get(0)).isEqualTo("localhost:8080"); + assertThat(additionalScrapeConfigs.get(0).get("metrics_path")).isEqualTo("/scm/api/v2/metrics/prometheus"); + assertThat(additionalScrapeConfigs.get(0).get("scheme")).isEqualTo("http"); + + List> staticConfigs1 = (List>) additionalScrapeConfigs.get(1).get( + "static_configs"); + List targets1 = (List) staticConfigs1.get(0).get("targets"); + assertThat(targets1.get(0)).isEqualTo("jenkins.foo-jenkins.svc.cluster.local"); + assertThat(additionalScrapeConfigs.get(1).get("scheme")).isEqualTo("http"); + assertThat(additionalScrapeConfigs.get(1).get("metrics_path")).isEqualTo("/prometheus"); + } + + @Test + void usesRemoteJenkinsUrlIfRequested() throws GitAPIException, IOException { + config.getJenkins().setInternal(false); + config.getJenkins().setUrl("https://localhost:9090/jenkins"); + install(createStack(scmManagerMock)); + + Map prometheus = (Map) parseActualYaml().get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + List> additionalScrapeConfigs = + (List>) prometheusSpec.get("additionalScrapeConfigs"); + + List> staticConfigs0 = (List>) additionalScrapeConfigs.get(0).get( + "static_configs"); + List targets0 = (List) staticConfigs0.get(0).get("targets"); + assertThat(targets0.get(0)).isEqualTo("localhost:8080"); + assertThat(additionalScrapeConfigs.get(0).get("scheme")).isEqualTo("http"); + assertThat(additionalScrapeConfigs.get(0).get("metrics_path")).isEqualTo("/scm/api/v2/metrics/prometheus"); + + List> staticConfigs1 = (List>) additionalScrapeConfigs.get(1).get( + "static_configs"); + List targets1 = (List) staticConfigs1.get(0).get("targets"); + assertThat(targets1.get(0)).isEqualTo("localhost:9090"); + assertThat(additionalScrapeConfigs.get(1).get("metrics_path")).isEqualTo("/jenkins/prometheus"); + assertThat(additionalScrapeConfigs.get(1).get("scheme")).isEqualTo("https"); + } + + @Test + void configuresCustomMetricsUserForJenkins() throws GitAPIException, IOException { + config.getJenkins().setMetricsUsername("external-metrics-username"); + config.getJenkins().setMetricsPassword("hunter2"); + install(createStack(scmManagerMock)); + + Map prometheus = (Map) parseActualYaml().get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + List> additionalScrapeConfigs = + (List>) prometheusSpec.get("additionalScrapeConfigs"); + Map basicAuth = (Map) additionalScrapeConfigs.get(1).get("basic_auth"); + assertThat(basicAuth.get("username")).isEqualTo("external-metrics-username"); + } + + @Test + void configuresCustomImageForGrafana() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().getHelm().setGrafanaImage("localhost:5000/grafana/grafana:the-tag"); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map image = (Map) grafana.get("image"); + assertThat(image.get("registry")).isEqualTo("localhost:5000"); + assertThat(image.get("repository")).isEqualTo("grafana/grafana"); + assertThat(image.get("tag")).isEqualTo("the-tag"); + } + + @Test + void configuresCustomImageForGrafanaSidecar() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().getHelm().setGrafanaSidecarImage("localhost:5000/grafana/sidecar:the-tag"); + install(createStack(scmManagerMock)); + + Map grafana = (Map) parseActualYaml().get("grafana"); + Map sidecar = (Map) grafana.get("sidecar"); + Map image = (Map) sidecar.get("image"); + assertThat(image.get("registry")).isEqualTo("localhost:5000"); + assertThat(image.get("repository")).isEqualTo("grafana/sidecar"); + assertThat(image.get("tag")).isEqualTo("the-tag"); + } + + @Test + void configuresCustomImageForPrometheusAndOperator() throws GitAPIException, IOException { + config.getFeatures().getMonitoring().getHelm().setPrometheusImage("localhost:5000/prometheus/prometheus:v1"); + config.getFeatures().getMonitoring().getHelm().setPrometheusOperatorImage( + "localhost:5000/prometheus-operator/prometheus-operator:v2" + ); + config.getFeatures().getMonitoring().getHelm().setPrometheusConfigReloaderImage( + "localhost:5000/prometheus-operator/prometheus-config-reloader:v3" + ); + + install(createStack(scmManagerMock)); + + Map actualYaml = parseActualYaml(); + Map prometheus = (Map) actualYaml.get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + Map prometheusImage = (Map) prometheusSpec.get("image"); + assertThat(prometheusImage.get("registry")).isEqualTo("localhost:5000"); + assertThat(prometheusImage.get("repository")).isEqualTo("prometheus/prometheus"); + assertThat(prometheusImage.get("tag")).isEqualTo("v1"); + + Map prometheusOperator = (Map) actualYaml.get("prometheusOperator"); + Map operatorImage = (Map) prometheusOperator.get("image"); + assertThat(operatorImage.get("registry")).isEqualTo("localhost:5000"); + assertThat(operatorImage.get("repository")).isEqualTo("prometheus-operator/prometheus-operator"); + assertThat(operatorImage.get("tag")).isEqualTo("v2"); + + Map configReloader = (Map) prometheusOperator.get("prometheusConfigReloader"); + Map reloaderImage = (Map) configReloader.get("image"); + assertThat(reloaderImage.get("registry")).isEqualTo("localhost:5000"); + assertThat(reloaderImage.get("repository")).isEqualTo("prometheus-operator/prometheus-config-reloader"); + assertThat(reloaderImage.get("tag")).isEqualTo("v3"); + } + + @Test + void deploysImagePullSecretsForProxyRegistry() throws GitAPIException, IOException { + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-pw"); + + install(createStack(scmManagerMock)); + + Map global = (Map) parseActualYaml().get("global"); + assertThat(global.get("imagePullSecrets")).isEqualTo(List.of(Map.of("name", "proxy-registry"))); + } + + @Test + void helmReleaseIsInstalled() throws GitAPIException, IOException { + install(createStack(scmManagerMock)); + + verify(deployer).deployFeature( + "https://prom", + "monitoring", + "kube-prometheus-stack", + "19.2.2", + "foo-monitoring", + "kube-prometheus-stack", + temporaryYamlFilePrometheus, + RepoType.HELM, + false, + deploymentContext, + repositoryWorkspace + ); + + Map yaml = parseActualYaml(); + Map grafana = (Map) yaml.get("grafana"); + Map admin = (Map) grafana.get("admin"); + assertThat(admin.get("existingSecret")).isEqualTo("grafana-admin-credentials"); + assertThat(grafana).doesNotContainKeys("adminUser", "adminPassword"); + + Map prometheusOperator = (Map) yaml.get("prometheusOperator"); + Map sidecar = (Map) grafana.get("sidecar"); + Map prometheus = (Map) yaml.get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + + assertThat(prometheusOperator).doesNotContainKey("resources"); + assertThat(grafana).doesNotContainKey("resources"); + assertThat(sidecar).doesNotContainKey("resources"); + assertThat(prometheusSpec).doesNotContainKey("resources"); + + assertThat(prometheusOperator.get("securityContext")).isNull(); + assertThat(grafana.get("securityContext")).isNull(); + assertThat(prometheusSpec.get("securityContext")).isNull(); + + assertThat(yaml.get("kubeApiServer")).isNull(); + + Map admissionWebhooks = (Map) prometheusOperator.get("admissionWebhooks"); + assertThat(admissionWebhooks.get("enabled")).isEqualTo(false); + Map tls = (Map) prometheusOperator.get("tls"); + assertThat(tls.get("enabled")).isEqualTo(false); + assertThat(prometheusOperator.get("kubeletService")).isNull(); + assertThat(prometheusOperator.get("namespaces")).isNull(); + assertThat(yaml).doesNotContainKey("global"); + + assertThat(grafana.get("rbac")).isNull(); + Map dashboards = (Map) sidecar.get("dashboards"); + assertThat(dashboards.get("searchNamespace")).isEqualTo("ALL"); + + assertThat(yaml.get("crds")).isNull(); + assertThat(new File(clusterResourcesRepoDir, "apps/monitoring/misc/rbac")).doesNotExist(); + } + + @Test + void publishesMonitoringResourcesThroughRepositoryWorkspace() throws GitAPIException { + install(createStack(scmManagerMock)); + + verify(repositoryWorkspace).commitAndPushClusterResourcesChanges("Update monitoring GitOps resources"); + } + + @Test + void skipsCrds() throws GitAPIException, IOException { + config.getApplication().setSkipCrds(true); + + install(createStack(scmManagerMock)); + + Map crds = (Map) parseActualYaml().get("crds"); + assertThat(crds.get("enabled")).isEqualTo(false); + } + + @Test + void setsPodResourceLimitsAndRequests() throws GitAPIException, IOException { + config.getApplication().setPodResources(true); + + install(createStack(scmManagerMock)); + + Map yaml = parseActualYaml(); + Map prometheusOperator = (Map) yaml.get("prometheusOperator"); + Map configReloader = (Map) prometheusOperator.get("prometheusConfigReloader"); + Map grafana = (Map) yaml.get("grafana"); + Map sidecar = (Map) grafana.get("sidecar"); + Map prometheus = (Map) yaml.get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + + assertThat((Map) prometheusOperator.get("resources")).containsKeys("limits", "requests"); + assertThat((Map) configReloader.get("resources")).containsKeys("limits", "requests"); + assertThat((Map) grafana.get("resources")).containsKeys("limits", "requests"); + assertThat((Map) sidecar.get("resources")).containsKeys("limits", "requests"); + assertThat((Map) prometheusSpec.get("resources")).containsKeys("limits", "requests"); + } + + @Test + void worksWithOpenshift() throws GitAPIException, IOException { + config.getApplication().setOpenshift(true); + when(k8sClient.getAnnotation("namespace", "foo-monitoring", "openshift.io/sa.scc.uid-range")) + .thenReturn("1000920000/10000"); + install(createStack(scmManagerMock)); + + Map yaml = parseActualYaml(); + Map prometheusOperator = (Map) yaml.get("prometheusOperator"); + Map operatorSecurityContext = (Map) prometheusOperator.get("securityContext"); + assertThat(operatorSecurityContext).isNotNull(); + assertThat(operatorSecurityContext.get("fsGroup")).isNull(); + assertThat(operatorSecurityContext.get("runAsGroup")).isNull(); + assertThat(operatorSecurityContext.get("runAsUser")).isNull(); + + Map grafana = (Map) yaml.get("grafana"); + Map grafanaSecurityContext = (Map) grafana.get("securityContext"); + assertThat(grafanaSecurityContext).isNotNull(); + assertThat(grafanaSecurityContext.get("fsGroup")).isEqualTo(1000920000); + assertThat(grafanaSecurityContext.get("runAsGroup")).isEqualTo(1000920000); + assertThat(grafanaSecurityContext.get("runAsUser")).isEqualTo(1000920000); + + Map prometheus = (Map) yaml.get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + Map prometheusSecurityContext = (Map) prometheusSpec.get("securityContext"); + assertThat(prometheusSecurityContext).isNotNull(); + assertThat(prometheusSecurityContext.get("fsGroup")).isNull(); + assertThat(prometheusSpec.get("runAsGroup")).isNull(); + assertThat(prometheusSpec.get("runAsUser")).isNull(); + } + + @Test + void worksWithNamespaceIsolation() throws GitAPIException, IOException { + config.getApplication().setNamespaceIsolation(true); + + Monitoring prometheusStack = createStack(scmManagerMock); + install(prometheusStack); + + Map yaml = parseActualYaml(); + Map global = (Map) yaml.get("global"); + Map globalRbac = (Map) global.get("rbac"); + assertThat(globalRbac.get("create")).isEqualTo(false); + + for (String namespace : config.getApplication().getNamespaces().getActiveNamespaces()) { + File rbacYaml = new File( + clusterResourcesRepoDir, + "apps/monitoring/misc/rbac/" + namespace + ".yaml" + ); + String rbacText = Files.readString(rbacYaml.toPath()); + assertThat(rbacText).contains("namespace: " + namespace); + assertThat(rbacText).contains(" namespace: foo-monitoring"); + } + + Map kubeApiServer = (Map) yaml.get("kubeApiServer"); + assertThat(kubeApiServer.get("enabled")).isEqualTo(false); + + Map prometheusOperator = (Map) yaml.get("prometheusOperator"); + Map kubeletService = (Map) prometheusOperator.get("kubeletService"); + assertThat(kubeletService.get("enabled")).isEqualTo(false); + + Map namespaces = (Map) prometheusOperator.get("namespaces"); + assertThat(namespaces.get("releaseNamespace")).isEqualTo(false); + assertThat((List) namespaces.get("additional")) + .hasSameElementsAs(config.getApplication().getNamespaces().getActiveNamespaces()); + + Map grafana = (Map) yaml.get("grafana"); + Map rbac = (Map) grafana.get("rbac"); + assertThat(rbac.get("create")).isEqualTo(false); + Map sidecar = (Map) grafana.get("sidecar"); + Map dashboards = (Map) sidecar.get("dashboards"); + assertThat(dashboards.get("searchNamespace")) + .isEqualTo(String.join(",", config.getApplication().getNamespaces().getActiveNamespaces())); + } + + @Test + void networkPoliciesAreCreatedForPrometheus() throws GitAPIException, IOException { + config.getApplication().setNetpols(true); + Monitoring prometheusStack = createStack(scmManagerMock); + install(prometheusStack); + + for (String namespace : config.getApplication().getNamespaces().getActiveNamespaces()) { + File netPolsYaml = new File( + clusterResourcesRepoDir, + "apps/monitoring/misc/netpols/" + namespace + ".yaml" + ); + assertThat(Files.readString(netPolsYaml.toPath())).contains("namespace: " + namespace); + } + } + + @Test + void helmReleasesAreInstalledInAirGappedMode() throws GitAPIException, IOException, URISyntaxException { + config.getApplication().setMirrorRepos(true); + when(airGappedUtils.mirrorHelmRepoToGit(any(HelmChartConfig.class))).thenReturn("a/b"); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path prometheusSourceChart = rootChartsFolder.resolve("kube-prometheus-stack"); + Files.createDirectories(prometheusSourceChart); + + Map prometheusChartYaml = Map.of("version", "1.2.3"); + fileSystemUtils.writeYaml(prometheusChartYaml, prometheusSourceChart.resolve("Chart.yaml").toFile()); + + scmManagerMock.setInClusterBase(new URI("http://scmm.foo-scm-manager.svc.cluster.local/scm")); + install(createStack(scmManagerMock)); + + ArgumentCaptor helmConfig = ArgumentCaptor.forClass(HelmChartConfig.class); + verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()); + assertThat(helmConfig.getValue().chart()).isEqualTo("kube-prometheus-stack"); + assertThat(helmConfig.getValue().repoURL()).isEqualTo("https://prom"); + assertThat(helmConfig.getValue().version()).isEqualTo("19.2.2"); + + verify(deployer).deployFeature( + "http://scmm.foo-scm-manager.svc.cluster.local/scm/repo/a/b", + "monitoring", + ".", + "1.2.3", + "foo-monitoring", + "kube-prometheus-stack", + temporaryYamlFilePrometheus, + RepoType.GIT, + false, + deploymentContext, + repositoryWorkspace + ); + } + + @Test + void mergesAdditionalHelmValuesMergedWithDefaultValues() throws GitAPIException, IOException { + Map prometheusSpec = new HashMap<>(); + prometheusSpec.put("scrapeConfigSelectorNilUsesHelmValues", null); + + Map prometheus = new HashMap<>(); + prometheus.put("prometheusSpec", prometheusSpec); + + Map values = new HashMap<>(); + values.put("key", Map.of("some", "thing", "one", 1)); + values.put("prometheus", prometheus); + config.getFeatures().getMonitoring().getHelm().setValues(values); + + install(createStack(scmManagerMock)); + Map actual = parseActualYaml(); + + Map key = (Map) actual.get("key"); + assertThat(key.get("some")).isEqualTo("thing"); + assertThat(key.get("one")).isEqualTo(1); + + Map actualPrometheus = (Map) actual.get("prometheus"); + Map actualPrometheusSpec = (Map) actualPrometheus.get("prometheusSpec"); + assertThat(actualPrometheusSpec.get("scrapeConfigSelectorNilUsesHelmValues")).isEqualTo(null); + } + + @Test + void serviceMonitorSelectors() throws GitAPIException, IOException { + config.getApplication().setNamePrefix("test1-"); + config.getFeatures().getArgocd().setActive(true); + config.getFeatures().getSecrets().setActive(true); + config.getFeatures().getIngress().setActive(false); + + LinkedHashSet namespaceList = new LinkedHashSet<>(List.of( + "test1-argocd", + "test1-monitoring", + "test1-example-apps-staging", + "test1-example-apps-production", + "test1-secrets" + )); + config.getApplication().getNamespaces().setDedicatedNamespaces(namespaceList); + + install(createStack(scmManagerMock)); + Map actual = parseActualYaml(); + + Map expectedSelector = YAML_MAPPER.readValue( + """ + matchExpressions: + - key: kubernetes.io/metadata.name + operator: In + values: + - test1-argocd + - test1-monitoring + - test1-example-apps-staging + - test1-example-apps-production + - test1-secrets + """, YAML_MAP_TYPE + ); + + Map prometheus = (Map) actual.get("prometheus"); + Map prometheusSpec = (Map) prometheus.get("prometheusSpec"); + assertThat(prometheusSpec.get("serviceMonitorNamespaceSelector")).isEqualTo(expectedSelector); + } + + private Monitoring createStack(ScmManagerProviderMock scmManagerMock) throws GitAPIException { + when(gitHandler.getResourcesScm()).thenReturn(scmManagerMock); + + TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, new FileSystemUtils()) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, scmManagerMock); + clusterResourcesRepoDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + + File dashboardDir = new File(clusterResourcesRepoDir, "apps/monitoring/misc/dashboard"); + dashboardDir.mkdirs(); + + try { + Files.writeString(new File(dashboardDir, "traefik-dashboard.yaml").toPath(), "dummy"); + Files.writeString( + new File(dashboardDir, "traefik-dashboard-requests-handling.yaml").toPath(), + "dummy" + ); + Files.writeString(new File(dashboardDir, "jenkins-dashboard.yaml").toPath(), "dummy"); + Files.writeString(new File(dashboardDir, "scmm-dashboard.yaml").toPath(), "dummy"); + } catch (IOException e) { + throw new RuntimeException(e); + } + + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoProvider.create("argocd/cluster-resources", scmManagerMock); + + repositoryWorkspace = spy(new RepositoryWorkspace(clusterResourcesRepo)); + doNothing().when(repositoryWorkspace).commitAndPushClusterResourcesChanges(anyString()); + + return new Monitoring( + new FileSystemUtils() { + @Override + public Path writeTempFile(Map mapValues) { + Path ret = super.writeTempFile(mapValues); + temporaryYamlFilePrometheus = Path.of(ret.toString().replace(".ftl", "")); + return ret; + } + }, + deployer, + k8sClient, + airGappedUtils, + gitHandler, + imagePullSecretCreator, + new MonitoringToolConfigMapper(config), + new CredentialsResolver(k8sClient) + ); + } + + private boolean install(Monitoring monitoring) { + deploymentContext = new ContextBuilder(config).build(); + return monitoring.execute(deploymentContext, repositoryWorkspace); + } + + private Map parseActualYaml() throws IOException { + return YAML_MAPPER.readValue(temporaryYamlFilePrometheus.toFile(), YAML_MAP_TYPE); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/MonitoringToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/MonitoringToolConfigMapperTest.java new file mode 100644 index 000000000..4db3d7f86 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/MonitoringToolConfigMapperTest.java @@ -0,0 +1,233 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; + +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class MonitoringToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getApplication().getNamespaces().setDedicatedNamespaces(new LinkedHashSet<>(List.of( + "jenkins", + "monitoring" + ))); + config.getApplication().getNamespaces().setTenantNamespaces(new LinkedHashSet<>(List.of("team-a", "team-b"))); + config.getApplication().setNamespaceIsolation(true); + config.getApplication().setNetpols(true); + config.getApplication().setSkipCrds(true); + // Intentionally differs from the DeploymentContext to verify derived values come from the context. + config.getApplication().setOpenshift(false); + config.getApplication().setPodResources(true); + config.getApplication().setPassword("application-password"); + config.getApplication().setUsername("application-user"); + config.getApplication().setCredentials( + new Credentials(null, null, "application-secret", "gop-job", "app-user", "app-password") + ); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setPassword("registry-password"); + config.getJenkins().setActive(true); + config.getJenkins().setInternal(false); + config.getJenkins().setNamespace("jenkins-system"); + config.getJenkins().setUrl("https://jenkins.example.org"); + config.getJenkins().setMetricsUsername("jenkins-metrics-user"); + config.getJenkins().setMetricsPassword("jenkins-metrics-password"); + config.getJenkins().setMetricsCredentials( + new Credentials(null, null, "jenkins-metrics-secret", "gop-job", "metrics-user", "metrics-password") + ); + config.getFeatures().getIngress().setActive(true); + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setIssuer("production-issuer"); + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.org"); + config.getFeatures().getMail().setSmtpPort(2525); + config.getFeatures().getMail().setSmtpUser("smtp-user"); + config.getFeatures().getMail().setSmtpPassword("smtp-password"); + config.getFeatures().getMail().setCredentials( + new Credentials(null, null, "smtp-credentials", "gop-job", "smtp-user", "smtp-password") + ); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getMonitoring().setNamespace("observability"); + config.getFeatures().getMonitoring().setGrafanaUrl("https://grafana.example.org"); + config.getFeatures().getMonitoring().setGrafanaEmailFrom("grafana@example.org"); + config.getFeatures().getMonitoring().setGrafanaEmailTo("team@example.org"); + config.getFeatures().getMonitoring().getOidc().setClientId("grafana-client"); + config.getFeatures().getMonitoring().getHelm().setRepoURL("https://monitoring.example.org"); + config.getFeatures().getMonitoring().getHelm().setChart("monitoring-chart"); + config.getFeatures().getMonitoring().getHelm().setVersion("6.7.8"); + config.getFeatures().getMonitoring().getHelm().setValues(Map.of("retention", "30d")); + config.getFeatures().getMonitoring().getHelm().setGrafanaImage("grafana-image"); + config.getFeatures().getMonitoring().getHelm().setGrafanaSidecarImage("sidecar-image"); + config.getFeatures().getMonitoring().getHelm().setPrometheusImage("prometheus-image"); + config.getFeatures().getMonitoring().getHelm().setPrometheusOperatorImage("operator-image"); + config.getFeatures().getMonitoring().getHelm().setPrometheusConfigReloaderImage("reloader-image"); + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setNamespace("source-control"); + config.getScm().setScmManager(scmManager); + + MonitoringToolConfig actual = new MonitoringToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(MonitoringToolConfig.builder() + .active(true) + .namespace("test-observability") + .namePrefix("test-") + .activeNamespaces(List.of( + "jenkins", + "monitoring", + "team-a", + "team-b" + )) + .namespaceIsolation(true) + .netpols(true) + .skipCrds(true) + .openshift(true) + .airgapped(true) + .applicationUsername("application-user") + .applicationPassword("application-password") + .applicationCredentials(new CredentialsReference( + "application-secret", + "gop-job", + "app-user", + "app-password" + )) + .jenkinsMetricsUsername("jenkins-metrics-user") + .jenkinsMetricsPassword("jenkins-metrics-password") + .jenkinsMetricsCredentials(new CredentialsReference( + "jenkins-metrics-secret", + "gop-job", + "metrics-user", + "metrics-password" + )) + .smtpUser("smtp-user") + .smtpPassword("smtp-password") + .smtpCredentials(new CredentialsReference( + "smtp-credentials", + "gop-job", + "smtp-user", + "smtp-password" + )) + .grafanaUrl("https://grafana.example.org") + .jenkinsInternal(false) + .jenkinsNamespace("jenkins-system") + .jenkinsUrl("https://jenkins.example.org") + .scmProviderType(ScmProviderType.SCM_MANAGER) + .ingressActive(true) + .jenkinsActive(true) + .helm(HelmChartConfig.builder() + .repoURL("https://monitoring.example.org") + .chart("monitoring-chart") + .version("6.7.8") + .values(Map.of("retention", "30d")) + .localHelmChartFolder("/charts") + .build()) + .imagePullSecret(imagePullSecret()) + .templateConfig(Map.of( + "application", Map.of( + "namePrefix", "test-", + "namespaceIsolation", true, + "openshift", true, + "podResources", true, + "skipCrds", true + ), + "features", Map.of( + "certManager", + Map.of("active", true, "issuer", "production-issuer"), + "mail", + Map.of( + "active", true, + "smtpAddress", "smtp.example.org", + "smtpCredentialsConfigured", true, + "smtpPort", 2525 + ), + "monitoring", + Map.of( + "grafanaEmailFrom", "grafana@example.org", + "grafanaEmailTo", "team@example.org", + "grafanaUrl", "https://grafana.example.org", + "namespace", "observability", + "oidc", Map.of( + "providerName", + "Keycloak", + "issuerUrl", + "", + "clientId", + "grafana-client", + "clientSecret", + "", + "scopes", + List.of("openid", "profile", "email"), + "adminGroupName", + "", + "enabled", + false + ), + "helm", Map.of( + "grafanaImage", + "grafana-image", + "grafanaSidecarImage", + "sidecar-image", + "prometheusConfigReloaderImage", + "reloader-image", + "prometheusImage", + "prometheus-image", + "prometheusOperatorImage", + "operator-image" + ) + ) + ), + "jenkins", Map.of("active", true), + "registry", Map.of("createImagePullSecrets", true), + "scm", Map.of( + "scmManager", Map.of("namespace", "source-control"), + "scmProviderType", ScmProviderType.SCM_MANAGER + ) + )) + .build()); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.MULTI_TENANT, + DeploymentContext.ScmManagerDeploymentMode.INTERNAL, + true, + DeploymentContext.ClusterDistribution.OPENSHIFT + ); + } + + private static ImagePullSecretConfig imagePullSecret() { + return ImagePullSecretConfig.builder() + .create(true) + .proxyUrl("proxy.example.org") + .url("registry.example.org") + .proxyUsername("proxy-user") + .readOnlyUsername("read-only-user") + .username("registry-user") + .proxyPassword("proxy-password") + .readOnlyPassword("read-only-password") + .password("registry-password") + .build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/RegistryTest.java b/src/test/java/com/cloudogu/gitops/tools/RegistryTest.java new file mode 100644 index 000000000..f75224b69 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/RegistryTest.java @@ -0,0 +1,154 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.K8sClientForTest; +import com.cloudogu.gitops.utils.YamlUtils; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.LinkedHashMap; +import java.util.Map; + +import static com.cloudogu.gitops.config.Config.DEFAULT_REGISTRY_PORT; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.verify; + +@ExtendWith(MockitoExtension.class) +class RegistryTest { + + private K8sClientForTest k8sClient; + private Path temporaryYamlFile; + private HelmClient helmClient; + private DeploymentContext deploymentContext; + + @Mock + private Deployer deployer; + + @Mock + private RepositoryWorkspace repositoryWorkspace; + + @Test + void isDisabledWhenExternalRegistryIsConfigured() { + Config.RegistrySchema registryConfig = new Config.RegistrySchema(); + + assertFalse(createRegistry(registryConfig).isEnabled(createContext(registryConfig))); + } + + @Test + void isInstalled() throws IOException, GitAPIException { + Config.RegistrySchema registryConfig = new Config.RegistrySchema(); + registryConfig.setActive(true); + registryConfig.setInternal(true); + + install(createRegistry(registryConfig), registryConfig); + + Map actualYaml = parseActualYaml(); + Map service = (Map) actualYaml.get("service"); + assertThat(service.get("nodePort")).isEqualTo(DEFAULT_REGISTRY_PORT); + assertThat(service.get("type")).isEqualTo("NodePort"); + + verify(deployer).deployFeature( + anyString(), + eq("registry"), + eq("docker-registry"), + anyString(), + eq("foo-registry"), + eq("docker-registry"), + any(Path.class), + eq(RepoType.HELM), + eq(true), + eq(deploymentContext), + eq(repositoryWorkspace) + ); + + verify(repositoryWorkspace).commitAndPushClusterResourcesChanges("Update registry GitOps resources"); + } + + @Test + void injectCustomValueIntoChart() throws IOException, GitAPIException { + Config.RegistrySchema registryConfig = new Config.RegistrySchema(); + registryConfig.setActive(true); + registryConfig.setInternal(true); + + Config.HelmConfigWithValues helm = new Config.HelmConfigWithValues(); + helm.setChart("test"); + + Map service = new LinkedHashMap<>(); + service.put("type", "NodePortTest"); + Map values = new LinkedHashMap<>(); + values.put("service", service); + values.put("customValue", "testinjectionValue"); + helm.setValues(values); + registryConfig.setHelm(helm); + + install(createRegistry(registryConfig), registryConfig); + + assertThat(String.valueOf(parseActualYaml().get("service"))).contains("NodePortTest"); + assertThat(String.valueOf(parseActualYaml().get("customValue"))).contains("testinjectionValue"); + + verify(repositoryWorkspace).commitAndPushClusterResourcesChanges("Update registry GitOps resources"); + } + + private Registry createRegistry() { + return createRegistry(new Config.RegistrySchema()); + } + + private Registry createRegistry(Config.RegistrySchema registryConfig) { + Config config = createConfig(registryConfig); + k8sClient = new K8sClientForTest(); + + FileSystemUtils fileUtil = new FileSystemUtils() { + @Override + public Path writeTempFile(Map mergeMap) { + Path result = super.writeTempFile(mergeMap); + temporaryYamlFile = Path.of(result.toString().replace(".ftl", "")); + return result; + } + }; + + AirGappedUtils airGappedUtils = new AirGappedUtils(null, fileUtil, helmClient, null); + + return new Registry(fileUtil, k8sClient, airGappedUtils, deployer, new RegistryToolConfigMapper(config)); + } + + private boolean install(Registry registry, Config.RegistrySchema registryConfig) { + deploymentContext = createContext(registryConfig); + return registry.execute(deploymentContext, repositoryWorkspace); + } + + private DeploymentContext createContext(Config.RegistrySchema registryConfig) { + return new ContextBuilder(createConfig(registryConfig)).build(); + } + + private Config createConfig(Config.RegistrySchema registryConfig) { + Config.ApplicationSchema application = new Config.ApplicationSchema(); + application.setNamePrefix("foo-"); + + Config config = new Config(); + config.setApplication(application); + config.setRegistry(registryConfig); + return config; + } + + private Map parseActualYaml() throws IOException { + return YamlUtils.parseYamlMap(Files.readString(temporaryYamlFile)); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/RegistryToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/RegistryToolConfigMapperTest.java new file mode 100644 index 000000000..7550b7f4c --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/RegistryToolConfigMapperTest.java @@ -0,0 +1,64 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class RegistryToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getRegistry().setActive(true); + config.getRegistry().setInternal(true); + config.getRegistry().setNamespace("images"); + config.getRegistry().setInternalPort(32000); + config.getRegistry().getHelm().setRepoURL("https://registry.example.org"); + config.getRegistry().getHelm().setChart("registry-chart"); + config.getRegistry().getHelm().setVersion("4.5.6"); + config.getRegistry().getHelm().setValues(Map.of("storage", "memory")); + + RegistryToolConfig actual = new RegistryToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(RegistryToolConfig.builder() + .active(true) + .internal(true) + .namespace("test-images") + .bootstrapNodePort(Config.DEFAULT_REGISTRY_PORT) + .internalPort(32000) + .helm(HelmChartConfig.builder() + .repoURL("https://registry.example.org") + .chart("registry-chart") + .version("4.5.6") + .values(Map.of("storage", "memory")) + .localHelmChartFolder("/charts") + .build()) + .build()); + } + + @Test + void doesNotExposeANamespaceForAnExternalRegistry() { + Config config = new Config(); + config.getRegistry().setInternal(false); + + RegistryToolConfig actual = new RegistryToolConfigMapper(config).map(context()); + + assertThat(actual.namespace()).isNull(); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.SINGLE_TENANT, + DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + false, + DeploymentContext.ClusterDistribution.KUBERNETES + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/VaultTest.java b/src/test/java/com/cloudogu/gitops/tools/VaultTest.java new file mode 100644 index 000000000..0827d3844 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/VaultTest.java @@ -0,0 +1,470 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.CommandExecutorForTest; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Base64; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@EnableKubernetesMockClient(crud = true) +@MockitoSettings(strictness = Strictness.LENIENT) +class VaultTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + private final Config config = new Config(); + + private final CommandExecutorForTest helmCommands = new CommandExecutorForTest(); + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + private final Deployer deployer = mock(Deployer.class); + private final AirGappedUtils airGappedUtils = mock(AirGappedUtils.class); + + private final ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + private final GitHandler gitHandler = new GitHandlerForTests(scmManagerMock); + private final ImagePullSecretCreator imagePullSecretCreator = mock(ImagePullSecretCreator.class); + + private Path temporaryYamlFile; + private File clusterResourcesRepoDir; + private RepositoryWorkspace repositoryWorkspace; + private DeploymentContext deploymentContext; + + private K8sClient k8sClient; + KubernetesClient client; + + VaultTest() { + config.getApplication().setNamePrefix("foo-"); + config.getFeatures().getSecrets().setActive(true); + } + + @BeforeEach + void init() { + k8sClient = new K8sClient(); + k8sClient.setClient(client); + } + + @Test + void isDisabledViaActiveFlag() throws GitAPIException { + config.getFeatures().getSecrets().setActive(false); + + assertFalse(createVault().isEnabled(new ContextBuilder(config).build())); + } + + @Test + void preparesVaultAppContentInClusterResourcesWorkspaceWithoutCopyingTemplates() throws GitAPIException { + install(createVault()); + + assertThat(new File(clusterResourcesRepoDir, "apps/vault")).exists(); + assertThat(new File(clusterResourcesRepoDir, "apps/vault/templates")).doesNotExist(); + } + + @Test + void usesIngressIfEnabled() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setUrl("http://vault.local"); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + Map ingressYaml = (Map) server.get("ingress"); + assertThat(ingressYaml.get("enabled")).isEqualTo(true); + List> hosts = (List>) ingressYaml.get("hosts"); + assertThat(hosts.get(0).get("host")).isEqualTo("vault.local"); + } + + @Test + void usesIngressIfEnabledAndImageSet() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setUrl("http://vault.local"); + // Also set image to make sure ingress and image work at the same time under the server block + // config.getFeatures().getSecrets().getVault().getHelm().setImage("localhost:5000/hashicorp/vault:1.12.0"); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + Map ingressYaml = (Map) server.get("ingress"); + assertThat(ingressYaml.get("enabled")).isEqualTo(true); + } + + @Test + void doesNotUseIngressByDefault() throws GitAPIException, IOException { + install(createVault()); + + assertThat(parseActualYaml()).doesNotContainKey("server"); + } + + @Test + void devModeCanBeEnabledViaConfig() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.DEV); + config.getApplication().setUsername("abc"); + config.getApplication().setPassword("123"); + config.getFeatures().getArgocd().setActive(true); + + Vault vault = createVault(); + + install(vault); + + Map actualYaml = parseActualYaml(); + Map server = (Map) actualYaml.get("server"); + Map dev = (Map) server.get("dev"); + assertThat(dev.get("enabled")).isEqualTo(true); + + assertThat(dev.get("devRootToken")).isNotEqualTo("root"); + assertThat(dev.get("devRootToken")).isNotEqualTo(config.getApplication().getPassword()); + + List actualPostStart = (List) server.get("postStart"); + assertThat(actualPostStart.get(0)).isEqualTo("/bin/sh"); + assertThat(actualPostStart.get(1)).isEqualTo("-c"); + + assertThat(normalizeShellCommand((String) actualPostStart.get(2))) + .isEqualTo( + "ARGOCD=true OIDC_ENABLED=false /var/opt/scripts/dev-post-start.sh 2>&1 | tee /tmp/dev-post-start.log"); + + List> actualVolumes = (List>) server.get("volumes"); + List> actualVolumeMounts = (List>) server.get("volumeMounts"); + assertThat(actualVolumes.get(0).get("name")).isEqualTo(actualVolumeMounts.get(0).get("name")); + Map configMap = (Map) actualVolumes.get(0).get("configMap"); + assertThat(configMap.get("defaultMode")).isEqualTo(Integer.valueOf(0774)); + + assertThat(actualVolumeMounts.get(0).get("readOnly")).isEqualTo(true); + assertThat((String) actualPostStart.get(2)) + .contains((String) actualVolumeMounts.get(0).get("mountPath") + "/dev-post-start.sh"); + + assertThat(server).doesNotContainKey("resources"); + } + + @Test + void devModeCanBeEnabledViaConfigWithArgoCDDisabled() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.DEV); + config.getApplication().setUsername("abc"); + config.getApplication().setPassword("123"); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + List actualPostStart = (List) server.get("postStart"); + assertThat(normalizeShellCommand((String) actualPostStart.get(2))) + .isEqualTo( + "ARGOCD=false OIDC_ENABLED=false /var/opt/scripts/dev-post-start.sh 2>&1 | tee /tmp/dev-post-start.log"); + } + + @Test + void devModeEnablesOIDCOnlyWhenConfigured() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.DEV); + config.getFeatures().getSecrets().getVault().setUrl("http://vault.localhost"); + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setClientId("vault-client"); + oidc.setClientSecret("vault-secret"); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setAdminGroupName("gop-admins"); + config.getFeatures().getSecrets().getVault().setOidc(oidc); + config.getApplication().setPassword("admin"); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + List actualPostStart = (List) server.get("postStart"); + assertThat(normalizeShellCommand((String) actualPostStart.get(2))) + .isEqualTo( + "ARGOCD=false OIDC_ENABLED=true OIDC_CLIENT_ID=vault-client OIDC_CLIENT_SECRET=vault-secret OIDC_DISCOVERY_URL=http://keycloak.local.gd/realms/gop OIDC_ADMIN_GROUP=gop-admins VAULT_EXTERNAL_URL=http://vault.localhost /var/opt/scripts/dev-post-start.sh 2>&1 | tee /tmp/dev-post-start.log"); + } + + @Test + void devModeDoesNotEnableOIDCWhenOIDCConfigIsIncomplete() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.DEV); + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setClientSecret("vault-secret"); + config.getFeatures().getSecrets().getVault().setOidc(oidc); + config.getApplication().setUsername("admin"); + config.getApplication().setPassword("admin"); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + List actualPostStart = (List) server.get("postStart"); + assertThat(normalizeShellCommand((String) actualPostStart.get(2))) + .isEqualTo( + "ARGOCD=false OIDC_ENABLED=false /var/opt/scripts/dev-post-start.sh 2>&1 | tee /tmp/dev-post-start.log"); + } + + @Test + void devModeResolvesApplicationCredentialsWithoutRenderingThem() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.DEV); + config.getApplication().setUsername("fallback-user"); + config.getApplication().setPassword("fallback-password"); + + Credentials reference = new Credentials(); + reference.setSecretName("application-credentials"); + reference.setSecretNamespace("gop-job"); + config.getApplication().setCredentials(reference); + + Secret sourceSecret = new SecretBuilder() + .withNewMetadata() + .withName("application-credentials") + .withNamespace("gop-job") + .endMetadata() + .withType("Opaque") + .withData(Map.of( + "username", Base64.getEncoder().encodeToString("secret-user".getBytes(StandardCharsets.UTF_8)), + "password", Base64.getEncoder().encodeToString("secret-password".getBytes(StandardCharsets.UTF_8)) + )) + .build(); + + client.secrets().inNamespace("gop-job").resource(sourceSecret).create(); + + install(createVault()); + + var targetSecret = client.secrets() + .inNamespace("foo-secrets") + .withName("vault-user-credentials") + .get(); + assertThat(secretValue(targetSecret, "username")).isEqualTo("secret-user"); + assertThat(secretValue(targetSecret, "password")).isEqualTo("secret-password"); + + Map server = (Map) parseActualYaml().get("server"); + List> secretEnv = (List>) server.get("extraSecretEnvironmentVars"); + assertThat(secretEnv).containsExactly( + Map.of( + "envName", "USERNAME", + "secretName", "vault-user-credentials", + "secretKey", "username" + ), + Map.of( + "envName", "PASSWORD", + "secretName", "vault-user-credentials", + "secretKey", "password" + ) + ); + + String renderedValues = Files.readString(temporaryYamlFile); + assertThat(renderedValues).doesNotContain("secret-user", "secret-password"); + assertThat(config.getApplication().getUsername()).isEqualTo("fallback-user"); + assertThat(config.getApplication().getPassword()).isEqualTo("fallback-password"); + assertThat(config.getApplication().getCredentials().getSecretName()).isEqualTo("application-credentials"); + } + + @Test + void prodModeCanBeEnabled() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.PROD); + + install(createVault()); + + assertThat(parseActualYaml()).doesNotContainKey("server"); + } + + @Test + void customImageIsUsed() throws GitAPIException, IOException { + config.getFeatures().getSecrets().getVault().getHelm().setImage("localhost:5000/hashicorp/vault:1.12.0"); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + Map image = (Map) server.get("image"); + assertThat(image.get("repository")).isEqualTo("localhost:5000/hashicorp/vault"); + assertThat(image.get("tag")).isEqualTo("1.12.0"); + } + + @Test + void helmReleaseIsInstalled() throws GitAPIException, IOException { + Config.SecretsSchema.VaultSchema.VaultHelmSchema helm = new Config.SecretsSchema.VaultSchema.VaultHelmSchema(); + helm.setChart("vault"); + helm.setRepoURL("https://vault-reg"); + helm.setVersion("42.23.0"); + config.getFeatures().getSecrets().getVault().setHelm(helm); + + install(createVault()); + + verify(deployer).deployFeature( + "https://vault-reg", + "vault", + "vault", + "42.23.0", + "foo-secrets", + "vault", + temporaryYamlFile, + RepoType.HELM, + false, + deploymentContext, + repositoryWorkspace + ); + + assertThat(parseActualYaml()).doesNotContainKey("global"); + } + + @Test + void helmReleaseIsInstalledInAirGappedMode() throws GitAPIException, IOException { + config.getApplication().setMirrorRepos(true); + Config.SecretsSchema.VaultSchema.VaultHelmSchema helm = new Config.SecretsSchema.VaultSchema.VaultHelmSchema(); + helm.setChart("vault"); + helm.setRepoURL("https://vault-reg"); + helm.setVersion("42.23.0"); + config.getFeatures().getSecrets().getVault().setHelm(helm); + + when(airGappedUtils.mirrorHelmRepoToGit(any(HelmChartConfig.class))).thenReturn("a/b"); + + Path rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + config.getApplication().setLocalHelmChartFolder(rootChartsFolder.toString()); + + Path sourceChart = rootChartsFolder.resolve("vault"); + Files.createDirectories(sourceChart); + + Map chartYaml = Map.of("version", "1.2.3"); + fileSystemUtils.writeYaml(chartYaml, sourceChart.resolve("Chart.yaml").toFile()); + + install(createVault()); + + ArgumentCaptor helmConfig = ArgumentCaptor.forClass(HelmChartConfig.class); + verify(airGappedUtils).mirrorHelmRepoToGit(helmConfig.capture()); + assertThat(helmConfig.getValue().chart()).isEqualTo("vault"); + assertThat(helmConfig.getValue().repoURL()).isEqualTo("https://vault-reg"); + assertThat(helmConfig.getValue().version()).isEqualTo("42.23.0"); + + verify(deployer).deployFeature( + "http://scmm.scm-manager.svc.cluster.local/scm/repo/a/b", + "vault", + ".", + "1.2.3", + "foo-secrets", + "vault", + temporaryYamlFile, + RepoType.GIT, + false, + deploymentContext, + repositoryWorkspace + ); + } + + @Test + void setsPodResourceLimitsAndRequests() throws GitAPIException, IOException { + config.getApplication().setPodResources(true); + + install(createVault()); + + Map server = (Map) parseActualYaml().get("server"); + assertThat((Map) server.get("resources")).containsKeys("limits", "requests"); + } + + @Test + void deploysImagePullSecretsForProxyRegistry() throws GitAPIException, IOException { + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-pw"); + + install(createVault()); + + Map global = (Map) parseActualYaml().get("global"); + assertThat(global.get("imagePullSecrets")).isEqualTo(List.of(Map.of("name", "proxy-registry"))); + } + + private Vault createVault() throws GitAPIException { + // We use the real FileSystemUtils and not a mock to make sure file editing works as expected + FileSystemUtils testFileSystemUtils = new FileSystemUtils() { + @Override + public Path writeTempFile(Map mapValues) { + Path ret = super.writeTempFile(mapValues); + temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")); + return ret; + } + }; + + TestGitRepoFactory repoProvider = new TestGitRepoFactory(config, testFileSystemUtils) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, gitProvider); + clusterResourcesRepoDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoProvider.create("argocd/cluster-resources", scmManagerMock); + + repositoryWorkspace = spy(new RepositoryWorkspace(clusterResourcesRepo)); + doNothing().when(repositoryWorkspace).commitAndPushClusterResourcesChanges(anyString()); + + return new Vault( + testFileSystemUtils, + deployer, + k8sClient, + airGappedUtils, + gitHandler, + imagePullSecretCreator, + new VaultToolConfigMapper(config), + new CredentialsResolver(k8sClient) + ); + } + + private boolean install(Vault vault) { + deploymentContext = new ContextBuilder(config).build(); + return vault.execute(deploymentContext, repositoryWorkspace); + } + + private Map parseActualYaml() throws IOException { + return YAML_MAPPER.readValue(temporaryYamlFile.toFile(), YAML_MAP_TYPE); + } + + private static String secretValue(Secret secret, String key) { + if (secret.getStringData() != null && secret.getStringData().get(key) != null) { + return secret.getStringData().get(key); + } + + return decodeSecretValue(secret.getData().get(key)); + } + + private static String decodeSecretValue(String value) { + return new String(Base64.getDecoder().decode(value), StandardCharsets.UTF_8); + } + + private static String normalizeShellCommand(String command) { + return command + .replaceAll("\\\\\\s*\\r?\\n\\s*", " ") + .replaceAll("\\s+", " ") + .trim(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/VaultToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/VaultToolConfigMapperTest.java new file mode 100644 index 000000000..0e66558eb --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/VaultToolConfigMapperTest.java @@ -0,0 +1,165 @@ +package com.cloudogu.gitops.tools; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class VaultToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = config(); + config.getFeatures().getSecrets().getVault().setMode(Config.VaultMode.PROD); + config.getApplication().getCredentials().setSecretName("application-credentials"); + config.getApplication().getCredentials().setSecretNamespace("gop-job"); + + VaultToolConfig actual = new VaultToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(VaultToolConfig.builder() + .active(true) + .namespace("test-secrets") + .namePrefix("test-") + .url("https://vault.example.org") + .applicationUsername("application-user") + .applicationPassword("application-password") + .applicationCredentials(new CredentialsReference( + "application-credentials", "gop-job", "username", "password" + )) + .developmentMode(false) + .helm(HelmChartConfig.builder() + .repoURL("https://vault-chart.example.org") + .chart("vault-chart") + .version("5.6.7") + .values(Map.of("ha", true)) + .localHelmChartFolder("/charts") + .build()) + .imagePullSecret(imagePullSecret()) + .templateConfig(Map.of( + "application", Map.of( + "namePrefix", "test-", + "namespaceIsolation", true, + "openshift", true, + "podResources", true + ), + "features", Map.of( + "argocd", Map.of("active", true), + "certManager", Map.of( + "active", true, + "issuer", "production-issuer" + ), + "secrets", Map.of( + "vault", Map.of( + "oidc", Map.of( + "providerName", + "Keycloak", + "issuerUrl", + "", + "clientId", + "vault-client", + "clientSecret", + "", + "scopes", + java.util.List.of("openid", "profile", "email"), + "adminGroupName", + "", + "enabled", + false + ), + "helm", Map.of("image", "vault-image") + ) + ) + ), + "registry", Map.of("createImagePullSecrets", true) + )) + .build()); + } + + @ParameterizedTest + @CsvSource({ + "DEV, true", + "PROD, false" + }) + void mapsVaultModeToDevelopmentMode(Config.VaultMode mode, boolean expectedDevelopmentMode) { + Config config = config(); + config.getFeatures().getSecrets().getVault().setMode(mode); + + VaultToolConfig actual = new VaultToolConfigMapper(config).map(context()); + + assertThat(actual.developmentMode()).isEqualTo(expectedDevelopmentMode); + } + + private static Config config() { + Config config = new Config(); + + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getApplication().setNamespaceIsolation(true); + // Intentionally differs from the DeploymentContext to verify derived values come from the context. + config.getApplication().setOpenshift(false); + config.getApplication().setPassword("application-password"); + config.getApplication().setPodResources(true); + config.getApplication().setUsername("application-user"); + config.getApplication().setCredentials(new Credentials()); + + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setPassword("registry-password"); + + config.getFeatures().getArgocd().setActive(true); + + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setIssuer("production-issuer"); + + config.getFeatures().getSecrets().setActive(true); + config.getFeatures().getSecrets().setNamespace("secrets"); + config.getFeatures().getSecrets().getVault().setUrl("https://vault.example.org"); + config.getFeatures().getSecrets().getVault().getOidc().setClientId("vault-client"); + + config.getFeatures().getSecrets().getVault().getHelm().setRepoURL("https://vault-chart.example.org"); + config.getFeatures().getSecrets().getVault().getHelm().setChart("vault-chart"); + config.getFeatures().getSecrets().getVault().getHelm().setVersion("5.6.7"); + config.getFeatures().getSecrets().getVault().getHelm().setValues(Map.of("ha", true)); + config.getFeatures().getSecrets().getVault().getHelm().setImage("vault-image"); + + return config; + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.SINGLE_TENANT, + DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + false, + DeploymentContext.ClusterDistribution.OPENSHIFT + ); + } + + private static ImagePullSecretConfig imagePullSecret() { + return ImagePullSecretConfig.builder() + .create(true) + .proxyUrl("proxy.example.org") + .url("registry.example.org") + .proxyUsername("proxy-user") + .readOnlyUsername("read-only-user") + .username("registry-user") + .proxyPassword("proxy-password") + .readOnlyPassword("read-only-password") + .password("registry-password") + .build(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/common/AbstractToolTest.java b/src/test/java/com/cloudogu/gitops/tools/common/AbstractToolTest.java new file mode 100644 index 000000000..dac89d893 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/common/AbstractToolTest.java @@ -0,0 +1,75 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.Mockito.mock; + +class AbstractToolTest { + + @Test + void executeStoresContextAndRepositoryWorkspaceAndMapsConfigBeforeLifecycleExecution() { + ToolForTest tool = new ToolForTest(); + DeploymentContext newContext = new ContextBuilder(new Config()).build(); + RepositoryWorkspace workspace = new RepositoryWorkspace(mock(GitRepo.class)); + + tool.execute(newContext, workspace); + + assertThat(tool.context).isSameAs(newContext); + assertThat(tool.repositoryWorkspace).isSameAs(workspace); + assertThat(tool.configSeenDuringValidation).isTrue(); + } + + @Test + void activationUsesMappedToolConfig() { + ToolForTest tool = new ToolForTest(ignored -> false); + + assertThat(tool.isEnabled(new ContextBuilder(new Config()).build())).isFalse(); + } + + @Test + void mappedToolsRejectAMissingMapper() { + assertThatThrownBy(() -> new ToolForTest(null)) + .isInstanceOf(NullPointerException.class) + .hasMessage("Tool config mapper must not be null"); + } + + @Test + void mappedToolsRejectANullMapperResult() { + DeploymentContext context = new ContextBuilder(new Config()).build(); + ToolForTest tool = new ToolForTest(ignored -> null); + + assertThatThrownBy(() -> tool.isEnabled(context)) + .isInstanceOf(NullPointerException.class) + .hasMessageContaining("Tool config mapper returned null"); + } + + private static class ToolForTest extends AbstractMappedTool { + + private Boolean configSeenDuringValidation; + + ToolForTest() { + this(ignored -> true); + } + + ToolForTest(ToolConfigMapper mapper) { + super(mapper); + } + + @Override + protected boolean isEnabled(Boolean config) { + return config; + } + + @Override + public void validate() { + configSeenDuringValidation = toolConfig(); + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/common/ImagePullSecretCreatorTest.java b/src/test/java/com/cloudogu/gitops/tools/common/ImagePullSecretCreatorTest.java new file mode 100644 index 000000000..a9af961c5 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/common/ImagePullSecretCreatorTest.java @@ -0,0 +1,239 @@ +package com.cloudogu.gitops.tools.common; + +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; +import java.util.Base64; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +@EnableKubernetesMockClient(crud = true) +class ImagePullSecretCreatorTest { + + private static final String NAMESPACE = "foo-my-ns"; + private static final String SECRET_NAME = "proxy-registry"; + + KubernetesClient client; + private K8sClient k8sClient; + private ImagePullSecretCreator imagePullSecretCreator; + + @BeforeEach + void init() { + k8sClient = new K8sClient(); + k8sClient.setClient(client); + imagePullSecretCreator = new ImagePullSecretCreator(k8sClient, new CredentialsResolver(k8sClient)); + } + + @Test + void doesNotCreateImagePullSecretWhenDisabled() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(false); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + assertThat(secret()).isNull(); + } + + @Test + void createsImagePullSecretWithProxyCredentialsWhenProxyIsConfigured() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-pw"); + config.getRegistry().setUrl("url"); + config.getRegistry().setReadOnlyUsername("ROuser"); + config.getRegistry().setReadOnlyPassword("ROpw"); + config.getRegistry().setUsername("user"); + config.getRegistry().setPassword("pw"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + Secret secret = secret(); + + assertThat(secret).isNotNull(); + assertThat(secret.getType()).isEqualTo("kubernetes.io/dockerconfigjson"); + assertDockerConfigContains(secret, "proxy-url", "proxy-user", "proxy-pw"); + } + + @Test + void createsImagePullSecretWithReadOnlyCredentialsWhenProxyCredentialsAreNotConfigured() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setUrl("url"); + config.getRegistry().setReadOnlyUsername("ROuser"); + config.getRegistry().setReadOnlyPassword("ROpw"); + config.getRegistry().setUsername("user"); + config.getRegistry().setPassword("pw"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + Secret secret = secret(); + + assertThat(secret).isNotNull(); + assertThat(secret.getType()).isEqualTo("kubernetes.io/dockerconfigjson"); + assertDockerConfigContains(secret, "url", "ROuser", "ROpw"); + } + + @Test + void createsImagePullSecretWithDefaultCredentialsWhenReadOnlyCredentialsAreNotConfigured() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setUrl("url"); + config.getRegistry().setUsername("user"); + config.getRegistry().setPassword("pw"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + Secret secret = secret(); + + assertThat(secret).isNotNull(); + assertThat(secret.getType()).isEqualTo("kubernetes.io/dockerconfigjson"); + assertDockerConfigContains(secret, "url", "user", "pw"); + } + + @Test + void createsImagePullSecretWithProxySecretCredentials() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy-url"); + config.getRegistry().setProxyCredentials( + new Credentials(null, null, "proxy-credentials", "gop-job") + ); + config.getRegistry().setUrl("url"); + config.getRegistry().setUsername("user"); + config.getRegistry().setPassword("pw"); + createCredentialsSecret("proxy-credentials", "proxy-secret-user", "proxy-secret-password"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + assertDockerConfigContains(secret(), "proxy-url", "proxy-secret-user", "proxy-secret-password"); + } + + @Test + void createsImagePullSecretWithReadOnlySecretCredentials() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setUrl("url"); + config.getRegistry().setReadOnlyCredentials( + new Credentials(null, null, "read-only-credentials", "gop-job") + ); + config.getRegistry().setUsername("user"); + config.getRegistry().setPassword("pw"); + createCredentialsSecret("read-only-credentials", "read-only-secret-user", "read-only-secret-password"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + assertDockerConfigContains(secret(), "url", "read-only-secret-user", "read-only-secret-password"); + } + + @Test + void createsImagePullSecretWithDefaultSecretCredentials() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setUrl("url"); + config.getRegistry().setCredentials( + new Credentials(null, null, "registry-credentials", "gop-job") + ); + createCredentialsSecret("registry-credentials", "secret-user", "secret-password"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + assertDockerConfigContains(secret(), "url", "secret-user", "secret-password"); + } + + @Test + void createsNamespaceBeforeCreatingImagePullSecret() { + Config config = new Config(); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setUrl("url"); + config.getRegistry().setUsername("user"); + config.getRegistry().setPassword("pw"); + + imagePullSecretCreator.createIfRequired( + ToolConfigMapperSupport.imagePullSecret(config.getRegistry()), + NAMESPACE + ); + + assertThat(client.namespaces().withName(NAMESPACE).get()).isNotNull(); + assertThat(secret()).isNotNull(); + } + + private void createCredentialsSecret(String name, String username, String password) { + Secret secret = new SecretBuilder() + .withNewMetadata() + .withName(name) + .withNamespace("gop-job") + .endMetadata() + .withType("Opaque") + .withData(Map.of( + "username", Base64.getEncoder().encodeToString(username.getBytes(StandardCharsets.UTF_8)), + "password", Base64.getEncoder().encodeToString(password.getBytes(StandardCharsets.UTF_8)) + )) + .build(); + + client.secrets().inNamespace("gop-job").resource(secret).create(); + } + + private Secret secret() { + return client.secrets() + .inNamespace(NAMESPACE) + .withName(SECRET_NAME) + .get(); + } + + private static void assertDockerConfigContains( + Secret secret, + String expectedUrl, + String expectedUsername, + String expectedPassword) { + String dockerConfigJson = decodeSecretValue(secret, ".dockerconfigjson"); + + assertThat(dockerConfigJson).contains(expectedUrl); + assertThat(dockerConfigJson).contains(expectedUsername); + assertThat(dockerConfigJson).contains(expectedPassword); + } + + private static String decodeSecretValue(Secret secret, String key) { + if (secret.getStringData() != null && secret.getStringData().containsKey(key)) { + return secret.getStringData().get(key); + } + + if (secret.getData() != null && secret.getData().containsKey(key)) { + return new String(Base64.getDecoder().decode(secret.getData().get(key)), StandardCharsets.UTF_8); + } + + return null; + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/common/ImmutableConfigDataTest.java b/src/test/java/com/cloudogu/gitops/tools/common/ImmutableConfigDataTest.java new file mode 100644 index 000000000..110476304 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/common/ImmutableConfigDataTest.java @@ -0,0 +1,52 @@ +package com.cloudogu.gitops.tools.common; + +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class ImmutableConfigDataTest { + + @Test + void createsADeepDefensiveCopyWhilePreservingNullValues() { + Map nested = new LinkedHashMap<>(); + nested.put("value", "before"); + List list = new ArrayList<>(List.of(nested)); + list.add(null); + + Map source = new LinkedHashMap<>(); + source.put("nullable", null); + source.put("nested", nested); + source.put("list", list); + + Map result = ImmutableConfigData.copyMap(source); + + nested.put("value", "after"); + list.add("later"); + source.put("additional", true); + + Map expectedNested = new LinkedHashMap<>(); + expectedNested.put("value", "before"); + List expectedList = new ArrayList<>(); + expectedList.add(expectedNested); + expectedList.add(null); + + Map expected = new LinkedHashMap<>(); + expected.put("nullable", null); + expected.put("nested", expectedNested); + expected.put("list", expectedList); + + assertThat(result).isEqualTo(expected); + assertThatThrownBy(() -> result.put("other", "value")) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> ((Map) result.get("nested")).put("other", "value")) + .isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> ((List) result.get("list")).add("value")) + .isInstanceOf(UnsupportedOperationException.class); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/common/TemplateConfigTest.java b/src/test/java/com/cloudogu/gitops/tools/common/TemplateConfigTest.java new file mode 100644 index 000000000..573621252 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/common/TemplateConfigTest.java @@ -0,0 +1,33 @@ +package com.cloudogu.gitops.tools.common; + +import org.junit.jupiter.api.Test; + +import java.util.LinkedHashMap; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class TemplateConfigTest { + + @Test + void buildsAnImmutableNestedTemplateView() { + Map result = new TemplateConfig() + .put("application.namePrefix", "test-") + .put("application.optionalValue", null) + .put("features.argocd.active", true) + .values(); + + Map application = new LinkedHashMap<>(); + application.put("namePrefix", "test-"); + application.put("optionalValue", null); + + Map expected = new LinkedHashMap<>(); + expected.put("application", application); + expected.put("features", Map.of("argocd", Map.of("active", true))); + + assertThat(result).isEqualTo(expected); + assertThatThrownBy(() -> ((Map) result.get("application")).put("other", true)) + .isInstanceOf(UnsupportedOperationException.class); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/JenkinsTest.java b/src/test/java/com/cloudogu/gitops/tools/core/JenkinsTest.java new file mode 100644 index 000000000..5fe3f8870 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/JenkinsTest.java @@ -0,0 +1,721 @@ +package com.cloudogu.gitops.tools.core; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.credentials.ResolvedCredentials; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.jenkins.GlobalPropertyManager; +import com.cloudogu.gitops.infrastructure.jenkins.JenkinsApiClient; +import com.cloudogu.gitops.infrastructure.jenkins.JobManager; +import com.cloudogu.gitops.infrastructure.jenkins.PrometheusConfigurator; +import com.cloudogu.gitops.infrastructure.jenkins.UserManager; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.common.ImagePullSecretCreator; +import com.cloudogu.gitops.utils.AirGappedUtils; +import com.cloudogu.gitops.utils.CommandExecutorForTest; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.NetworkingUtils; +import com.cloudogu.gitops.utils.Tuple; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +import static com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy.RepoType; +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyMap; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class JenkinsTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + + private final Config config; + private final String expectedNodeName = "something"; + + private final CommandExecutorForTest commandExecutor = new CommandExecutorForTest(); + private final GlobalPropertyManager globalPropertyManager = mock(GlobalPropertyManager.class); + private final JobManager jobManger = mock(JobManager.class); + private final UserManager userManager = mock(UserManager.class); + private final PrometheusConfigurator prometheusConfigurator = mock(PrometheusConfigurator.class); + private final Deployer deployer = mock(Deployer.class); + private Path temporaryYamlFile; + private final NetworkingUtils networkingUtils = mock(NetworkingUtils.class); + private final K8sClient k8sClient = mock(K8sClient.class); + private final CredentialsResolver credentialsResolver = new CredentialsResolver(k8sClient); + private final JenkinsApiClient jenkinsApiClient = mock(JenkinsApiClient.class); + private final ImagePullSecretCreator imagePullSecretCreator = mock(ImagePullSecretCreator.class); + + private final ScmManagerProviderMock scmManagerMock = new ScmManagerProviderMock(); + private final GitHandler gitHandler = new GitHandlerForTests(scmManagerMock); + + private RepositoryWorkspace repositoryWorkspace; + private DeploymentContext deploymentContext; + private File localTempDir; + + JenkinsTest() { + config = new Config(); + + ScmTenantSchema scm = new ScmTenantSchema(); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setUrlForJenkins("testUrlJenkins"); + scm.setScmManager(scmManager); + config.setScm(scm); + + Config.JenkinsSchema jenkins = new Config.JenkinsSchema(); + jenkins.setActive(true); + config.setJenkins(jenkins); + } + + @BeforeEach + void setup() { + // waitForInternalNodeIp -> waitForNode() + when(k8sClient.waitForNode()).thenReturn("node/" + expectedNodeName); + when(k8sClient.run(anyString(), anyString(), anyString(), anyMap(), any(String[].class))).thenReturn(""); + } + + @Test + void installsJenkins() throws GitAPIException, IOException { + Jenkins jenkins = createJenkins(); + + config.getJenkins().setUrl("http://jenkins"); + config.getJenkins().getHelm().setChart("jen-chart"); + config.getJenkins().getHelm().setRepoURL("https://jen-repo"); + config.getJenkins().getHelm().setVersion("4.8.1"); + config.getJenkins().setUsername("jenusr"); + config.getJenkins().setPassword("jenpw"); + config.getJenkins().setJenkinsImage("localhost:5000/proxy/jenkins-helm:custom"); + config.getJenkins().setInternalBashImage("bash:42"); + config.getJenkins().setInternalDockerClientVersion("23"); + + when(k8sClient.run(anyString(), anyString(), anyString(), anyMap(), any(String[].class))).thenReturn(""" + root:x:0: + daemon:x:1: + docker:x:42:me + me:x:1000:"""); + + install(jenkins); + + verify(deployer).deployFeature( + eq("https://jen-repo"), + eq("jenkins"), + eq("jen-chart"), + eq("4.8.1"), + eq("jenkins"), + eq("jenkins"), + eq(temporaryYamlFile), + eq(RepoType.HELM), + eq(true), + eq(deploymentContext), + eq(repositoryWorkspace) + ); + + verify(repositoryWorkspace).commitAndPushClusterResourcesChanges("Update jenkins GitOps resources"); + + verify(k8sClient).label("node", expectedNodeName, new Tuple<>("node", "jenkins")); + verify(k8sClient).labelRemove("node", "--all", "", "node"); + verify(k8sClient).createSecret( + "generic", + "jenkins-credentials", + "jenkins", + new Tuple<>("jenkins-admin-user", "jenusr"), + new Tuple<>("jenkins-admin-password", "jenpw") + ); + + Map actual = parseActualYaml(); + assertThat(actual.get("dockerClientVersion").toString()).isEqualTo("23"); + + Map controller = (Map) actual.get("controller"); + Map image = (Map) controller.get("image"); + assertThat(image.get("registry")).isEqualTo("localhost:5000"); + assertThat(image.get("repository")).isEqualTo("proxy/jenkins-helm"); + assertThat(image.get("tag")).isEqualTo("custom"); + assertThat(controller.get("installPlugins")).isEqualTo(false); + + assertThat(controller.get("jenkinsUrl")).isEqualTo("http://jenkins"); + assertThat(controller.get("serviceType")).isEqualTo("NodePort"); + + assertThat(controller.get("ingress")).isNull(); + + List> customInitContainers = (List>) controller.get( + "customInitContainers"); + assertThat(customInitContainers.get(0).get("image")).isEqualTo("bash:42"); + + Map agent = (Map) actual.get("agent"); + assertThat(agent.get("runAsUser")).isEqualTo(1000); + assertThat(agent.get("runAsGroup")).isEqualTo(42); + + ArgumentCaptor nameCaptor = ArgumentCaptor.forClass(String.class); + ArgumentCaptor overridesCaptor = ArgumentCaptor.forClass(Map.class); + verify(k8sClient).run( + nameCaptor.capture(), + anyString(), + eq(jenkins.getNamespace()), + overridesCaptor.capture(), + any(String[].class) + ); + assertThat(nameCaptor.getValue()).startsWith("tmp-docker-gid-grepper-"); + + Map spec = (Map) overridesCaptor.getValue().get("spec"); + List> containers = (List>) spec.get("containers"); + assertThat(containers.get(0).get("image").toString()).isEqualTo("bash:42"); + } + + @Test + void resolvesJenkinsCredentialsAtRuntimeWithoutMutatingConfig() throws GitAPIException, IOException { + config.getJenkins().setUsername("fallback-admin"); + config.getJenkins().setPassword("fallback-password"); + config.getJenkins().setCredentials( + new Credentials(null, null, "jenkins-source", "gop-job") + ); + config.getJenkins().setMetricsUsername("fallback-metrics"); + config.getJenkins().setMetricsPassword("fallback-metrics-password"); + config.getJenkins().setMetricsCredentials( + new Credentials(null, null, "jenkins-metrics-source", "gop-job") + ); + config.getJenkins().getOidc().setIssuerUrl("https://id.example.org"); + config.getJenkins().getOidc().setClientSecret("oidc-secret"); + + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))).thenAnswer(invocation -> { + Credentials reference = invocation.getArgument(0); + if ("jenkins-source".equals(reference.getSecretName())) { + return new Credentials("secret-admin", "secret-password"); + } + if ("jenkins-metrics-source".equals(reference.getSecretName())) { + return new Credentials("secret-metrics", "secret-metrics-password"); + } + throw new IllegalArgumentException("Unexpected Secret reference " + reference.getSecretName()); + }); + + install(createJenkins()); + + verify(k8sClient).createSecret( + "generic", + "jenkins-credentials", + "jenkins", + new Tuple<>("jenkins-admin-user", "secret-admin"), + new Tuple<>("jenkins-admin-password", "secret-password") + ); + verify(jenkinsApiClient).setRuntimeCredentials( + new ResolvedCredentials("secret-admin", "secret-password") + ); + verify(userManager).createUser("secret-metrics", "secret-metrics-password"); + verify(userManager).grantPermission("secret-metrics", UserManager.Permissions.METRICS_VIEW); + + Map env = getEnvAsMap(); + assertThat(env.get("JENKINS_USERNAME")).isEqualTo("secret-admin"); + assertThat(env.get("JENKINS_PASSWORD")).isEqualTo("secret-password"); + + assertThat(config.getJenkins().getUsername()).isEqualTo("fallback-admin"); + assertThat(config.getJenkins().getPassword()).isEqualTo("fallback-password"); + assertThat(config.getJenkins().getMetricsUsername()).isEqualTo("fallback-metrics"); + assertThat(config.getJenkins().getMetricsPassword()).isEqualTo("fallback-metrics-password"); + assertThat(config.getJenkins().getCredentials().getSecretName()).isEqualTo("jenkins-source"); + + String renderedValues = Files.readString(temporaryYamlFile); + assertThat(renderedValues).contains("${GOP_JENKINS_ADMIN_USER}"); + assertThat(renderedValues).contains("${GOP_JENKINS_ADMIN_PASSWORD}"); + assertThat(renderedValues).doesNotContain("secret-password", "secret-metrics-password"); + } + + @Test + void preparesJenkinsAppContentInClusterResourcesWorkspace() throws GitAPIException { + install(createJenkins()); + + assertThat(new File(localTempDir, "apps/jenkins")).exists(); + assertThat(new File(localTempDir, "apps/jenkins/templates")).doesNotExist(); + } + + @Test + void installsJenkinsWithoutDockerGid() throws GitAPIException, IOException { + when(k8sClient.run(anyString(), anyString(), anyString(), anyMap(), any(String[].class))).thenReturn(""" + root:x:0: + daemon:x:1: + me:x:1000:"""); + + install(createJenkins()); + + Map agent = (Map) parseActualYaml().get("agent"); + assertThat(agent.get("runAsUser")).isEqualTo("0"); + assertThat(agent.get("runAsGroup")).isEqualTo("133"); + } + + @Test + void installsOidcPluginBeforeJenkinsStartupWhenOidcIsConfigured() throws GitAPIException, IOException { + config.getJenkins().setUsername("admin"); + config.getJenkins().setPassword("admin"); + + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setClientId("jenkins"); + oidc.setClientSecret("jenkins-secret"); + oidc.setAdminGroupName("gop-admins"); + config.getJenkins().setOidc(oidc); + + install(createJenkins()); + + Map controller = (Map) parseActualYaml().get("controller"); + List installedPlugins = (List) controller.get("installPlugins"); + List installedPluginNames = installedPlugins.stream() + .map(plugin -> plugin.toString().split(":")[0]) + .collect(Collectors.toList()); + assertThat(installedPluginNames).containsExactly("oic-auth", "json-path-api", "matrix-auth"); + + Map jCasC = (Map) controller.get("JCasC"); + Map configScripts = (Map) jCasC.get("configScripts"); + String casc = (String) configScripts.get("oidc-auth"); + + assertThat(casc).contains("clientId: \"jenkins\""); + assertThat(casc).contains( + "wellKnownOpenIDConfigurationUrl: \"http://keycloak.local.gd/realms/gop/.well-known/openid-configuration\"" + ); + assertThat(casc).contains("escapeHatch:"); + assertThat(casc).contains("username: \"${GOP_JENKINS_ADMIN_USER}\""); + assertThat(casc).contains("secret: \"${GOP_JENKINS_ADMIN_PASSWORD}\""); + assertThat(casc).contains("group: \"gop-admins\""); + assertThat(casc).contains("globalMatrix:"); + assertThat(casc).contains("name: \"gop-admins\""); + } + + @Test + void usesDefaultJenkinsOidcScopesWhenScopesAreNull() throws GitAPIException, IOException { + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setClientId("jenkins"); + oidc.setClientSecret("jenkins-secret"); + oidc.setScopes(null); + config.getJenkins().setOidc(oidc); + + install(createJenkins()); + + Map controller = (Map) parseActualYaml().get("controller"); + Map jCasC = (Map) controller.get("JCasC"); + Map configScripts = (Map) jCasC.get("configScripts"); + String casc = (String) configScripts.get("oidc-auth"); + + assertThat(casc).contains("scopesOverride: \"openid profile email\""); + } + + @Test + void installsOnlyIfInternal() throws GitAPIException { + config.getJenkins().setInternal(false); + config.getRegistry().setCreateImagePullSecrets(true); + + install(createJenkins()); + + verify(deployer, never()).deployFeature( + anyString(), + anyString(), + anyString(), + anyString(), + anyString(), + anyString(), + any(Path.class), + any(), + anyBoolean(), + any(DeploymentContext.class), + any(RepositoryWorkspace.class) + ); + + verify(repositoryWorkspace, never()).commitAndPushClusterResourcesChanges(anyString()); + + verify(k8sClient, never()).createNamespace(any()); + verify(k8sClient, never()).createImagePullSecret( + anyString(), + anyString(), + anyString(), + anyString(), + anyString() + ); + + assertThat(temporaryYamlFile).isNull(); + } + + @Test + void additionalHelmValuesAreMergedWithDefaultValues() throws GitAPIException, IOException { + config.getJenkins().getHelm().setValues(Map.of( + "controller", + Map.of("nodePort", 42) + )); + + install(createJenkins()); + + Map controller = (Map) parseActualYaml().get("controller"); + assertThat(controller.get("nodePort")).isEqualTo(42); + } + + @Test + void enablesIngressWhenBaseUrlIsSet() throws GitAPIException, IOException { + config.getJenkins().setIngress("jenkins.localhost"); + config.getApplication().setBaseUrl("someBaseUrl"); + + install(createJenkins()); + + Map controller = (Map) parseActualYaml().get("controller"); + Map ingress = (Map) controller.get("ingress"); + assertThat(ingress.get("enabled")).isEqualTo(true); + assertThat(ingress.get("hostName")).isEqualTo("jenkins.localhost"); + } + + @Test + void mapsConfigProperly() throws GitAPIException { + config.getApplication().setTrace(true); + config.getFeatures().getArgocd().setActive(true); + config.getScm().getScmManager().setUrl("http://scmm.scm-manager.svc.cluster.local/scm"); + config.getScm().getScmManager().setUsername("scmm-usr"); + config.getScm().getScmManager().setPassword("scmm-pw"); + config.getApplication().setNamePrefix("my-prefix-"); + config.getApplication().setNamePrefixForEnvVars("MY_PREFIX_"); + config.getRegistry().setUrl("reg-url"); + config.getRegistry().setPath("reg-path"); + config.getRegistry().setUsername("reg-usr"); + config.getRegistry().setPassword("reg-pw"); + config.getRegistry().setProxyUrl("reg-proxy-url"); + config.getRegistry().setProxyPath("reg-proxy-path"); + config.getRegistry().setProxyUsername("reg-proxy-usr"); + config.getRegistry().setProxyPassword("reg-proxy-pw"); + config.getJenkins().setInternal(false); + config.getJenkins().getHelm().setVersion("4.8.1"); + config.getJenkins().setUsername("jenusr"); + config.getJenkins().setPassword("jenpw"); + config.getJenkins().setUrl("http://jenkins"); + config.getJenkins().setMetricsUsername("metrics-usr"); + config.getJenkins().setMetricsPassword("metrics-pw"); + config.getJenkins().setSkipPlugins(true); + config.getJenkins().setSkipRestart(true); + + install(createJenkins()); + + Map env = getEnvAsMap(); + assertThat(commandExecutor.getActualCommands().get(0)) + .isEqualTo(System.getProperty("user.dir") + "/scripts/jenkins/init-jenkins.sh"); + + assertThat(env.get("TRACE")).isEqualTo("true"); + assertThat(env.get("INTERNAL_JENKINS")).isEqualTo("false"); + assertThat(env.get("JENKINS_HELM_CHART_VERSION")).isEqualTo("4.8.1"); + assertThat(env.get("JENKINS_URL")).isEqualTo("http://jenkins"); + assertThat(env.get("JENKINS_USERNAME")).isEqualTo("jenusr"); + assertThat(env.get("JENKINS_PASSWORD")).isEqualTo("jenpw"); + assertThat(env.get("JENKINS_USERNAME")).isEqualTo("jenusr"); + assertThat(env.get("NAME_PREFIX")).isEqualTo("my-prefix-"); + assertThat(env.get("INSECURE")).isEqualTo("false"); + + assertThat(env.get("SCM_URL")).isEqualTo("http://scmm.scm-manager.svc.cluster.local/scm"); + assertThat(env.get("SCM_PASSWORD")).isEqualTo(scmManagerMock.getCredentials().getPassword()); + assertThat(env.get("INSTALL_ARGOCD")).isEqualTo("true"); + + assertThat(env.get("SKIP_PLUGINS")).isEqualTo("true"); + assertThat(env.get("SKIP_RESTART")).isEqualTo("true"); + + verify(globalPropertyManager).setGlobalProperty( + "MY_PREFIX_SCM_URL", + "http://scmm.scm-manager.svc.cluster.local/scm" + ); + verify(globalPropertyManager).setGlobalProperty("MY_PREFIX_K8S_VERSION", Config.K8S_VERSION); + + verify(globalPropertyManager).setGlobalProperty("MY_PREFIX_REGISTRY_URL", "reg-url"); + verify(globalPropertyManager).setGlobalProperty("MY_PREFIX_REGISTRY_PATH", "reg-path"); + verify(globalPropertyManager, never()).setGlobalProperty(eq("MY_PREFIX_REGISTRY_PROXY_URL"), anyString()); + verify(globalPropertyManager, never()).setGlobalProperty(eq("MY_PREFIX_REGISTRY_PROXY_PATH"), anyString()); + verify(globalPropertyManager, never()).setGlobalProperty(eq("MAVEN_CENTRAL_MIRROR"), anyString()); + + verify(userManager).createUser("metrics-usr", "metrics-pw"); + verify(userManager).grantPermission("metrics-usr", UserManager.Permissions.METRICS_VIEW); + } + + @Test + void usesRuntimeScmCredentialsForJenkinsJob() throws GitAPIException { + config.getApplication().setNamePrefix("test-"); + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + config.getScm().getScmManager().setPassword("config-scm-password"); + scmManagerMock.setCredentials(new Credentials("runtime-scm-user", "runtime-scm-password")); + + Jenkins jenkins = createJenkins(); + install(jenkins); + jenkins.createJenkinsjob("namespace", "repo"); + + verify(jobManger).createCredential( + "test-repo", + "scm-user", + "test-gitops", + "runtime-scm-password", + "credentials for accessing scm-manager" + ); + } + + @Test + void usesRuntimeRegistryCredentialsForJenkinsJob() throws GitAPIException { + config.getApplication().setNamePrefix("test-"); + config.getRegistry().setUsername("fallback-registry-user"); + config.getRegistry().setPassword("fallback-registry-password"); + config.getRegistry().setCredentials( + new Credentials(null, null, "registry-credentials", "gop-job") + ); + config.getRegistry().setTwoRegistries(true); + config.getRegistry().setProxyUsername("fallback-proxy-user"); + config.getRegistry().setProxyPassword("fallback-proxy-password"); + config.getRegistry().setProxyCredentials( + new Credentials(null, null, "registry-proxy-credentials", "gop-job") + ); + when(k8sClient.getCredentialsFromSecret(any(Credentials.class))).thenAnswer(invocation -> { + Credentials reference = invocation.getArgument(0); + if ("registry-credentials".equals(reference.getSecretName())) { + return new Credentials("runtime-registry-user", "runtime-registry-password"); + } + return new Credentials("runtime-proxy-user", "runtime-proxy-password"); + }); + + Jenkins jenkins = createJenkins(); + install(jenkins); + jenkins.createJenkinsjob("namespace", "repo"); + + verify(jobManger).createCredential( + "test-repo", + "registry-user", + "runtime-registry-user", + "runtime-registry-password", + "credentials for accessing the docker-registry for writing images built on jenkins" + ); + verify(jobManger).createCredential( + "test-repo", + "registry-proxy-user", + "runtime-proxy-user", + "runtime-proxy-password", + "credentials for accessing the docker-registry that contains 3rd party or base images" + ); + assertThat(config.getRegistry().getPassword()).isEqualTo("fallback-registry-password"); + assertThat(config.getRegistry().getProxyPassword()).isEqualTo("fallback-proxy-password"); + } + + @Test + void doesNotConfigurePrometheusWhenExternalJenkins() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(true); + config.getJenkins().setInternal(false); + + install(createJenkins()); + + verify(prometheusConfigurator, never()).enableAuthentication(); + } + + @Test + void doesNotConfigurePrometheusWhenMonitoringOff() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(false); + config.getJenkins().setInternal(true); + + install(createJenkins()); + + verify(prometheusConfigurator, never()).enableAuthentication(); + } + + @Test + void configuresPrometheus() throws GitAPIException { + config.getFeatures().getMonitoring().setActive(true); + config.getJenkins().setInternal(true); + + install(createJenkins()); + + verify(prometheusConfigurator).enableAuthentication(); + } + + @Test + void usesK8sServiceNameIfRunningAsK8sPod() throws GitAPIException { + config.getJenkins().setInternal(true); + config.getApplication().setRunningInsideK8s(true); + + install(createJenkins()); + + assertThat(config.getJenkins().getUrl()).isEqualTo("http://jenkins.jenkins.svc.cluster.local:80"); + } + + @Test + void usesLocalIpAndNodePortWhenOutsideOfK8s() throws GitAPIException { + config.getJenkins().setInternal(true); + config.getApplication().setRunningInsideK8s(false); + + when(networkingUtils.findClusterBindAddress()).thenReturn("192.168.16.2"); + when(k8sClient.waitForNodePort(anyString(), anyString())).thenReturn("42"); + + install(createJenkins()); + + assertThat(config.getJenkins().getUrl()).endsWith("192.168.16.2:42"); + } + + @Test + void handlesTwoRegistries() throws GitAPIException { + config.getRegistry().setTwoRegistries(true); + config.getApplication().setNamePrefix("my-prefix-"); + config.getApplication().setNamePrefixForEnvVars("MY_PREFIX_"); + + config.getRegistry().setUrl("reg-url"); + config.getRegistry().setPath("reg-path"); + config.getRegistry().setUsername("reg-usr"); + config.getRegistry().setPassword("reg-pw"); + config.getRegistry().setProxyUrl("reg-proxy-url"); + config.getRegistry().setProxyPath("reg-proxy-path"); + config.getRegistry().setProxyUsername("reg-proxy-usr"); + config.getRegistry().setProxyPassword("reg-proxy-pw"); + + install(createJenkins()); + + verify(globalPropertyManager).setGlobalProperty("MY_PREFIX_REGISTRY_PROXY_URL", "reg-proxy-url"); + verify(globalPropertyManager).setGlobalProperty("MY_PREFIX_REGISTRY_PROXY_PATH", "reg-proxy-path"); + + verify(globalPropertyManager).setGlobalProperty(eq("MY_PREFIX_REGISTRY_URL"), anyString()); + verify(globalPropertyManager).setGlobalProperty(eq("MY_PREFIX_REGISTRY_PATH"), anyString()); + } + + @Test + void doesNotCreateMetricsUserIfSecurityRealmDoesNotSupportLocalUserCreation() throws GitAPIException { + config.getApplication().setNamePrefixForEnvVars("MY_PREFIX_"); + when(userManager.isUsingSecurityRealmWithoutLocalUserCreation()).thenReturn(true); + + install(createJenkins()); + + verify(userManager, never()).createUser(anyString(), anyString()); + } + + @Test + void globalPropertyIsSetForAdditionalEnvs() throws GitAPIException { + config.getJenkins().setAdditionalEnvs(Map.of("ADDITIONAL_DOCKER_RUN_ARGS", "-u0:0")); + + install(createJenkins()); + + verify(globalPropertyManager).setGlobalProperty(eq("ADDITIONAL_DOCKER_RUN_ARGS"), eq("-u0:0")); + } + + @Test + void doesNotCreateUserIfCasSecurityRealmIsUsed() throws GitAPIException { + config.getFeatures().getArgocd().setActive(false); + + install(createJenkins()); + + verify(jobManger, never()).createCredential(anyString(), anyString(), anyString(), anyString(), anyString()); + verify(jobManger, never()).startJob(anyString()); + } + + @Test + void properlyHandlesNullValues() throws GitAPIException { + config.getApplication().setBaseUrl(null); + + install(createJenkins()); + + Map env = getEnvAsMap(); + assertThat(env.get("BASE_URL")).isNotEqualTo("null"); + } + + @Test + void setsMavenMirror() throws GitAPIException { + config.getRegistry().setUrl("some value"); + config.getJenkins().setMavenCentralMirror("http://test"); + config.getApplication().setNamePrefixForEnvVars("MY_PREFIX_"); + + install(createJenkins()); + + verify(globalPropertyManager).setGlobalProperty( + eq("MY_PREFIX_MAVEN_CENTRAL_MIRROR"), + eq("http://test") + ); + } + + protected Map getEnvAsMap() { + Map env = new LinkedHashMap<>(); + for (String entry : commandExecutor.getEnvironment()) { + String[] parts = entry.split("="); + env.put(parts[0], parts.length > 1 ? parts[1] : null); + } + return env; + } + + private Jenkins createJenkins() throws GitAPIException { + when(networkingUtils.createUrl(anyString(), anyString(), anyString())).thenCallRealMethod(); + when(networkingUtils.createUrl(anyString(), anyString())).thenCallRealMethod(); + + FileSystemUtils fileSystemUtils = new FileSystemUtils() { + @Override + public Path writeTempFile(Map mergeMap) { + Path ret = super.writeTempFile(mergeMap); + temporaryYamlFile = Path.of(ret.toString().replace(".ftl", "")); + // Path after template invocation + return ret; + } + }; + + TestGitRepoFactory repoFactory = new TestGitRepoFactory(config, new FileSystemUtils()) { + @Override + public GitRepo create(String repoTarget, GitProvider gitProvider) { + GitRepo repo = super.create(repoTarget, gitProvider); + localTempDir = new File(repo.getAbsoluteLocalRepoTmpDir()); + return repo; + } + }; + + GitRepo clusterResourcesRepo = repoFactory.create("argocd/cluster-resources", scmManagerMock); + + repositoryWorkspace = spy(new RepositoryWorkspace(clusterResourcesRepo)); + doNothing().when(repositoryWorkspace).commitAndPushClusterResourcesChanges(anyString()); + + AirGappedUtils airGappedUtils = new AirGappedUtils(null, fileSystemUtils, null, gitHandler); + + return new Jenkins( + commandExecutor, + fileSystemUtils, + globalPropertyManager, + jobManger, + userManager, + prometheusConfigurator, + deployer, + k8sClient, + networkingUtils, + airGappedUtils, + gitHandler, + imagePullSecretCreator, + new JenkinsToolConfigMapper(config), + new JenkinsConfigUpdater(config), + credentialsResolver, + jenkinsApiClient + ); + } + + private boolean install(Jenkins jenkins) { + deploymentContext = new ContextBuilder(config).build(); + return jenkins.execute(deploymentContext, repositoryWorkspace); + } + + private Map parseActualYaml() throws IOException { + return YAML_MAPPER.readValue(temporaryYamlFile.toFile(), YAML_MAP_TYPE); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/JenkinsToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/core/JenkinsToolConfigMapperTest.java new file mode 100644 index 000000000..6b0de876c --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/JenkinsToolConfigMapperTest.java @@ -0,0 +1,222 @@ +package com.cloudogu.gitops.tools.core; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class JenkinsToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setNamePrefixForEnvVars("TEST_"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getApplication().setRunningInsideK8s(true); + config.getApplication().setTrace(true); + config.getApplication().setInsecure(true); + config.getApplication().setBaseUrl("example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setPath("images"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setPassword("registry-password"); + config.getRegistry().setTwoRegistries(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setProxyPath("proxy-images"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setCreateImagePullSecrets(true); + config.getJenkins().setActive(true); + config.getJenkins().setInternal(true); + config.getJenkins().setNamespace("automation"); + config.getJenkins().setUrl("https://jenkins.example.org"); + config.getJenkins().setUsername("jenkins-user"); + config.getJenkins().setPassword("jenkins-password"); + config.getJenkins().setCredentials( + new Credentials(null, null, "jenkins-secret", "gop-job", "admin-user", "admin-password") + ); + config.getJenkins().setMetricsUsername("metrics-user"); + config.getJenkins().setMetricsPassword("metrics-password"); + config.getJenkins().setMetricsCredentials( + new Credentials(null, null, "jenkins-metrics-secret", "gop-job", "metrics-user", "metrics-password") + ); + config.getJenkins().setSkipRestart(true); + config.getJenkins().setSkipPlugins(true); + config.getJenkins().setMavenCentralMirror("https://maven.example.org"); + config.getJenkins().setInternalBashImage("bash:custom"); + config.getJenkins().setInternalDockerClientVersion("28.0.0"); + config.getJenkins().setJenkinsImage("jenkins:custom"); + config.getJenkins().setIngress("jenkins-ingress.example.org"); + config.getJenkins().setAdditionalEnvs(Map.of("FIRST", "one", "SECOND", "two")); + config.getJenkins().getOidc().setIssuerUrl("https://id.example.org"); + config.getJenkins().getOidc().setClientId("jenkins-client"); + config.getJenkins().getOidc().setClientSecret("jenkins-client-secret"); + config.getJenkins().getHelm().setRepoURL("https://jenkins-chart.example.org"); + config.getJenkins().getHelm().setChart("jenkins-chart"); + config.getJenkins().getHelm().setVersion("7.8.9"); + config.getJenkins().getHelm().setValues(Map.of("controller", Map.of("replicas", 2))); + config.getFeatures().getArgocd().setActive(true); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setIssuer("production-issuer"); + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setPassword("scmm-password"); + config.getScm().setScmManager(scmManager); + ScmTenantSchema.GitlabTenantConfig gitlab = new ScmTenantSchema.GitlabTenantConfig(); + gitlab.setUsername("gitlab-user"); + gitlab.setPassword("gitlab-password"); + config.getScm().setGitlab(gitlab); + + JenkinsToolConfig actual = new JenkinsToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(JenkinsToolConfig.builder() + .active(true) + .internal(true) + .namespace("test-automation") + .application(JenkinsToolConfig.Application.builder() + .namePrefix("test-") + .environmentPrefix( + "TEST_") + .runningInsideK8s(true) + .trace(true) + .insecure(true) + .build()) + .server(JenkinsToolConfig.Server.builder() + .url("https://jenkins.example.org") + .username("jenkins-user") + .password("jenkins-password") + .credentials(new CredentialsReference( + "jenkins-secret", + "gop-job", + "admin-user", + "admin-password" + )) + .metricsUsername("metrics-user") + .metricsPassword( + "metrics-password") + .metricsCredentials(new CredentialsReference( + "jenkins-metrics-secret", + "gop-job", + "metrics-user", + "metrics-password" + )) + .skipRestart(true) + .skipPlugins(true) + .mavenCentralMirror( + "https://maven.example.org") + .internalBashImage("bash:custom") + .oidcConfigured(true) + .additionalEnvironments(Map.of( + "FIRST", + "one", + "SECOND", + "two" + )) + .build()) + .scm(JenkinsToolConfig.Scm.builder() + .providerType(ScmProviderType.SCM_MANAGER) + .build()) + .registry(JenkinsToolConfig.Registry.builder() + .url("registry.example.org") + .path("images") + .username("registry-user") + .password("registry-password") + .twoRegistries(true) + .proxyUrl("proxy.example.org") + .proxyPath("proxy-images") + .proxyUsername("proxy-user") + .proxyPassword( + "proxy-password") + .build()) + .argocdActive(true) + .monitoringActive(true) + .kubernetesVersion(Config.K8S_VERSION) + .helm(HelmChartConfig.builder() + .repoURL("https://jenkins-chart.example.org") + .chart("jenkins-chart") + .version("7.8.9") + .values(Map.of( + "controller", + Map.of("replicas", 2) + )) + .localHelmChartFolder("/charts") + .build()) + .imagePullSecret(ImagePullSecretConfig.builder() + .create(true) + .proxyUrl( + "proxy.example.org") + .url("registry.example.org") + .proxyUsername("proxy-user") + .readOnlyUsername( + "read-only-user") + .username("registry-user") + .proxyPassword( + "proxy-password") + .readOnlyPassword( + "read-only-password") + .password( + "registry-password") + .build()) + .templateConfig(Map.of( + "application", Map.of("baseUrl", "example.org"), + "features", Map.of( + "certManager", Map.of( + "active", true, + "issuer", "production-issuer" + ) + ), + "jenkins", Map.of( + "helm", Map.of("version", "7.8.9"), + "ingress", "jenkins-ingress.example.org", + "internalBashImage", "bash:custom", + "internalDockerClientVersion", "28.0.0", + "jenkinsImage", "jenkins:custom", + "oidc", Map.of( + "providerName", "Keycloak", + "issuerUrl", "https://id.example.org", + "clientId", "jenkins-client", + "clientSecret", "jenkins-client-secret", + "scopes", List.of("openid", "profile", "email"), + "adminGroupName", "", + "enabled", true + ), + "url", "https://jenkins.example.org" + ), + "registry", Map.of("createImagePullSecrets", true) + )) + .build()); + } + + @Test + void doesNotExposeANamespaceForAnExternalJenkins() { + Config config = new Config(); + config.getJenkins().setInternal(false); + + JenkinsToolConfig actual = new JenkinsToolConfigMapper(config).map(context()); + + assertThat(actual.namespace()).isNull(); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.SINGLE_TENANT, + DeploymentContext.ScmManagerDeploymentMode.EXTERNAL, + false, + DeploymentContext.ClusterDistribution.KUBERNETES + ); + } +} \ No newline at end of file diff --git a/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDConfigurationTest.java b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDConfigurationTest.java new file mode 100644 index 000000000..d8881ada1 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDConfigurationTest.java @@ -0,0 +1,1812 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.CommandExecutorForTest; +import com.cloudogu.gitops.utils.K8sClientForTest; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import io.fabric8.kubernetes.api.model.NamespaceBuilder; +import io.fabric8.kubernetes.api.model.Secret; +import io.fabric8.kubernetes.api.model.SecretBuilder; +import io.fabric8.kubernetes.api.model.apiextensions.v1.CustomResourceDefinition; +import io.fabric8.kubernetes.api.model.apiextensions.v1.CustomResourceDefinitionBuilder; +import io.fabric8.kubernetes.client.KubernetesClient; +import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.security.crypto.bcrypt.BCrypt; + +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Field; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Base64; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; + +@EnableKubernetesMockClient(crud = true) +class ArgoCDConfigurationTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final TypeReference>> YAML_MAP_LIST_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + private final Config config = Config.fromMap(map( + "application", map( + "openshift", false, + "insecure", false, + "password", "123", + "username", "something", + "namePrefix", "", + "namePrefixForEnvVars", "", + "gitName", "Cloudogu", + "gitEmail", "hello@cloudogu.com", + "namespaces", map( + "dedicatedNamespaces", List.of("argocd", "monitoring", "traefik", "secrets"), + "tenantNamespaces", List.of("example-apps-staging", "example-apps-production") + ) + ), + "scm", map( + "scmManager", map("internal", true), + "gitlab", map("url", "") + ), + "multiTenant", map( + "scmManager", map("url", ""), + "gitlab", map("url", ""), + "useDedicatedInstance", false, + "centralArgocdNamespace", "argocd" + ), + "content", map( + "repos", List.of( + map( + "url", "https://github.com/cloudogu/gitops-build-lib", + "target", "3rd-party-dependencies/gitops-build-lib", + "overwriteMode", "RESET" + ), + map( + "url", "https://github.com/cloudogu/ces-build-lib", + "target", "3rd-party-dependencies/ces-build-lib", + "overwriteMode", "RESET" + ), + map( + "url", "https://github.com/cloudogu/spring-boot-helm-chart", + "target", "3rd-party-dependencies/spring-boot-helm-chart", + "overwriteMode", "RESET" + ), + map( + "url", "https://github.com/cloudogu/spring-petclinic", + "target", "argocd/petclinic-plain", + "ref", "feature/gitops_ready", + "targetRef", "main", + "overwriteMode", "UPGRADE", + "createJenkinsJob", true + ), + map( + "url", "https://github.com/cloudogu/spring-petclinic", + "target", "argocd/petclinic-helm", + "ref", "feature/gitops_ready", + "targetRef", "main", + "overwriteMode", "UPGRADE", + "createJenkinsJob", true + ), + map( + "url", "https://github.com/cloudogu/gitops-playground", + "path", "example-apps-via-content-loader/", + "ref", "main", + "templating", true, + "type", "FOLDER_BASED", + "overwriteMode", "UPGRADE" + ) + ), + "namespaces", List.of("example-apps-production", "example-apps-staging"), + "variables", map( + "petclinic", map("baseDomain", "petclinic.localhost"), + "images", map( + "kubectl", "alpine/kubectl:1.35.0", + "helm", "ghcr.io/cloudogu/helm:4.2.1-1", + "kubeval", "ghcr.io/cloudogu/helm:4.2.1-1", + "helmKubeval", "ghcr.io/cloudogu/helm:4.2.1-1", + "yamllint", "cytopia/yamllint:1.25-0.7", + "petclinic", "eclipse-temurin:17-jre-alpine", + "maven", "" + ) + ) + ), + "features", map( + "argocd", map( + "operator", false, + "active", true, + "configOnly", true, + "emailFrom", "argocd@example.org", + "emailToUser", "app-team@example.org", + "emailToAdmin", "infra@example.org", + "resourceInclusionsCluster", "" + ), + "monitoring", map( + "active", true, + "helm", map("chart", "kube-prometheus-stack", "version", "42.0.3") + ), + "ingress", map("active", true), + "secrets", map("active", true) + ) + )); + + KubernetesClient client; + private K8sClient k8sClient; + private final CommandExecutorForTest helmCommands = new CommandExecutorForTest(); + private ArgoCDRepoLayout clusterResourcesRepoLayout; + + @BeforeEach + void setupKubernetesClient() { + ArgoCDK8sClientForTest clientForTest = new ArgoCDK8sClientForTest(); + clientForTest.configure(client); + k8sClient = spy(clientForTest); + + // no need to wait in tests, we stub! + doNothing().when(k8sClient).waitForResourcePhase( + any(String.class), + any(String.class), + any(String.class), + any(String.class) + ); + } + + @Test + void resolvesAdminPasswordFromApplicationSecretWithoutMutatingConfig() { + Credentials reference = new Credentials(); + reference.setSecretName("argocd-credentials"); + reference.setSecretNamespace("gop-job"); + config.getApplication().setCredentials(reference); + config.getApplication().setUsername("fallback-user"); + config.getApplication().setPassword("fallback-password"); + createSecretIfMissing( + "argocd-credentials", + "gop-job", + Map.of("username", encode("secret-user"), "password", encode("secret-password")) + ); + + ArgoCDForTest argocd = (ArgoCDForTest) createArgoCD(); + + execute(argocd); + + Secret argocdSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-secret") + .get(); + String patchedPasswordHash = decodedSecretValue(argocdSecret, "admin.password"); + + assertThat(BCrypt.checkpw("secret-password", patchedPasswordHash)).isTrue(); + assertThat(config.getApplication().getUsername()).isEqualTo("fallback-user"); + assertThat(config.getApplication().getPassword()).isEqualTo("fallback-password"); + assertThat(reference.getUsername()).isNull(); + assertThat(reference.getPassword()).isNull(); + } + + @Test + void installsArgoCd() throws IOException { + ArgoCDForTest argocd = (ArgoCDForTest) createArgoCD(); + + execute(argocd); + + GitRepo clusterResourcesRepo = argocd.getClusterResourcesRepo(); + clusterResourcesRepoLayout = argocd.getClusterRepoLayout(); + + assertThat(client.namespaces().withName("argocd").get()).isNotNull(); + + List filesWithInternalScmManager = findFilesContaining( + new File(clusterResourcesRepoLayout.rootDir()), + clusterResourcesRepo.getGitProvider().getUrl() + ); + assertThat(filesWithInternalScmManager).isNotEmpty(); + + Map valuesYaml = parseActualYaml(actualHelmValuesFile()); + assertThat(value(valuesYaml, "argo-cd", "server", "service", "type")).isEqualTo("ClusterIP"); + assertThat(value(valuesYaml, "argo-cd", "notifications", "argocdUrl")).isNull(); + assertThat(value(valuesYaml, "argo-cd", "crds")).isNull(); + assertThat(valuesYaml.get("global")).isNull(); + + Secret repoCredentialsSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-repo-creds-scm") + .get(); + + assertThat(repoCredentialsSecret).isNotNull(); + assertThat(repoCredentialsSecret.getMetadata().getLabels().get("argocd.argoproj.io/secret-type")) + .isEqualTo("repo-creds"); + + assertThat(helmCommands.getActualCommands().get(0).trim()) + .isEqualTo("helm repo add argo https://argoproj.github.io/argo-helm"); + assertThat(helmCommands.getActualCommands().get(1).trim()) + .isEqualTo("helm dependency build " + clusterResourcesRepoLayout.helmDir()); + assertThat(helmCommands.getActualCommands().get(2).trim()) + .isEqualTo("helm upgrade -i argocd " + clusterResourcesRepoLayout.helmDir() + + " --create-namespace --namespace argocd"); + + Secret argocdSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-secret") + .get(); + + assertThat(argocdSecret).isNotNull(); + + String patchedPasswordHash = decodedSecretValue(argocdSecret, "admin.password"); + assertThat(BCrypt.checkpw(config.getApplication().getPassword(), patchedPasswordHash)) + .as("Password hash mismatch") + .isTrue(); + + assertThat(client.secrets() + .inNamespace("argocd") + .withLabels(Map.of("owner", "helm", "name", "argocd")) + .list() + .getItems()).isEmpty(); + + assertThat(Path.of(clusterResourcesRepoLayout.operatorConfigFile())).doesNotExist(); + assertThat(Path.of(clusterResourcesRepoLayout.operatorRbacDir())).doesNotExist(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + assertThat(sourceRepos) + .contains("https://prometheus-community.github.io/helm-charts") + .doesNotContain( + "http://scmm-scm-manager.default.svc.cluster.local/scm/repo/3rd-party-dependencies/" + + "kube-prometheus-stack" + ); + + Map argocdYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.applicationsDir(), "argocd.yaml").toString() + ); + assertThat(value(argocdYaml, "spec", "source", "directory")).isNull(); + assertThat((String) value(argocdYaml, "spec", "source", "path")) + .isIn("apps/argocd/argocd", "apps/argocd/argocd/"); + } + + @Test + void publishesArgoCdRepositoryContentThroughRepositoryWorkspace() throws GitAPIException { + ArgoCDForTest argocd = (ArgoCDForTest) createArgoCD(); + + execute(argocd); + + verify(argocd.repositoryWorkspace.getClusterResourcesRepository()) + .commitAndPush("Update ArgoCD repository content"); + } + + @Test + void usesRepositoryWorkspaceForClusterResourcesRepositoryContent() throws IOException { + ArgoCDForTest argocd = (ArgoCDForTest) createArgoCD(); + + execute(argocd); + + assertThat(argocd.repositoryWorkspace.getClusterResourcesRepository()) + .isSameAs(argocd.clusterResourcesRepo); + + clusterResourcesRepoLayout = argocd.getClusterRepoLayout(); + + assertThat(new File(clusterResourcesRepoLayout.rootDir()).getCanonicalFile()) + .isEqualTo(new File(argocd.clusterResourcesRepo.getAbsoluteLocalRepoTmpDir()).getCanonicalFile()); + } + + @Test + void configuresArgoCdUrlAndAdditionalRedirectUrls() throws IOException { + config.getFeatures().getArgocd().setUrl("https://argocd.localhost"); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map cm = mapValue( + parseActualYaml(actualHelmValuesFile()), + "argo-cd", + "configs", + "cm" + ); + assertThat(cm.get("url")).isEqualTo("https://argocd.localhost"); + assertThat((String) cm.get("additionalUrls")) + .contains("http://argocd.localhost", "https://argocd.localhost"); + } + + @Test + void configuresArgoCdOidcFromStructuredConfig() throws IOException { + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setClientId("argocd"); + oidc.setClientSecret("argocd-secret"); + oidc.setAdminGroupName("gop-admins"); + config.getFeatures().getArgocd().setOidc(oidc); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map valuesYaml = mapValue(parseActualYaml(actualHelmValuesFile()), "argo-cd", "configs"); + Map oidcConfig = parseYaml((String) value(valuesYaml, "cm", "oidc.config")); + assertThat(oidcConfig.get("issuer")).isEqualTo("http://keycloak.local.gd/realms/gop"); + assertThat(oidcConfig.get("clientID")).isEqualTo("argocd"); + assertThat((String) value(valuesYaml, "rbac", "policy.csv")).contains("g, gop-admins, role:admin"); + assertThat(value(valuesYaml, "rbac", "scopes")).isEqualTo("[groups]"); + } + + @Test + void doesNotIncludeOidcConfigurationWhenArgoCdOidcConfigIsNull() throws IOException { + config.getFeatures().getArgocd().setOidc(null); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map valuesYaml = mapValue(parseActualYaml(actualHelmValuesFile()), "argo-cd", "configs"); + assertThat(value(valuesYaml, "cm", "oidc.config")).isNull(); + assertThat(valuesYaml.get("rbac")).isNull(); + } + + @Test + void usesDefaultScopesWhenArgoCdOidcScopesAreNull() throws IOException { + Config.OidcSchema oidc = new Config.OidcSchema(); + oidc.setIssuerUrl("http://keycloak.local.gd/realms/gop"); + oidc.setClientId("argocd"); + oidc.setClientSecret("argocd-secret"); + oidc.setScopes(null); + config.getFeatures().getArgocd().setOidc(oidc); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map valuesYaml = mapValue(parseActualYaml(actualHelmValuesFile()), "argo-cd", "configs"); + Map oidcConfig = parseYaml((String) value(valuesYaml, "cm", "oidc.config")); + assertThat((List) oidcConfig.get("requestedScopes")) + .containsExactly("openid", "profile", "email"); + } + + @Test + void doesNotIncludeMailConfigurationWhenMailServerIsDisabled() throws IOException { + config.getFeatures().getMail().setActive(false); + + Map valuesYaml = executeAndReadHelmValues(); + + assertThat(value(valuesYaml, "argo-cd", "notifications", "enabled")).isEqualTo(false); + assertThat(value(valuesYaml, "argo-cd", "notifications", "notifiers")).isNull(); + } + + @Test + void includesMailConfigurationWhenMailServerIsEnabled() throws IOException { + config.getFeatures().getMail().setActive(true); + + Map valuesYaml = executeAndReadHelmValues(); + + assertThat(value(valuesYaml, "argo-cd", "notifications", "enabled")).isEqualTo(true); + assertThat(value(valuesYaml, "argo-cd", "notifications", "notifiers")).isNotNull(); + } + + @Test + void includesConfiguredEmailAddresses() throws IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getArgocd().setEmailFrom("argocd@example.com"); + config.getFeatures().getArgocd().setEmailToUser("app-team@example.com"); + config.getFeatures().getArgocd().setEmailToAdmin("argocd@example.com"); + + Map valuesYaml = executeAndReadHelmValues(); + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + Map argocdYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.applicationsDir(), "argocd.yaml").toString() + ); + Map defaultYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "default.yaml").toString() + ); + Map serviceEmail = parseYaml( + (String) value(valuesYaml, "argo-cd", "notifications", "notifiers", "service.email") + ); + + assertThat(serviceEmail.get("from")).isEqualTo("argocd@example.com"); + assertThat(value(clusterResourcesYaml, "metadata", "annotations", + "notifications.argoproj.io/subscribe.email")).isEqualTo("argocd@example.com"); + assertThat(value(argocdYaml, "metadata", "annotations", + "notifications.argoproj.io/subscribe.on-sync-status-unknown.email")) + .isEqualTo("argocd@example.com"); + assertThat(value(defaultYaml, "metadata", "annotations", + "notifications.argoproj.io/subscribe.email")).isEqualTo("argocd@example.com"); + } + + @Test + void usesDefaultEmailAddressesWhenNoneAreConfigured() throws IOException { + config.getFeatures().getMail().setActive(true); + + Map valuesYaml = executeAndReadHelmValues(); + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + Map argocdYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.applicationsDir(), "argocd.yaml").toString() + ); + Map defaultYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "default.yaml").toString() + ); + Map serviceEmail = parseYaml( + (String) value(valuesYaml, "argo-cd", "notifications", "notifiers", "service.email") + ); + + assertThat(serviceEmail.get("from")).isEqualTo("argocd@example.org"); + assertThat(value(clusterResourcesYaml, "metadata", "annotations", + "notifications.argoproj.io/subscribe.email")).isEqualTo("infra@example.org"); + assertThat(value(argocdYaml, "metadata", "annotations", + "notifications.argoproj.io/subscribe.on-sync-status-unknown.email")) + .isEqualTo("infra@example.org"); + assertThat(value(defaultYaml, "metadata", "annotations", + "notifications.argoproj.io/subscribe.email")).isEqualTo("infra@example.org"); + } + + @Test + void configuresExternalMailServer() throws IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpPort(1010110); + config.getFeatures().getMail().setSmtpUser("argo@example.com"); + config.getFeatures().getMail().setSmtpPassword("1101:ABCabc&/+*~"); + + Map valuesYaml = executeAndReadHelmValues(); + Map serviceEmail = parseYaml( + (String) value(valuesYaml, "argo-cd", "notifications", "notifiers", "service.email") + ); + + assertThat(serviceEmail.get("host")).isEqualTo(config.getFeatures().getMail().getSmtpAddress()); + assertThat(serviceEmail.get("port")).isEqualTo(config.getFeatures().getMail().getSmtpPort()); + assertThat(serviceEmail.get("username")).isEqualTo("$email-username"); + assertThat(serviceEmail.get("password")).isEqualTo("$email-password"); + + Secret mailSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-notifications-secret") + .get(); + + assertThat(mailSecret).isNotNull(); + assertThat(decodedSecretValue(mailSecret, "email-username")) + .isEqualTo(config.getFeatures().getMail().getSmtpUser()); + assertThat(decodedSecretValue(mailSecret, "email-password")) + .isEqualTo(config.getFeatures().getMail().getSmtpPassword()); + } + + @Test + void resolvesExternalMailServerCredentialsFromSecretWithoutMutatingConfig() throws IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpUser("fallback-user"); + config.getFeatures().getMail().setSmtpPassword("fallback-password"); + Credentials reference = new Credentials(); + reference.setSecretName("smtp-credentials"); + reference.setSecretNamespace("gop-job"); + config.getFeatures().getMail().setCredentials(reference); + createSecretIfMissing( + "smtp-credentials", + "gop-job", + Map.of("username", encode("secret-smtp-user"), "password", encode("secret-smtp-password")) + ); + + Map valuesYaml = executeAndReadHelmValues(); + Map serviceEmail = parseYaml( + (String) value(valuesYaml, "argo-cd", "notifications", "notifiers", "service.email") + ); + + assertThat(serviceEmail.get("username")).isEqualTo("$email-username"); + assertThat(serviceEmail.get("password")).isEqualTo("$email-password"); + Secret mailSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-notifications-secret") + .get(); + assertThat(decodedSecretValue(mailSecret, "email-username")).isEqualTo("secret-smtp-user"); + assertThat(decodedSecretValue(mailSecret, "email-password")).isEqualTo("secret-smtp-password"); + assertThat(config.getFeatures().getMail().getSmtpUser()).isEqualTo("fallback-user"); + assertThat(config.getFeatures().getMail().getSmtpPassword()).isEqualTo("fallback-password"); + assertThat(reference.getUsername()).isNull(); + assertThat(reference.getPassword()).isNull(); + assertThat(Files.readString(Path.of(actualHelmValuesFile()))) + .doesNotContain("secret-smtp-user", "secret-smtp-password"); + } + + @Test + void createsKubernetesSecretWhenExternalMailServerUsernameIsSet() { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpUser("argo@example.com"); + + execute(createArgoCD()); + + Secret mailSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-notifications-secret") + .get(); + + assertThat(mailSecret).isNotNull(); + assertThat(decodedSecretValue(mailSecret, "email-username")) + .isEqualTo(config.getFeatures().getMail().getSmtpUser()); + } + + @Test + void createsKubernetesSecretWhenExternalMailServerPasswordIsSet() { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + config.getFeatures().getMail().setSmtpPassword("1101:ABCabc&/+*~"); + + execute(createArgoCD()); + + Secret mailSecret = client.secrets() + .inNamespace("argocd") + .withName("argocd-notifications-secret") + .get(); + + assertThat(mailSecret).isNotNull(); + assertThat(decodedSecretValue(mailSecret, "email-password")) + .isEqualTo(config.getFeatures().getMail().getSmtpPassword()); + } + + @Test + void configuresExternalMailServerWithoutOptionalValues() throws IOException { + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.com"); + + Map valuesYaml = executeAndReadHelmValues(); + Map serviceEmail = parseYaml( + (String) value(valuesYaml, "argo-cd", "notifications", "notifiers", "service.email") + ); + + assertThat(client.secrets().inNamespace("argocd").withName("argocd-notifications-secret").get()).isNull(); + assertThat(serviceEmail.get("host")).isEqualTo("smtp.example.com"); + assertThat(serviceEmail).doesNotContainKeys("port", "username", "password"); + } + + @Test + void usesDefaultMailServerWhenNoExternalServerIsSet() throws IOException { + config.getFeatures().getMail().setActive(true); + + Map valuesYaml = executeAndReadHelmValues(); + Map serviceEmail = parseYaml( + (String) value(valuesYaml, "argo-cd", "notifications", "notifiers", "service.email") + ); + + assertThat(serviceEmail.get("port")).isEqualTo(1025); + assertThat(serviceEmail).doesNotHaveToString("username"); + assertThat(serviceEmail).doesNotHaveToString("password"); + } + + @Test + void rejectsNonStringArgoCdOperatorEnvironmentValues() throws NoSuchFieldException, IllegalAccessException { + config.getFeatures().getArgocd().setOperator(true); + Field envField = config.getFeatures().getArgocd().getClass().getDeclaredField("env"); + envField.setAccessible(true); + envField.set(config.getFeatures().getArgocd(), List.of(map("name", "REPLICAS", "value", 2))); + + assertThatThrownBy(() -> createArgoCD().postConfigInit(config)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Invalid entry found: [name:REPLICAS, value:2]"); + } + + @Test + void installsArgoCdWithCustomValues() throws IOException { + config.getFeatures().getArgocd().setValues(map("argo-cd", map("key", "value"))); + + Map valuesYaml = executeAndReadHelmValues(); + + assertThat(value(valuesYaml, "argo-cd", "key")).isEqualTo("value"); + } + + @Test + void preparesRepositoriesForAirGappedMode() throws IOException { + config.getFeatures().getMonitoring().setActive(false); + config.getApplication().setMirrorRepos(true); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + assertThat(sourceRepos) + .contains("http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/" + + "kube-prometheus-stack") + .doesNotContain("https://prometheus-community.github.io/helm-charts"); + } + + @Test + void generatesArgoCdYamlWithEmptyNamePrefix() throws IOException { + ArgoCDForTest argocd = (ArgoCDForTest) createArgoCD(); + execute(argocd); + + GitRepo clusterResourcesRepo = argocd.clusterResourcesRepo; + assertArgoCdYamlPrefixes( + clusterResourcesRepo.getGitProvider().getUrl(), + "", + argocd.getClusterRepoLayout() + ); + } + + @Test + void generatesArgoCdYamlWithNamePrefix() throws IOException { + config.getApplication().setNamePrefix("abc-"); + + ArgoCDForTest argocd = (ArgoCDForTest) createArgoCD(); + execute(argocd); + + GitRepo clusterResourcesRepo = argocd.clusterResourcesRepo; + assertArgoCdYamlPrefixes( + clusterResourcesRepo.getGitProvider().getUrl(), + config.getApplication().getNamePrefix(), + argocd.getClusterRepoLayout() + ); + } + + @Test + void skipsCrdsForArgoCd() throws IOException { + config.getApplication().setSkipCrds(true); + + Map valuesYaml = executeAndReadHelmValues(); + + assertThat(value(valuesYaml, "argo-cd", "crds", "install")).isEqualTo(false); + } + + @Test + void configuresArgoCdWithActiveNetworkPolicies() throws IOException { + config.getApplication().setNetpols(true); + config.getApplication().setNamePrefix("my-prefix-"); + config.getScm().getScmManager().setNamespace("my-prefix-scm-manager"); + + Map valuesYaml = executeAndReadHelmValues(); + String argocdValues = Files.readString( + Path.of(clusterResourcesRepoLayout.argocdRoot(), "argocd", "values.yaml") + ); + String allowNamespaces = Files.readString( + Path.of(clusterResourcesRepoLayout.argocdRoot(), "argocd", "templates", "allow-namespaces.yaml") + ); + + assertThat(value(valuesYaml, "argo-cd", "global", "networkPolicy", "create")).isEqualTo(true); + assertThat(argocdValues).contains("namespace: my-prefix-monitoring"); + assertThat(allowNamespaces) + .contains("namespace: my-prefix-scm-manager") + .doesNotContain("namespace: my-prefix-my-prefix-scm-manager") + .contains("kubernetes.io/metadata.name: my-prefix-argocd"); + } + + @Test + void setsOperatorServerInsecureToTrueWhenInsecureIsSet() throws IOException { + config.getApplication().setInsecure(true); + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + assertThat(value(yaml, "spec", "server", "insecure")).isEqualTo(true); + } + + @Test + void setsOperatorCustomValues() throws IOException { + config.getFeatures().getArgocd().setValues(map("spec", map("key", "value"))); + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + assertThat(value(yaml, "spec", "key")).isEqualTo("value"); + } + + @Test + void setsOperatorArgoCdUrlAndAdditionalRedirectUrls() throws IOException { + config.getFeatures().getArgocd().setUrl("https://argocd.localhost"); + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + Map extraConfig = mapValue(yaml, "spec", "extraConfig"); + assertThat(extraConfig.get("url")).isEqualTo("https://argocd.localhost"); + assertThat((String) extraConfig.get("additionalUrls")) + .contains("http://argocd.localhost", "https://argocd.localhost"); + } + + @Test + void setsOperatorServerInsecureToFalseWhenInsecureIsNotSet() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + assertThat(value(yaml, "spec", "server", "insecure")).isEqualTo(false); + } + + @Test + void generatesIngressWithExpectedHostWhenInsecureAndNotOnOpenShift() throws IOException { + config.getApplication().setInsecure(true); + config.getFeatures().getArgocd().setUrl("http://argocd.localhost"); + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml"); + assertThat(ingressFile) + .as("Ingress file should be generated for insecure mode on non-OpenShift") + .exists(); + + Map ingressYaml = parseActualYaml(ingressFile.toString()); + List> rules = mapListValue(ingressYaml, "spec", "rules"); + assertThat((String) rules.get(0).get("host")) + .as("Ingress host should match configured ArgoCD hostname") + .isEqualTo(URI.create(config.getFeatures().getArgocd().getUrl()).getHost()); + } + + @Test + void doesNotGenerateIngressWhenInsecureIsFalse() { + config.getApplication().setInsecure(false); + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml"); + assertThat(ingressFile) + .as("Ingress file should not be generated when insecure is false") + .doesNotExist(); + } + + @Test + void doesNotGenerateIngressOnOpenShift() { + config.getApplication().setInsecure(true); + ArgoCD argocd = setupOperatorTest(true); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml"); + assertThat(ingressFile) + .as("Ingress file should not be generated on OpenShift") + .doesNotExist(); + } + + @Test + void doesNotGenerateIngressWhenInsecureIsFalseAndOpenShiftIsTrue() { + config.getApplication().setInsecure(false); + ArgoCD argocd = setupOperatorTest(true); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml"); + assertThat(ingressFile) + .as("Ingress file should not be generated when both flags are false") + .doesNotExist(); + } + + @Test + void includesMonitoringAndExternalSecretsResourceInclusionsWhenFeaturesAreActive() throws IOException { + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getSecrets().setActive(true); + + String expectedMonitoring = "monitoring.coreos.com"; + String expectedExternalSecret = "external-secrets.io"; + + ArgoCD argocd = setupOperatorTest(true); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + String resourceInclusions = (String) value(yaml, "spec", "resourceInclusions"); + + assertThat(resourceInclusions).contains(expectedMonitoring, expectedExternalSecret); + } + + @Test + void excludesMonitoringAndExternalSecretsResourceInclusionsWhenFeaturesAreInactive() throws IOException { + config.getFeatures().getMonitoring().setActive(false); + config.getFeatures().getSecrets().setActive(false); + + String expectedMonitoring = "monitoring.coreos.com"; + String expectedExternalSecret = "external-secrets.io"; + + ArgoCD argocd = setupOperatorTest(true); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + String resourceInclusions = (String) value(yaml, "spec", "resourceInclusions"); + + assertThat(resourceInclusions).doesNotContain(expectedMonitoring, expectedExternalSecret); + } + + @Test + void configuresResourceInclusionsCluster() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + config.getFeatures().getArgocd().setResourceInclusionsCluster("https://192.168.0.1:6443"); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + String expectedClusterUrl = "https://192.168.0.1:6443"; + String resourceInclusions = (String) value(yaml, "spec", "resourceInclusions"); + List> parsedResourceInclusions = parseYamlList(resourceInclusions); + + for (Map resource : parsedResourceInclusions) { + assertThat(resource).containsKey("clusters"); + assertThat(listValue(resource, "clusters")).contains(expectedClusterUrl); + } + } + + @Test + void setsEnvironmentVariablesInArgoCdComponentsWhenProvided() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + config.getFeatures().getArgocd().setEnv(List.of( + Map.of("name", "ENV_VAR_1", "value", "value1"), + Map.of("name", "ENV_VAR_2", "value", "value2") + )); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + List> expectedEnv = List.of( + map("name", "ENV_VAR_1", "value", "value1"), + map("name", "ENV_VAR_2", "value", "value2") + ); + + assertThat(value(yaml, "spec", "applicationSet", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "notifications", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "controller", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "repo", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "server", "env")).isEqualTo(expectedEnv); + } + + @Test + void doesNotSetEnvironmentVariablesWhenNoneAreProvided() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + config.getFeatures().getArgocd().setEnv(List.of()); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + + assertThat(mapValue(yaml, "spec", "applicationSet")).doesNotContainKey("env"); + assertThat(mapValue(yaml, "spec", "notifications")).doesNotContainKey("env"); + assertThat(mapValue(yaml, "spec", "controller")).doesNotContainKey("env"); + assertThat(mapValue(yaml, "spec", "redis")).doesNotContainKey("env"); + assertThat(mapValue(yaml, "spec", "repo")).doesNotContainKey("env"); + assertThat(mapValue(yaml, "spec", "server")).doesNotContainKey("env"); + } + + @Test + void setsSingleEnvironmentVariableInArgoCdComponentsWhenProvided() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + config.getFeatures().getArgocd().setEnv(List.of( + Map.of("name", "ENV_VAR_SINGLE", "value", "singleValue") + )); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map yaml = parseActualYaml(clusterResourcesRepoLayout.operatorConfigFile()); + List> expectedEnv = List.of( + map("name", "ENV_VAR_SINGLE", "value", "singleValue") + ); + + assertThat(value(yaml, "spec", "applicationSet", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "notifications", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "controller", "env")).isEqualTo(expectedEnv); + assertThat(value(yaml, "spec", "server", "env")).isEqualTo(expectedEnv); + } + + @Test + void preparesArgoCdRepoWithOperatorConfigurationFile() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Path argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()); + Path rbacConfigPath = Path.of(clusterResourcesRepoLayout.operatorRbacDir()); + + assertThat(argocdConfigPath.toFile()).exists(); + assertThat(rbacConfigPath.toFile()).exists(); + + Map yaml = parseActualYaml(argocdConfigPath.toString()); + assertThat(yaml.get("apiVersion")).isEqualTo("argoproj.io/v1beta1"); + assertThat(yaml.get("kind")).isEqualTo("ArgoCD"); + } + + @Test + void doesNotCreateOperatorFilesWhenOperatorIsDisabled() { + ArgoCD argocd = createArgoCD(); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Path argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()); + Path rbacConfigPath = Path.of(clusterResourcesRepoLayout.operatorRbacDir()); + + assertThat(argocdConfigPath.toFile()).doesNotExist(); + assertThat(rbacConfigPath.toFile()).doesNotExist(); + } + + @Test + void deploysWithOperatorWithoutOpenShiftConfiguration() throws IOException { + ArgoCD argocd = setupOperatorTest(false); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + Path argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()); + + assertThat(argocdConfigPath.toFile()).exists(); + + Map yaml = parseActualYaml(argocdConfigPath.toString()); + assertThat(value(yaml, "spec", "rbac")).isNull(); + assertThat(value(yaml, "spec", "sso")).isNull(); + + Map argocdYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.applicationsDir(), "argocd.yaml").toString() + ); + assertThat(value(argocdYaml, "spec", "source", "directory", "recurse")).isEqualTo(true); + assertThat(value(argocdYaml, "spec", "source", "path")).isEqualTo("apps/argocd/operator/"); + } + + @Test + void generatesOperatorRbacsFromRbacDefinitions() throws IOException { + config.getApplication().setNamePrefix("testPrefix-"); + + List expectedNamespaces = List.of( + "testPrefix-monitoring", + "testPrefix-secrets", + "testPrefix-traefik", + "testPrefix-example-apps-staging", + "testPrefix-example-apps-production" + ); + + config.getApplication().getNamespaces().setDedicatedNamespaces(new LinkedHashSet<>(List.of( + "monitoring", + "secrets", + "traefik", + "example-apps-staging", + "example-apps-production" + ))); + + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File rbacPath = Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile(); + + for (String namespace : expectedNamespaces) { + File roleFile = new File(rbacPath, "role-argocd-" + namespace + ".yaml"); + File bindingFile = new File(rbacPath, "rolebinding-argocd-" + namespace + ".yaml"); + + assertThat(roleFile).exists(); + assertThat(bindingFile).exists(); + + Map roleYaml = parseActualYaml(roleFile.toString()); + Map bindingYaml = parseActualYaml(bindingFile.toString()); + + assertThat(roleYaml.get("kind")).isEqualTo("Role"); + assertThat(value(roleYaml, "metadata", "name")).isEqualTo("argocd"); + assertThat(value(roleYaml, "metadata", "namespace")).isEqualTo(namespace); + + assertThat(bindingYaml.get("kind")).isEqualTo("RoleBinding"); + assertThat(value(bindingYaml, "metadata", "name")).isEqualTo("argocd"); + assertThat(value(bindingYaml, "metadata", "namespace")).isEqualTo(namespace); + + List> subjects = mapListValue(bindingYaml, "subjects"); + assertThat(subjects).isNotEmpty(); + assertThat(subjects.stream().map(subject -> subject.get("kind")).toList()) + .containsOnly("ServiceAccount"); + assertThat(subjects.stream().map(subject -> subject.get("namespace")).toList()) + .containsOnly("testPrefix-argocd"); + assertThat(subjects.stream().map(subject -> subject.get("name")).toList()) + .containsExactlyInAnyOrder( + "argocd-argocd-server", + "argocd-argocd-application-controller", + "argocd-applicationset-controller" + ); + + Map roleRef = mapValue(bindingYaml, "roleRef"); + assertThat(roleRef).isNotNull(); + assertThat(roleRef.get("name")).isEqualTo("argocd"); + assertThat(roleRef.get("kind")).isEqualTo("Role"); + } + } + + @Test + void includesNodeAccessRulesInOperatorRbacWhenNotOnOpenShift() throws IOException { + config.getApplication().setNamePrefix("testprefix-"); + + ArgoCD argocd = setupOperatorTest(false); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File rbacDir = Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile(); + File roleFile = new File(rbacDir, "role-argocd-testprefix-monitoring.yaml"); + + Map yaml = parseActualYaml(roleFile.toString()); + List> rules = mapListValue(yaml, "rules"); + + assertThat(rules).anyMatch(rule -> { + List resources = listValue(rule, "resources"); + return resources.contains("nodes") && resources.contains("nodes/metrics"); + }); + } + + @Test + void doesNotIncludeNodeAccessRulesInOperatorRbacWhenOnOpenShift() throws IOException { + config.getApplication().setNamePrefix("testprefix-"); + + ArgoCD argocd = setupOperatorTest(true); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + File rbacDir = Path.of(clusterResourcesRepoLayout.operatorRbacDir()).toFile(); + File roleFile = new File(rbacDir, "role-argocd-testprefix-monitoring.yaml"); + + Map yaml = parseActualYaml(roleFile.toString()); + List> rules = mapListValue(yaml, "rules"); + + assertThat(rules).noneMatch(rule -> { + List resources = listValue(rule, "resources"); + return resources.contains("nodes") && resources.contains("nodes/metrics"); + }); + } + + @Test + void deploysWithOperatorWithOpenShiftConfiguration() throws IOException { + ArgoCD argocd = setupOperatorTest(true); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Path argocdConfigPath = Path.of(clusterResourcesRepoLayout.operatorConfigFile()); + assertThat(argocdConfigPath.toFile()).exists(); + + Map yaml = parseActualYaml(argocdConfigPath.toString()); + assertThat(value(yaml, "spec", "sso")).isNotNull(); + assertThat(value(yaml, "spec", "sso", "dex", "openShiftOAuth")).isEqualTo(true); + assertThat(value(yaml, "spec", "sso", "provider")).isEqualTo("dex"); + assertThat(value(yaml, "spec", "rbac")).isNotNull(); + assertThat(value(yaml, "spec", "server", "route", "enabled")).isEqualTo(true); + } + + @Test + void createsAllNecessaryNamespaces() { + ArgoCD argocd = createArgoCD(); + + execute(argocd); + + for (String namespace : config.getApplication().getNamespaces().getActiveNamespaces()) { + assertThat(client.namespaces().withName(namespace).get()).isNotNull(); + } + } + + @Test + void doesNotGenerateCentralBootstrapIngressWhenInsecureIsFalseInDedicatedMode() { + setupDedicatedInstanceMode(); + + assertThat(clusterResourcesRepoLayout).isNotNull(); + + File ingressFile = new File(clusterResourcesRepoLayout.operatorDir(), "ingress.yaml"); + assertThat(ingressFile) + .as("Ingress file should not be generated when insecure is false") + .doesNotExist(); + } + + @Test + void dedicatedModeAppliesCentralAndTenantBootstrapResources() { + config.getApplication().setNamePrefix("testPrefix-"); + config.getMultiTenant().getScmManager().setUrl("scmm.testhost/scm"); + config.getMultiTenant().getScmManager().setUsername("testUserName"); + config.getMultiTenant().getScmManager().setPassword("testPassword"); + config.getMultiTenant().setUseDedicatedInstance(true); + config.getFeatures().getArgocd().setOperator(true); + config.getFeatures().getArgocd().setResourceInclusionsCluster("https://192.168.0.1:6443"); + + doReturn("Applied").when(k8sClient).applyYaml(any(String.class)); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + + ArgoCDForTest argoCDForTest = (ArgoCDForTest) argocd; + ArgoCDRepoLayout clusterLayout = argoCDForTest.getClusterRepoLayout(); + ArgoCDRepoLayout tenantLayout = argoCDForTest.getTenantRepoLayout(); + + verify(k8sClient).applyYaml(Path.of(clusterLayout.projectsDir(), "tenant.yaml").toString()); + verify(k8sClient).applyYaml(Path.of(clusterLayout.applicationsDir(), "bootstrap.yaml").toString()); + verify(k8sClient).applyYaml(Path.of(tenantLayout.projectsDir(), "argocd.yaml").toString()); + verify(k8sClient).applyYaml(Path.of(tenantLayout.applicationsDir(), "bootstrap.yaml").toString()); + } + + @Test + void dedicatedModeCreatesCentralRepoCredentialsSecret() { + config.getApplication().setNamePrefix("testPrefix-"); + config.getMultiTenant().getScmManager().setUrl("scmm.testhost/scm"); + config.getMultiTenant().getScmManager().setUsername("testUserName"); + config.getMultiTenant().getScmManager().setPassword("testPassword"); + config.getMultiTenant().setUseDedicatedInstance(true); + config.getFeatures().getArgocd().setOperator(true); + config.getFeatures().getArgocd().setResourceInclusionsCluster("https://192.168.0.1:6443"); + + doReturn("Applied").when(k8sClient).applyYaml(any(String.class)); + + execute(createArgoCD()); + + Secret centralRepoCredentialsSecret = client.secrets() + .inNamespace(config.getMultiTenant().getCentralArgocdNamespace()) + .withName("argocd-repo-creds-central-scm") + .get(); + + assertThat(centralRepoCredentialsSecret).isNotNull(); + assertThat(centralRepoCredentialsSecret.getMetadata().getLabels().get("argocd.argoproj.io/secret-type")) + .isEqualTo("repo-creds"); + } + + @Test + void generatesCentralTemplatesForDedicatedInstances() throws IOException { + setupDedicatedInstanceMode(); + + assertThat(clusterResourcesRepoLayout).isNotNull(); + + assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/argocd.yaml")).exists(); + assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/bootstrap.yaml")).exists(); + assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/projects.yaml")).exists(); + assertThat(new File(clusterResourcesRepoLayout.argocdRoot() + "/applications/example-apps.yaml")).doesNotExist(); + + Map argocdYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.argocdRoot(), "applications/argocd.yaml").toString() + ); + Map bootstrapYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.argocdRoot(), "applications/bootstrap.yaml").toString() + ); + Map projectsYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.argocdRoot(), "applications/projects.yaml").toString() + ); + + assertThat(value(argocdYaml, "metadata", "name")).isEqualTo("testPrefix-argocd"); + assertThat(value(argocdYaml, "metadata", "namespace")).isEqualTo("argocd"); + assertThat(value(argocdYaml, "spec", "project")).isEqualTo("testPrefix"); + assertThat(value(argocdYaml, "spec", "source", "path")).isEqualTo("apps/argocd/operator/"); + + assertThat(value(bootstrapYaml, "metadata", "name")).isEqualTo("testPrefix-bootstrap"); + assertThat(value(bootstrapYaml, "metadata", "namespace")).isEqualTo("argocd"); + assertThat(value(bootstrapYaml, "spec", "project")).isEqualTo("testPrefix"); + assertThat(value(bootstrapYaml, "spec", "source", "repoURL")) + .isEqualTo("scmm.testhost/scm/repo/testPrefix-argocd/cluster-resources.git"); + + assertThat(value(projectsYaml, "metadata", "name")).isEqualTo("testPrefix-projects"); + assertThat(value(projectsYaml, "metadata", "namespace")).isEqualTo("argocd"); + assertThat(value(projectsYaml, "spec", "project")).isEqualTo("testPrefix"); + + File tenantProjectFile = new File(clusterResourcesRepoLayout.argocdRoot() + "/projects/tenant.yaml"); + assertThat(tenantProjectFile).exists(); + + Map tenantProject = parseActualYaml(tenantProjectFile.toString()); + assertThat(value(tenantProject, "metadata", "name")).isEqualTo("testPrefix"); + assertThat(value(tenantProject, "metadata", "namespace")).isEqualTo("argocd"); + assertThat(listValue(tenantProject, "spec", "sourceRepos")) + .first() + .isEqualTo("scmm.testhost/scm/repo/testPrefix-argocd/cluster-resources.git"); + } + + @Test + void appendsNamespacesToDefaultClusterConfigSecret() { + config.getApplication().getNamespaces().setDedicatedNamespaces(new LinkedHashSet<>(List.of( + "dedi-test1", + "dedi-test2", + "dedi-test3" + ))); + config.getApplication().getNamespaces().setTenantNamespaces(new LinkedHashSet<>(List.of( + "tenant-test1", + "tenant-test2", + "tenant-test3" + ))); + + setupDedicatedInstanceMode(); + + Secret defaultClusterConfig = client.secrets() + .inNamespace("argocd") + .withName("argocd-default-cluster-config") + .get(); + + assertThat(defaultClusterConfig).isNotNull(); + + String namespaces = decodedSecretValue(defaultClusterConfig, "namespaces"); + assertThat(namespaces) + .contains("testnamespace1") + .contains("testnamespace2") + .contains("testPrefix-dedi-test1") + .contains("testPrefix-dedi-test2") + .contains("testPrefix-dedi-test3") + .contains("testPrefix-tenant-test1") + .contains("testPrefix-tenant-test2") + .contains("testPrefix-tenant-test3"); + } + + @Test + void removesMultiTenantFolderWhenDedicatedModeIsDisabled() { + config.getMultiTenant().setUseDedicatedInstance(false); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), "multiTenant/")).doesNotExist(); + assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), "applications/")).exists(); + assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), "projects/")).exists(); + } + + @Test + void removesUnusedMultiTenantFolderInDedicatedMode() { + setupDedicatedInstanceMode(); + + assertThat(clusterResourcesRepoLayout).isNotNull(); + assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), "multiTenant/")).doesNotExist(); + assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), "applications/")).exists(); + assertThat(Path.of(clusterResourcesRepoLayout.argocdRoot(), "projects/")).exists(); + } + + @Test + void generatesDedicatedModeRbacs() throws IOException { + config.getApplication().getNamespaces().setTenantNamespaces(new LinkedHashSet<>(List.of( + "testprefix-tenant-test1", + "testprefix-tenant-test2", + "testprefix-tenant-test3" + ))); + setupDedicatedInstanceMode(); + + File rbacFolder = new File(clusterResourcesRepoLayout.operatorRbacDir()); + File rbacTenantFolder = new File(clusterResourcesRepoLayout.operatorRbacDir(), "tenant"); + assertThat(rbacFolder).exists(); + assertThat(rbacTenantFolder).exists(); + + assertThat(rbacFolder.listFiles(File::isFile)).hasSize(14); + assertThat(rbacTenantFolder.listFiles(File::isFile)).hasSize(6); + + for (File file : rbacFolder.listFiles()) { + if (file.getName().startsWith("role-") && file.getName().contains("dedi")) { + Map rbacFile = parseActualYaml(file.toString()); + assertThat(value(rbacFile, "metadata", "namespace")) + .isIn(config.getApplication().getNamespaces().getActiveNamespaces()); + } + if (file.getName().startsWith("rolebinding-") && file.getName().contains("dedi")) { + Map rbacFile = parseActualYaml(file.toString()); + List> subjects = mapListValue(rbacFile, "subjects"); + assertThat(subjects.stream().map(subject -> subject.get("namespace")).toList()) + .containsExactly("argocd", "argocd", "argocd"); + } + } + + for (File file : rbacTenantFolder.listFiles()) { + if (file.getName().startsWith("role-")) { + Map rbacFile = parseActualYaml(file.toString()); + assertThat(value(rbacFile, "metadata", "namespace")) + .isIn(config.getApplication().getNamespaces().getTenantNamespaces()); + } + + if (file.getName().startsWith("rolebinding-")) { + Map rbacFile = parseActualYaml(file.toString()); + List> subjects = mapListValue(rbacFile, "subjects"); + assertThat(subjects.stream().map(subject -> subject.get("namespace")).toList()) + .containsExactly("testPrefix-argocd", "testPrefix-argocd", "testPrefix-argocd"); + } + } + } + + @Test + void usesExternalSourceRepoUrlsWhenMirroringIsDisabled() throws IOException { + config.getApplication().setMirrorRepos(false); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + + assertThat(sourceRepos).contains( + "https://charts.external-secrets.io", + "https://codecentric.github.io/helm-charts", + "https://prometheus-community.github.io/helm-charts", + "https://traefik.github.io/charts", + "https://helm.releases.hashicorp.com", + "https://charts.jetstack.io" + ); + assertThat(sourceRepos).doesNotContain( + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager" + ); + assertThat(sourceRepos).doesNotContain( + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/kube-prometheus-stack.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/traefik.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/external-secrets.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/vault.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/cert-manager.git" + ); + } + + @Test + void usesScmManagerMirrorSourceRepoUrlsWhenMirroringIsEnabled() throws IOException { + config.getApplication().setMirrorRepos(true); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + + assertThat(sourceRepos).contains( + "http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack", + "http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik", + "http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets", + "http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault", + "http://scmm.scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager" + ); + assertThat(sourceRepos).doesNotContain( + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/kube-prometheus-stack.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/traefik.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/external-secrets.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/vault.git", + "http://scmm.scm-manager.svc.cluster.local/scm/3rd-party-dependencies/cert-manager.git" + ); + } + + @Test + void usesGitLabMirrorSourceRepoUrlsWhenMirroringIsEnabled() throws IOException { + config.getApplication().setMirrorRepos(true); + config.getScm().setScmProviderType(ScmProviderType.GITLAB); + config.getScm().getGitlab().setUrl("https://testGitLab.com/testgroup"); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + + assertThat(sourceRepos).contains( + "https://testGitLab.com/testgroup/3rd-party-dependencies/kube-prometheus-stack.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/traefik.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/external-secrets.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/vault.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/cert-manager.git" + ); + } + + @Test + void usesGitLabMirrorSourceRepoUrlsWithNamePrefix() throws IOException { + config.getApplication().setMirrorRepos(true); + config.getScm().setScmProviderType(ScmProviderType.GITLAB); + config.getScm().getGitlab().setUrl("https://testGitLab.com/testgroup"); + config.getApplication().setNamePrefix("test1-"); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + + assertThat(sourceRepos).contains( + "https://testGitLab.com/testgroup/3rd-party-dependencies/kube-prometheus-stack.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/traefik.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/external-secrets.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/vault.git", + "https://testGitLab.com/testgroup/3rd-party-dependencies/cert-manager.git" + ); + assertThat(sourceRepos).doesNotContain( + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager" + ); + } + + @Test + void usesScmManagerMirrorSourceRepoUrlsWithNamePrefix() throws IOException { + config.getApplication().setMirrorRepos(true); + config.getApplication().setNamePrefix("test1-"); + + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + + Map clusterResourcesYaml = parseActualYaml( + Path.of(clusterResourcesRepoLayout.projectsDir(), "cluster-resources.yaml").toString() + ); + List sourceRepos = listValue(clusterResourcesYaml, "spec", "sourceRepos"); + + assertThat(sourceRepos).contains( + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/kube-prometheus-stack", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/traefik", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/external-secrets", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/vault", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/repo/3rd-party-dependencies/cert-manager" + ); + assertThat(sourceRepos).doesNotContain( + "http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/kube-prometheus-stack.git", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/traefik.git", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/external-secrets.git", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/vault.git", + "http://scmm.test1-scm-manager.svc.cluster.local/scm/3rd-party-dependencies/cert-manager.git" + ); + } + + private void setupDedicatedInstanceMode() { + config.getApplication().setNamePrefix("testPrefix-"); + config.getMultiTenant().getScmManager().setUrl("scmm.testhost/scm"); + config.getMultiTenant().getScmManager().setUsername("testUserName"); + config.getMultiTenant().getScmManager().setPassword("testPassword"); + config.getMultiTenant().setUseDedicatedInstance(true); + ArgoCD argocd = setupOperatorTest(false); + + doReturn("Applied").when(k8sClient).applyYaml(any(String.class)); + + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + } + + private ArgoCD setupOperatorTest(boolean openshift) { + config.getFeatures().getArgocd().setOperator(true); + config.getFeatures().getArgocd().setResourceInclusionsCluster("https://192.168.0.1:6443"); + config.getApplication().setOpenshift(openshift); + return createArgoCD(); + } + + private void assertArgoCdYamlPrefixes( + String scmmUrl, + String expectedPrefix, + ArgoCDRepoLayout repoLayout) throws IOException { + assertAllYamlFiles(new File(repoLayout.argocdRoot()), "projects", 3, file -> { + Map yaml = parseActualYaml(file.toString()); + List sourceRepos = listValue(yaml, "spec", "sourceRepos"); + + if (sourceRepos != null) { + for (String sourceRepo : sourceRepos) { + if (sourceRepo.startsWith(scmmUrl)) { + assertThat(sourceRepo) + .as(file + " sourceRepos have name prefix") + .startsWith(scmmUrl + "/repo/" + expectedPrefix + "argocd"); + } + } + } + + String metadataNamespace = (String) value(yaml, "metadata", "namespace"); + if (metadataNamespace != null && !metadataNamespace.isEmpty()) { + assertThat(metadataNamespace) + .as(file + " metadata.namespace has name prefix") + .isEqualTo(expectedPrefix + "argocd"); + } + + List sourceNamespaces = listValue(yaml, "spec", "sourceNamespaces"); + if (sourceNamespaces != null) { + for (String sourceNamespace : sourceNamespaces) { + if (!"*".equals(sourceNamespace)) { + assertThat(sourceNamespace) + .as(file + " spec.sourceNamespace has name prefix") + .startsWith(expectedPrefix); + } + } + } + }); + + assertAllYamlFiles(new File(repoLayout.argocdRoot()), "applications", 3, file -> { + Map yaml = parseActualYaml(file.toString()); + assertThat((String) value(yaml, "spec", "source", "repoURL")) + .as(file + " repoURL have name prefix") + .startsWith(scmmUrl + "/repo/" + expectedPrefix + "argocd"); + assertThat(value(yaml, "metadata", "namespace")) + .as(file + " metadata.namespace has name prefix") + .isEqualTo(expectedPrefix + "argocd"); + assertThat(value(yaml, "spec", "destination", "namespace")) + .as(file + " spec.destination.namespace has name prefix") + .isEqualTo(expectedPrefix + "argocd"); + }); + } + + private static List findFilesContaining(File folder, String stringToSearch) throws IOException { + List result = new ArrayList<>(); + try (Stream files = Files.walk(folder.toPath())) { + for (Path file : files.filter(Files::isRegularFile).toList()) { + if (new String(Files.readAllBytes(file), StandardCharsets.UTF_8).contains(stringToSearch)) { + result.add(file); + } + } + } + return result; + } + + private static void assertAllYamlFiles( + File rootDir, + String childDir, + int numberOfFiles, + PathAssertion assertion) throws IOException { + Path rootPath = Path.of(rootDir.getAbsolutePath(), childDir); + List yamlFiles; + try (Stream files = Files.walk(rootPath)) { + yamlFiles = files + .filter(Files::isRegularFile) + .filter(path -> { + String normalizedPath = path.toString().replace('\\', '/'); + return normalizedPath.endsWith(".yaml") || normalizedPath.endsWith(".yml"); + }) + .toList(); + } + + for (Path yamlFile : yamlFiles) { + assertion.accept(yamlFile); + } + + assertThat(yamlFiles).hasSize(numberOfFiles); + } + + private Map executeAndReadHelmValues() throws IOException { + ArgoCD argocd = createArgoCD(); + execute(argocd); + clusterResourcesRepoLayout = ((ArgoCDForTest) argocd).getClusterRepoLayout(); + return parseActualYaml(actualHelmValuesFile()); + } + + private String actualHelmValuesFile() { + return clusterResourcesRepoLayout.helmDir() + "/values.yaml"; + } + + private ArgoCD createArgoCD() { + prepareKubernetesObjectsForArgoCd(); + + ArgoCDForTest argoCD = ArgoCDForTest.newWithAutoProviders(config, k8sClient, helmCommands); + return argoCD; + } + + private boolean execute(ArgoCD argoCD) { + return ((ArgoCDForTest) argoCD).execute(); + } + + private void prepareKubernetesObjectsForArgoCd() { + String namePrefix = config.getApplication().getNamePrefix() == null + ? "" + : config.getApplication().getNamePrefix(); + String configuredNamespace = config.getFeatures().getArgocd().getNamespace(); + String namespace = namePrefix + (configuredNamespace == null || configuredNamespace.isEmpty() + ? "argocd" + : configuredNamespace); + String centralNamespace = config.getMultiTenant().getCentralArgocdNamespace() == null + || config.getMultiTenant().getCentralArgocdNamespace().isEmpty() + ? "argocd" + : config.getMultiTenant().getCentralArgocdNamespace(); + + createNamespaceIfMissing(namespace); + createNamespaceIfMissing(centralNamespace); + createArgoCdCrds(); + + config.getApplication().getNamespaces().getActiveNamespaces().forEach(this::createNamespaceIfMissing); + + createSecretIfMissing("argocd-secret", namespace, Map.of()); + createSecretIfMissing("argocd-cluster", namespace, Map.of()); + createSecretIfMissing( + "argocd-default-cluster-config", + namespace, + Map.of("namespaces", encode("testnamespace1,testnamespace2")) + ); + + if (Boolean.TRUE.equals(config.getMultiTenant().getUseDedicatedInstance())) { + createSecretIfMissing( + "argocd-default-cluster-config", + centralNamespace, + Map.of("namespaces", encode("testnamespace1,testnamespace2")) + ); + } + } + + private void createArgoCdCrds() { + createNamespacedCrd( + "appprojects.argoproj.io", + "argoproj.io", + "v1alpha1", + "AppProject", + "appprojects", + "appproject" + ); + createNamespacedCrd( + "applications.argoproj.io", + "argoproj.io", + "v1alpha1", + "Application", + "applications", + "application" + ); + createNamespacedCrd( + "argocds.argoproj.io", + "argoproj.io", + "v1beta1", + "ArgoCD", + "argocds", + "argocd" + ); + } + + private void createNamespacedCrd( + String name, + String group, + String version, + String kind, + String plural, + String singular) { + if (client.apiextensions().v1().customResourceDefinitions().withName(name).get() != null) { + return; + } + + CustomResourceDefinition crd = new CustomResourceDefinitionBuilder() + .withNewMetadata() + .withName(name) + .endMetadata() + .withNewSpec() + .withGroup(group) + .withScope("Namespaced") + .withNewNames() + .withKind(kind) + .withPlural(plural) + .withSingular(singular) + .endNames() + .addNewVersion() + .withName(version) + .withServed(true) + .withStorage(true) + .withNewSchema() + .withNewOpenAPIV3Schema() + .withType("object") + .withXKubernetesPreserveUnknownFields(true) + .endOpenAPIV3Schema() + .endSchema() + .endVersion() + .endSpec() + .build(); + + client.apiextensions().v1().customResourceDefinitions().resource(crd).create(); + } + + private void createNamespaceIfMissing(String name) { + if (name == null || name.isEmpty()) { + throw new IllegalArgumentException(); + } + + if (client.namespaces().withName(name).get() == null) { + client.namespaces().resource(new NamespaceBuilder() + .withNewMetadata() + .withName(name) + .endMetadata() + .build()) + .create(); + } + } + + private String decodedSecretValue(Secret secret, String key) { + if (secret.getStringData() != null && secret.getStringData().containsKey(key)) { + return secret.getStringData().get(key); + } + + if (secret.getData() != null && secret.getData().containsKey(key)) { + return new String(Base64.getDecoder().decode(secret.getData().get(key)), StandardCharsets.UTF_8); + } + + return null; + } + + private void createSecretIfMissing(String name, String namespace, Map data) { + if (namespace == null || namespace.isEmpty()) { + throw new IllegalArgumentException(); + } + + createNamespaceIfMissing(namespace); + + if (client.secrets().inNamespace(namespace).withName(name).get() == null) { + Secret secret = new SecretBuilder() + .withNewMetadata() + .withName(name) + .withNamespace(namespace) + .endMetadata() + .withType("Opaque") + .withData(data) + .build(); + + client.secrets().inNamespace(namespace).resource(secret).create(); + } + } + + private static String encode(String value) { + return Base64.getEncoder().encodeToString(value.getBytes(StandardCharsets.UTF_8)); + } + + private static Map parseActualYaml(String pathToYamlFile) throws IOException { + return YAML_MAPPER.readValue(new File(pathToYamlFile), YAML_MAP_TYPE); + } + + private static Map parseYaml(String yaml) throws IOException { + return YAML_MAPPER.readValue(yaml, YAML_MAP_TYPE); + } + + private static List> parseYamlList(String yaml) throws IOException { + return YAML_MAPPER.readValue(yaml, YAML_MAP_LIST_TYPE); + } + + private static Object value(Map yaml, String... path) { + Object current = yaml; + for (String key : path) { + if (!(current instanceof Map currentMap)) { + return null; + } + current = currentMap.get(key); + } + return current; + } + + @SuppressWarnings("unchecked") + private static Map mapValue(Map yaml, String... path) { + return (Map) value(yaml, path); + } + + @SuppressWarnings("unchecked") + private static List> mapListValue(Map yaml, String... path) { + return (List>) value(yaml, path); + } + + @SuppressWarnings("unchecked") + private static List listValue(Map yaml, String... path) { + return (List) value(yaml, path); + } + + private static Map map(Object... keyValues) { + Map result = new LinkedHashMap<>(); + for (int index = 0; index < keyValues.length; index += 2) { + result.put((String) keyValues[index], keyValues[index + 1]); + } + return result; + } + + private static class ArgoCDK8sClientForTest extends K8sClientForTest { + + void configure(KubernetesClient client) { + setClient(client); + sleepTimeMillis = 1; + defaultRetries = 1; + } + } + + @FunctionalInterface + private interface PathAssertion { + + void accept(Path path) throws IOException; + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDForTest.java b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDForTest.java new file mode 100644 index 000000000..8ff7e5ae6 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDForTest.java @@ -0,0 +1,184 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.credentials.CredentialsResolver; +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.TestGitProvider; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.tools.core.argocd.mode.DeploymentModeFactory; +import com.cloudogu.gitops.utils.CommandExecutorForTest; +import com.cloudogu.gitops.utils.FileSystemUtils; +import org.eclipse.jgit.api.errors.GitAPIException; + +import java.util.LinkedHashSet; +import java.util.Map; +import java.util.stream.Collectors; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doNothing; + +class ArgoCDForTest extends ArgoCD { + + final Config cfg; + final GitProvider tenantProvider; + final GitProvider centralProvider; + final GitHandler gitHandler; + final RepositoryWorkspace repositoryWorkspace; + + GitRepo clusterResourcesRepo; + GitRepo tenantBootstrapRepo; + + static ArgoCDForTest newWithAutoProviders( + Config cfg, + K8sClient k8sClient, + CommandExecutorForTest helmCommands) { + Map providers = TestGitProvider.buildProviders(cfg); + + GitProvider tenantProvider = providers.get("tenant"); + GitProvider centralProvider = providers.get("central"); + + ArgoCDTestContext testContext = createTestContext(cfg, tenantProvider, centralProvider); + + return new ArgoCDForTest( + cfg, + k8sClient, + helmCommands, + tenantProvider, + centralProvider, + testContext + ); + } + + private static ArgoCDTestContext createTestContext( + Config cfg, + GitProvider tenantProvider, + GitProvider centralProvider) { + TestGitRepoFactory repoFactory = new TestGitRepoFactory(cfg, new FileSystemUtils()); + + GitProvider clusterResourcesProvider = Boolean.TRUE.equals(cfg.getMultiTenant().getUseDedicatedInstance()) + ? centralProvider + : tenantProvider; + + GitRepo clusterResourcesRepo = repoFactory.create("argocd/cluster-resources", clusterResourcesProvider); + stubCommitAndPush(clusterResourcesRepo); + + RepositoryWorkspace repositoryWorkspace; + GitRepo tenantBootstrapRepo = null; + + if (Boolean.TRUE.equals(cfg.getMultiTenant().getUseDedicatedInstance())) { + /* + * Test-only workspace separation: + * + * In the real dedicated multi-tenant setup, the central cluster-resources repo + * and the tenant bootstrap repo use the same logical repo target in different + * SCM-Manager instances. + * + * TestGitRepoFactory derives the local workspace from the repo target only. + * Therefore both GitRepo objects would otherwise point to the same local directory + * and tenant bootstrap templates would overwrite central bootstrap templates. + */ + tenantBootstrapRepo = repoFactory.create( + "argocd/tenant-bootstrap-cluster-resources", + tenantProvider + ); + stubCommitAndPush(tenantBootstrapRepo); + + repositoryWorkspace = new RepositoryWorkspace(clusterResourcesRepo, tenantBootstrapRepo); + } else { + repositoryWorkspace = new RepositoryWorkspace(clusterResourcesRepo); + } + + GitHandler gitHandler = new GitHandlerForTests(tenantProvider, centralProvider); + + return new ArgoCDTestContext( + gitHandler, + repositoryWorkspace, + clusterResourcesRepo, + tenantBootstrapRepo + ); + } + + private static void stubCommitAndPush(GitRepo repository) { + try { + doNothing().when(repository).commitAndPush(any(String.class)); + } catch (GitAPIException e) { + throw new IllegalStateException("Failed to configure GitRepo test spy", e); + } + } + + ArgoCDForTest( + Config cfg, + K8sClient k8sClient, + CommandExecutorForTest helmCommands, + GitProvider tenantProvider, + GitProvider centralProvider, + ArgoCDTestContext testContext) { + super( + k8sClient, + new HelmClient(helmCommands), + new FileSystemUtils(), + testContext.gitHandler(), + new DeploymentModeFactory(), + new ArgoCDToolConfigMapper(cfg), + new CredentialsResolver(k8sClient) + ); + + this.cfg = cfg; + this.tenantProvider = tenantProvider; + this.centralProvider = centralProvider; + this.gitHandler = testContext.gitHandler(); + this.repositoryWorkspace = testContext.repositoryWorkspace(); + this.clusterResourcesRepo = testContext.clusterResourcesRepo(); + this.tenantBootstrapRepo = testContext.tenantBootstrapRepo(); + + mockPrefixActiveNamespaces(cfg); + } + + private static void mockPrefixActiveNamespaces(Config config) { + String prefix = config.getApplication().getNamePrefix() == null + ? "" + : config.getApplication().getNamePrefix(); + + Config.ApplicationSchema.NamespaceSchema namespaces = config.getApplication().getNamespaces(); + namespaces.setDedicatedNamespaces( + namespaces.getDedicatedNamespaces().stream() + .map(namespace -> prefix + namespace) + .collect(Collectors.toCollection(LinkedHashSet::new)) + ); + namespaces.setTenantNamespaces( + namespaces.getTenantNamespaces().stream() + .map(namespace -> prefix + namespace) + .collect(Collectors.toCollection(LinkedHashSet::new)) + ); + } + + boolean execute() { + return super.execute(new ContextBuilder(cfg).build(), repositoryWorkspace); + } + + GitRepo getClusterResourcesRepo() { + return clusterResourcesRepo; + } + + ArgoCDRepoLayout getClusterRepoLayout() { + return getRepoSetup().clusterRepoLayout(); + } + + ArgoCDRepoLayout getTenantRepoLayout() { + return getRepoSetup().tenantRepoLayout(); + } + + private record ArgoCDTestContext( + GitHandler gitHandler, + RepositoryWorkspace repositoryWorkspace, + GitRepo clusterResourcesRepo, + GitRepo tenantBootstrapRepo) { + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetupTest.java b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetupTest.java new file mode 100644 index 000000000..0c92248d0 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDRepoSetupTest.java @@ -0,0 +1,380 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.TestGitProvider; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.databind.MappingIterator; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class ArgoCDRepoSetupTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + + private Config config; + + @BeforeEach + void setUp() { + config = Config.fromMap(Map.of( + "application", Map.of( + "namePrefix", "", + "tenantName", "", + "netpols", true, + "namespaces", Map.of( + "dedicatedNamespaces", List.of("argocd", "monitoring", "secrets"), + "tenantNamespaces", List.of("example-apps-staging", "example-apps-production") + ) + ), + "scm", Map.of( + "scmProviderType", ScmProviderType.SCM_MANAGER, + "scmManager", Map.of("internal", true), + "gitlab", Map.of("url", "") + ), + "multiTenant", Map.of( + "scmManager", Map.of("url", ""), + "gitlab", Map.of("url", ""), + "useDedicatedInstance", false, + "centralArgocdNamespace", "argocd" + ), + "features", Map.of( + "argocd", Map.of( + "operator", false, + "active", true, + "namespace", "argocd" + ), + "certManager", Map.of("active", false), + "ingress", Map.of("active", true), + "monitoring", Map.of( + "active", true, + "helm", Map.of( + "chart", "kube-prometheus-stack", + "version", "42.0.3" + ) + ), + "mail", Map.of("active", false), + "secrets", Map.of("active", true) + ) + )); + } + + private ArgoCDRepoSetupTestContext createSetup(FileSystemUtils fs) { + Map providers = TestGitProvider.buildProviders(config); + GitProvider tenantProvider = providers.get("tenant"); + GitProvider centralProvider = providers.get("central"); + + TestGitRepoFactory repoFactory = new TestGitRepoFactory(config, new FileSystemUtils()); + + GitRepo clusterResourcesRepo = repoFactory.create( + "argocd/cluster-resources", + Boolean.TRUE.equals(config.getMultiTenant().getUseDedicatedInstance()) ? centralProvider : tenantProvider + ); + + RepositoryWorkspace repositoryWorkspace; + + if (Boolean.TRUE.equals(config.getMultiTenant().getUseDedicatedInstance())) { + /* + * Test-only workspace separation: + * + * In the real dedicated multi-tenant setup, central cluster-resources and + * tenant bootstrap use the same logical repo target in different SCM-Manager + * instances. For this unit test, TestGitRepoFactory derives the local workspace + * from the repo target. Therefore we use a dedicated test target here to avoid + * both GitRepo objects pointing to the same local directory. + */ + GitRepo tenantBootstrapRepo = repoFactory.create( + "argocd/tenant-bootstrap-cluster-resources", + tenantProvider + ); + + repositoryWorkspace = new RepositoryWorkspace(clusterResourcesRepo, tenantBootstrapRepo); + } else { + repositoryWorkspace = new RepositoryWorkspace(clusterResourcesRepo); + } + + GitHandlerForTests gitHandler = new GitHandlerForTests(tenantProvider, centralProvider); + + DeploymentContext context = new ContextBuilder(config).build(); + return new ArgoCDRepoSetupTestContext( + ArgoCDRepoSetup.create( + fs, + gitHandler, + repositoryWorkspace, + new ArgoCDToolConfigMapper(config).map(context) + ), + repositoryWorkspace + ); + } + + @Test + void createSingleInstanceUsesClusterResourcesRepositoryOnly() { + config.getMultiTenant().setUseDedicatedInstance(false); + + ArgoCDRepoSetupTestContext testContext = createSetup(new FileSystemUtils()); + + assertThat(testContext.repositoryWorkspace.getClusterResourcesRepository()).isNotNull(); + assertThat(testContext.repositoryWorkspace.getClusterResourcesRepository().getRepoTarget()) + .isEqualTo("argocd/cluster-resources"); + assertThat(testContext.repositoryWorkspace.hasTenantBootstrapRepository()).isFalse(); + + assertThat(testContext.setup.clusterRepoLayout()).isNotNull(); + } + + @Test + void createDedicatedInstanceUsesClusterResourcesAndTenantBootstrapRepositoriesFromWorkspace() { + config.getMultiTenant().setUseDedicatedInstance(true); + + ArgoCDRepoSetupTestContext testContext = createSetup(new FileSystemUtils()); + + assertThat(testContext.repositoryWorkspace.getClusterResourcesRepository()).isNotNull(); + assertThat(testContext.repositoryWorkspace.getTenantBootstrapRepository()).isNotNull(); + assertThat(testContext.repositoryWorkspace.hasTenantBootstrapRepository()).isTrue(); + + assertThat(testContext.setup.clusterRepoLayout()).isNotNull(); + assertThat(testContext.setup.tenantRepoLayout()).isNotNull(); + } + + @Test + void dedicatedModeUsesSeparateLocalWorkspacesForCentralAndTenantBootstrapRepositories() throws IOException { + config.getMultiTenant().setUseDedicatedInstance(true); + + ArgoCDRepoSetupTestContext testContext = createSetup(new FileSystemUtils()); + + assertThat(new File(testContext.repositoryWorkspace.clusterResourcesRootDir()).getCanonicalPath()) + .isNotEqualTo(new File(testContext.repositoryWorkspace.tenantBootstrapRootDir()).getCanonicalPath()); + } + + @Test + void tenantRepoLayoutThrowsInSingleInstanceMode() { + config.getMultiTenant().setUseDedicatedInstance(false); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + assertThrows(IllegalStateException.class, setup::tenantRepoLayout); + } + + @Test + void tenantRepoLayoutIsAvailableInDedicatedInstanceMode() { + config.getMultiTenant().setUseDedicatedInstance(true); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + assertThat(setup.tenantRepoLayout()).isNotNull(); + } + + @Test + void prepareRepositoriesDeletesHelmDirWhenOperatorIsEnabled() { + config.getFeatures().getArgocd().setOperator(true); + config.getMultiTenant().setUseDedicatedInstance(false); + config.getApplication().setNetpols(true); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = setup.clusterRepoLayout(); + + assertThat(Path.of(clusterRepoLayout.helmDir())).doesNotExist(); + } + + @Test + void prepareRepositoriesDeletesOperatorDirWhenOperatorIsDisabled() { + config.getFeatures().getArgocd().setOperator(false); + config.getMultiTenant().setUseDedicatedInstance(false); + config.getApplication().setNetpols(true); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = setup.clusterRepoLayout(); + + assertThat(Path.of(clusterRepoLayout.operatorDir())).doesNotExist(); + assertThat(Path.of(clusterRepoLayout.helmDir())).exists(); + } + + @Test + void prepareRepositoriesInDedicatedModeReplacesSingleInstanceResourcesWithCentralResources() { + config.getFeatures().getArgocd().setOperator(false); + config.getMultiTenant().setUseDedicatedInstance(true); + config.getApplication().setNetpols(true); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = setup.clusterRepoLayout(); + + assertThat(Path.of(clusterRepoLayout.applicationsDir())).exists(); + assertThat(Path.of(clusterRepoLayout.projectsDir())).exists(); + assertThat(Path.of(clusterRepoLayout.multiTenantDir())).doesNotExist(); + } + + @Test + @SuppressWarnings("unchecked") + void prepareRepositoriesInDedicatedModeKeepsCentralAndTenantBootstrapTemplatesSeparated() throws IOException { + config.getApplication().setNamePrefix("testPrefix-"); + config.getMultiTenant().setUseDedicatedInstance(true); + config.getMultiTenant().getScmManager().setUrl("scmm.testhost/scm"); + config.getMultiTenant().setCentralArgocdNamespace("argocd"); + config.getFeatures().getArgocd().setOperator(true); + + ArgoCDRepoSetupTestContext testContext = createSetup(new FileSystemUtils()); + + testContext.setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = testContext.setup.clusterRepoLayout(); + ArgoCDRepoLayout tenantRepoLayout = testContext.setup.tenantRepoLayout(); + + File centralBootstrapFile = new File(clusterRepoLayout.applicationsDir(), "bootstrap.yaml"); + File tenantBootstrapFile = new File(tenantRepoLayout.applicationsDir(), "bootstrap.yaml"); + + assertThat(centralBootstrapFile).exists(); + assertThat(tenantBootstrapFile).exists(); + + Map centralBootstrapYaml = YAML_MAPPER.readValue(centralBootstrapFile, YAML_MAP_TYPE); + List> tenantBootstrapYaml; + try (MappingIterator> documents = YAML_MAPPER + .readerFor(YAML_MAP_TYPE) + .readValues(tenantBootstrapFile)) { + tenantBootstrapYaml = documents.readAll(); + } + + assertThat(centralBootstrapYaml) + .as("central bootstrap.yaml must contain exactly one central Application") + .isInstanceOf(Map.class); + + Map centralMetadata = (Map) centralBootstrapYaml.get("metadata"); + Map centralSpec = (Map) centralBootstrapYaml.get("spec"); + Map centralDestination = (Map) centralSpec.get("destination"); + Map centralSource = (Map) centralSpec.get("source"); + + assertThat(centralMetadata.get("name")).isEqualTo("testPrefix-bootstrap"); + assertThat(centralMetadata.get("namespace")).isEqualTo("argocd"); + assertThat(centralDestination.get("namespace")).isEqualTo("testPrefix-argocd"); + assertThat(centralSpec.get("project")).isEqualTo("testPrefix"); + assertThat(centralSource.get("path")).isEqualTo("apps/argocd/applications/"); + assertThat(centralSource.get("repoURL")) + .isEqualTo("scmm.testhost/scm/repo/testPrefix-argocd/cluster-resources.git"); + + assertThat(tenantBootstrapYaml) + .as("tenant bootstrap.yaml should contain tenant bootstrap Applications") + .isInstanceOf(List.class); + + List> tenantBootstrapDocuments = tenantBootstrapYaml; + + List tenantApplicationNames = tenantBootstrapDocuments.stream() + .map(document -> (String) ((Map) document.get( + "metadata")).get("name")) + .collect(Collectors.toList()); + + List tenantApplicationNamespaces = tenantBootstrapDocuments.stream() + .map(document -> (String) ((Map) document.get( + "metadata")).get("namespace")) + .collect(Collectors.toList()); + + List tenantApplicationProjects = tenantBootstrapDocuments.stream() + .map(document -> (String) ((Map) document.get( + "spec")).get("project")) + .collect(Collectors.toList()); + + assertThat(tenantApplicationNames).containsExactly("bootstrap", "projects"); + assertThat(tenantApplicationNamespaces).containsOnly("testPrefix-argocd"); + assertThat(tenantApplicationProjects).containsOnly("argocd"); + } + + @Test + void prepareRepositoriesInSingleInstanceDeletesMultiTenantFolder() { + config.getFeatures().getArgocd().setOperator(false); + config.getMultiTenant().setUseDedicatedInstance(false); + config.getApplication().setNetpols(true); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = setup.clusterRepoLayout(); + + assertThat(Path.of(clusterRepoLayout.multiTenantDir())).doesNotExist(); + } + + @Test + void prepareRepositoriesDeletesNetpolFileWhenNetpolsDisabled() { + config.getApplication().setNetpols(false); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = setup.clusterRepoLayout(); + + assertThat(Path.of(clusterRepoLayout.netpolFile())).doesNotExist(); + } + + @Test + void prepareRepositoriesKeepsNetpolFileWhenNetpolsEnabled() { + config.getApplication().setNetpols(true); + + ArgoCDRepoSetup setup = createSetup(new FileSystemUtils()).setup; + + setup.prepareRepositories(); + + ArgoCDRepoLayout clusterRepoLayout = setup.clusterRepoLayout(); + + assertThat(Path.of(clusterRepoLayout.netpolFile())).exists(); + } + + @Test + void prepareRepositoriesPreparesTenantBootstrapRepositoryInDedicatedMode() { + config.getMultiTenant().setUseDedicatedInstance(true); + + ArgoCDRepoSetupTestContext testContext = createSetup(new FileSystemUtils()); + + testContext.setup.prepareRepositories(); + + assertThat(Path.of(testContext.repositoryWorkspace.tenantBootstrapRootDir())).exists(); + assertThat(Path.of(testContext.repositoryWorkspace.tenantBootstrapRootDir()).toFile().listFiles()).isNotEmpty(); + } + + @Test + void prepareRepositoriesDoesNotPrepareTenantBootstrapRepositoryInSingleInstanceMode() { + config.getMultiTenant().setUseDedicatedInstance(false); + + ArgoCDRepoSetupTestContext testContext = createSetup(new FileSystemUtils()); + + testContext.setup.prepareRepositories(); + + assertThat(testContext.repositoryWorkspace.hasTenantBootstrapRepository()).isFalse(); + } + + static class ArgoCDRepoSetupTestContext { + ArgoCDRepoSetup setup; + RepositoryWorkspace repositoryWorkspace; + + ArgoCDRepoSetupTestContext(ArgoCDRepoSetup setup, RepositoryWorkspace repositoryWorkspace) { + this.setup = setup; + this.repositoryWorkspace = repositoryWorkspace; + } + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfigMapperTest.java new file mode 100644 index 000000000..89b92d26e --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/argocd/ArgoCDToolConfigMapperTest.java @@ -0,0 +1,200 @@ +package com.cloudogu.gitops.tools.core.argocd; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.application.credentials.CredentialsReference; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import org.junit.jupiter.api.Test; + +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class ArgoCDToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("tenant-a-"); + config.getApplication().setUsername("application-user"); + config.getApplication().setPassword("application-password"); + Credentials applicationCredentials = new Credentials(); + applicationCredentials.setSecretName("argocd-credentials"); + applicationCredentials.setSecretNamespace("gop-job"); + applicationCredentials.setUsernameKey("admin-user"); + applicationCredentials.setPasswordKey("admin-password"); + config.getApplication().setCredentials(applicationCredentials); + config.getApplication().getNamespaces().setDedicatedNamespaces(new LinkedHashSet<>(List.of( + "argocd", + "monitoring" + ))); + config.getApplication().getNamespaces().setTenantNamespaces(new LinkedHashSet<>(List.of("team-a", "team-b"))); + config.getApplication().setNetpols(true); + config.getApplication().setClusterAdmin(true); + config.getApplication().setInsecure(true); + // Intentionally differs from the DeploymentContext to verify derived values come from the context. + config.getApplication().setMirrorRepos(false); + config.getApplication().setOpenshift(false); + config.getApplication().setSkipCrds(true); + config.getFeatures().getArgocd().setActive(true); + config.getFeatures().getArgocd().setNamespace("gitops"); + config.getFeatures().getArgocd().setOperator(true); + config.getFeatures().getArgocd().setUrl("https://argocd.example.org"); + config.getFeatures().getArgocd().setEmailFrom("argocd@example.org"); + config.getFeatures().getArgocd().setEmailToAdmin("admins@example.org"); + config.getFeatures().getArgocd().setEnv(List.of(Map.of("name", "FIRST", "value", "one"))); + config.getFeatures().getArgocd().setResourceInclusionsCluster("https://cluster.example.org"); + config.getFeatures().getArgocd().setValues(Map.of("server", Map.of("replicas", 2))); + config.getFeatures().getArgocd().getOidc().setClientId("argocd-client"); + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setIssuer("production-issuer"); + config.getFeatures().getMail().setActive(true); + config.getFeatures().getMail().setSmtpAddress("smtp.example.org"); + config.getFeatures().getMail().setSmtpPort(2525); + config.getFeatures().getMail().setSmtpUser("smtp-user"); + config.getFeatures().getMail().setSmtpPassword("smtp-password"); + config.getFeatures().getMail().setCredentials( + new Credentials(null, null, "smtp-credentials", "gop-job", "smtp-user", "smtp-password") + ); + config.getFeatures().getMonitoring().setActive(true); + config.getFeatures().getMonitoring().setNamespace("observability"); + config.getFeatures().getSecrets().setActive(true); + config.getMultiTenant().setCentralArgocdNamespace("central-gitops"); + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setNamespace("source-control"); + config.getScm().setScmManager(scmManager); + Config.ContentSchema.HelmReleaseSchema helmRelease = new Config.ContentSchema.HelmReleaseSchema(); + helmRelease.setName("database"); + helmRelease.setChart("postgresql"); + helmRelease.setRepoURL("https://charts.example.org"); + config.getContent().setHelmReleases(List.of(helmRelease)); + + ArgoCDToolConfig actual = new ArgoCDToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(ArgoCDToolConfig.builder() + .active(true) + .namespace("tenant-a-gitops") + .username("application-user") + .password("application-password") + .credentials(new CredentialsReference( + "argocd-credentials", + "gop-job", + "admin-user", + "admin-password" + )) + .operator(true) + .activeNamespaces(List.of( + "argocd", + "monitoring", + "team-a", + "team-b" + )) + .smtpUser("smtp-user") + .smtpPassword("smtp-password") + .smtpCredentials(new CredentialsReference( + "smtp-credentials", + "gop-job", + "smtp-user", + "smtp-password" + )) + .values(Map.of("server", Map.of("replicas", 2))) + .multiTenant(true) + .netpols(true) + .tenantName("tenant-a") + .url("https://argocd.example.org") + .tenantNamespaces(List.of("team-a", "team-b")) + .centralNamespace("central-gitops") + .clusterAdmin(true) + .scmProviderType(ScmProviderType.SCM_MANAGER) + .templateConfig(Map.of( + "application", + Map.of( + "clusterAdmin", true, + "insecure", true, + "mirrorRepos", true, + "namePrefix", "tenant-a-", + "netpols", true, + "openshift", true, + "skipCrds", true + ), + "content", + Map.of( + "helmReleases", + List.of(Map.of("repoURL", "https://charts.example.org")) + ), + "features", + Map.of( + "argocd", + Map.of( + "emailFrom", + "argocd@example.org", + "emailToAdmin", + "admins@example.org", + "env", + List.of(Map.of("name", "FIRST", "value", "one")), + "namespace", + "gitops", + "oidc", + Map.of( + "providerName", "Keycloak", + "issuerUrl", "", + "clientId", "argocd-client", + "clientSecret", "", + "scopes", List.of("openid", "profile", "email"), + "adminGroupName", "", + "enabled", false + ), + "operator", + true, + "resourceInclusionsCluster", + "https://cluster.example.org", + "url", + "https://argocd.example.org" + ), + "certManager", + Map.of("active", true, "issuer", "production-issuer"), + "mail", + Map.of( + "active", true, + "smtpAddress", "smtp.example.org", + "smtpPasswordConfigured", true, + "smtpPort", 2525, + "smtpUserConfigured", true + ), + "monitoring", + Map.of("active", true, "namespace", "observability"), + "secrets", + Map.of("active", true) + ), + "multiTenant", + Map.of("centralArgocdNamespace", "central-gitops"), + "scm", + Map.of( + "scmManager", Map.of("namespace", "source-control"), + "scmProviderType", ScmProviderType.SCM_MANAGER + ) + )) + .rbacTemplateConfig(Map.of( + "application", Map.of("openshift", true), + "features", Map.of( + "monitoring", Map.of("active", true), + "secrets", Map.of("active", true) + ) + )) + .build()); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.MULTI_TENANT, + DeploymentContext.ScmManagerDeploymentMode.INTERNAL, + true, + DeploymentContext.ClusterDistribution.OPENSHIFT + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerSetupTest.java b/src/test/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerSetupTest.java new file mode 100644 index 000000000..1ca5ea5ab --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerSetupTest.java @@ -0,0 +1,474 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.application.context.ContextBuilder; +import com.cloudogu.gitops.application.repository.RepositoryWorkspace; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.Credentials; +import com.cloudogu.gitops.infrastructure.deployment.Deployer; +import com.cloudogu.gitops.infrastructure.deployment.DeploymentStrategy; +import com.cloudogu.gitops.infrastructure.deployment.HelmStrategy; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.GitProvider; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.ScmManagerProvider; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.PluginApi; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApi; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerApiClient; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.ScmManagerUser; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.UsersApi; +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import com.cloudogu.gitops.utils.FileSystemUtils; +import com.cloudogu.gitops.utils.Tuple; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import okhttp3.internal.http.RealResponseBody; +import okio.BufferedSource; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import retrofit2.Call; +import retrofit2.Response; + +import java.io.IOException; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyMap; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class ScmManagerSetupTest { + + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + + private final ScmManagerProvider scmManager = mock(ScmManagerProvider.class); + + private final Deployer deployer = mock(Deployer.class); + private final HelmStrategy helmStrategy = mock(HelmStrategy.class); + + private final GitProvider tenantProvider = mock(GitProvider.class); + private final GitProvider centralProvider = mock(GitProvider.class); + + private final GitRepo clusterResourcesRepo = mock(GitRepo.class); + private final GitRepo tenantBootstrapRepo = mock(GitRepo.class); + + private final ScmManagerApiClient apiClient = mock(ScmManagerApiClient.class); + private final PluginApi pluginApi = mock(PluginApi.class); + private final ScmManagerApi generalApi = mock(ScmManagerApi.class); + private final K8sClient k8sClient = mock(K8sClient.class); + private final FileSystemUtils fileSystemUtils = spy(new FileSystemUtils()); + + private final Config config = Config.fromMap(Map.of( + "application", Map.of( + "namePrefix", "test", + "insecure", true + ), + "jenkins", Map.of( + "active", false, + "urlForScm", "http://jenkins.jenkins.svc.cluster.local" + ), + "scm", Map.of( + "scmManager", Map.ofEntries( + Map.entry("internal", true), + Map.entry("url", ""), + Map.entry("namespace", "scm-manager"), + Map.entry("username", "admin"), + Map.entry("password", "admin"), + Map.entry( + "helm", Map.of( + "chart", "scm-manager", + "repoURL", "https://packages.scm-manager.org/repository/helm-v2-releases/", + "version", "3.11.2", + "values", Map.of() + ) + ), + Map.entry("urlForJenkins", "http://scmm.scm-manager.svc.cluster.local/scm"), + Map.entry("ingress", "scmm.master.localhost"), + Map.entry("skipRestart", false), + Map.entry("skipPlugins", false), + Map.entry("gitOpsUsername", "gitops"), + Map.entry( + "credentials", Map.of( + "username", "admin", + "password", "admin" + ) + ) + ) + ) + )); + + @BeforeEach + void setUp() throws IOException { + when(scmManager.getCredentials()).thenReturn(new Credentials("admin", "admin")); + clusterResourcesRepo.setGitProvider(centralProvider); + tenantBootstrapRepo.setGitProvider(tenantProvider); + + doReturn(centralProvider).when(clusterResourcesRepo).getGitProvider(); + doReturn(tenantProvider).when(tenantBootstrapRepo).getGitProvider(); + + doReturn("argocd/cluster-resources") + .when(clusterResourcesRepo) + .getRepoTarget(); + + doReturn("argocd/cluster-resources") + .when(tenantBootstrapRepo) + .getRepoTarget(); + + doReturn(createTempDir("cluster-resources")) + .when(clusterResourcesRepo) + .getAbsoluteLocalRepoTmpDir(); + + doReturn(createTempDir("tenant-bootstrap")) + .when(tenantBootstrapRepo) + .getAbsoluteLocalRepoTmpDir(); + } + + @Test + @SuppressWarnings("unchecked") + void helmChartIsInstalledCorrectly() throws IOException { + when(scmManager.getScmmConfig()).thenReturn(config.getScm().getScmManager()); + when(scmManager.getCredentials()).thenReturn(new Credentials("resolved-admin", "SCMM_SECRET_SENTINEL")); + when(deployer.getHelmStrategy()).thenReturn(helmStrategy); + config.getScm().getScmManager().setScmmImage("localhost:5000/proxy/scm-manager:custom"); + // Usually ApplicationConfigurator modifies the namePrefix and sets it to "namePrefix-" + config.getApplication().setNamePrefix(config.getApplication().getNamePrefix() + "-"); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + new RepositoryWorkspace(clusterResourcesRepo), + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + scmManagerSetup.setupHelm(); + verify(fileSystemUtils).writeTempFile(anyMap()); + + ArgumentCaptor valuesPathCaptor = ArgumentCaptor.forClass(Path.class); + verify(helmStrategy).deployFeature( + eq("https://packages.scm-manager.org/repository/helm-v2-releases/"), + eq("scm-manager"), + eq("scm-manager"), + eq("3.11.2"), + eq("test-scm-manager"), + eq("test-scmm"), + valuesPathCaptor.capture(), + eq(DeploymentStrategy.RepoType.HELM) + ); + + Map values = YAML_MAPPER.readValue(valuesPathCaptor.getValue().toFile(), YAML_MAP_TYPE); + Map image = (Map) values.get("image"); + assertThat(image.get("repository")).isEqualTo("localhost:5000/proxy/scm-manager"); + assertThat(image.get("tag")).isEqualTo("custom"); + + verify(k8sClient).createNamespace("test-scm-manager"); + verify(k8sClient).createSecret( + "generic", + ScmManagerSetup.CREDENTIALS_SECRET_NAME, + "test-scm-manager", + new Tuple<>("SCM_WEBAPP_INITIALUSER", "resolved-admin"), + new Tuple<>("SCM_WEBAPP_INITIALPASSWORD", "SCMM_SECRET_SENTINEL") + ); + + String extraEnvFrom = (String) values.get("extraEnvFrom"); + assertThat(extraEnvFrom) + .contains("name: scm-manager-credentials") + .doesNotContain("resolved-admin") + .doesNotContain("SCMM_SECRET_SENTINEL"); + } + + @Test + void defaultUsersUseRuntimePassword() throws ReflectiveOperationException, IOException { + UsersApi usersApi = mock(UsersApi.class); + @SuppressWarnings("unchecked") + Call addUserCall = mock(Call.class); + @SuppressWarnings("unchecked") + Call permissionCall = mock(Call.class); + + when(scmManager.getApiClient()).thenReturn(apiClient); + when(scmManager.getCredentials()).thenReturn(new Credentials("resolved-admin", "runtime-password")); + when(apiClient.usersApi()).thenReturn(usersApi); + when(usersApi.addUser(any(ScmManagerUser.class))).thenReturn(addUserCall); + when(usersApi.setPermissionForUser(anyString(), anyMap())).thenReturn(permissionCall); + when(addUserCall.execute()).thenReturn(Response.success(null)); + when(permissionCall.execute()).thenReturn(Response.success(null)); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + new RepositoryWorkspace(clusterResourcesRepo), + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + invokePrivateAddDefaultUsers(scmManagerSetup); + + ArgumentCaptor userCaptor = ArgumentCaptor.forClass(ScmManagerUser.class); + verify(usersApi, times(2)).addUser(userCaptor.capture()); + assertThat(userCaptor.getAllValues()) + .extracting(ScmManagerUser::getPassword) + .containsOnly("runtime-password"); + } + + @Test + @SuppressWarnings("unchecked") + void helmValuesContainCertManagerIngressConfiguration() throws IOException { + when(scmManager.getScmmConfig()).thenReturn(config.getScm().getScmManager()); + when(deployer.getHelmStrategy()).thenReturn(helmStrategy); + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setIssuer("cluster-selfsigned"); + // Usually ApplicationConfigurator modifies the namePrefix and sets it to "namePrefix-" + config.getApplication().setNamePrefix(config.getApplication().getNamePrefix() + "-"); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + new RepositoryWorkspace(clusterResourcesRepo), + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + scmManagerSetup.setupHelm(); + + ArgumentCaptor valuesPathCaptor = ArgumentCaptor.forClass(Path.class); + verify(helmStrategy).deployFeature( + eq("https://packages.scm-manager.org/repository/helm-v2-releases/"), + eq("scm-manager"), + eq("scm-manager"), + eq("3.11.2"), + eq("test-scm-manager"), + eq("test-scmm"), + valuesPathCaptor.capture(), + eq(DeploymentStrategy.RepoType.HELM) + ); + + Map values = YAML_MAPPER.readValue(valuesPathCaptor.getValue().toFile(), YAML_MAP_TYPE); + Map ingress = (Map) values.get("ingress"); + List> tls = (List>) ingress.get("tls"); + Map tlsEntry = tls.get(0); + Map annotations = (Map) ingress.get("annotations"); + + assertThat(annotations.get("cert-manager.io/cluster-issuer")).isEqualTo("cluster-selfsigned"); + assertThat(tlsEntry.get("secretName")).isEqualTo("scm-manager-tls"); + assertThat((List) tlsEntry.get("hosts")).containsExactly("scmm.master.localhost"); + } + + @Test + void scmManagerPluginsAreInstalledCorrectly() throws IOException, ReflectiveOperationException { + when(scmManager.getScmmConfig()).thenReturn(config.getScm().getScmManager()); + when(scmManager.getApiClient()).thenReturn(apiClient); + + @SuppressWarnings("unchecked") + Call apiCall = mock(Call.class); + + when(pluginApi.install(any(String.class), anyBoolean())).thenReturn(apiCall); + when(generalApi.checkScmmAvailable()).thenReturn(apiCall); + + when(apiClient.pluginApi()).thenReturn(pluginApi); + when(apiClient.generalApi()).thenReturn(generalApi); + + when(apiCall.execute()).thenReturn(Response.success(null)); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + new RepositoryWorkspace(clusterResourcesRepo), + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + invokePrivateInstallScmmPlugins(scmManagerSetup); + + verify(pluginApi, times(10)).install(any(String.class), anyBoolean()); + } + + @Test + void stopsWaitingWhenInterrupted() throws IOException { + when(scmManager.getApiClient()).thenReturn(apiClient); + when(apiClient.generalApi()).thenReturn(generalApi); + + @SuppressWarnings("unchecked") + Call apiCall = mock(Call.class); + Response response = Response.error( + 503, + new RealResponseBody("text/plain", 0, mock(BufferedSource.class)) + ); + when(generalApi.checkScmmAvailable()).thenReturn(apiCall); + when(apiCall.execute()).thenReturn(response); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + new RepositoryWorkspace(clusterResourcesRepo), + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + Thread.currentThread().interrupt(); + try { + assertThatThrownBy(() -> scmManagerSetup.waitForScmmAvailable(10, 1000, 0)) + .isInstanceOf(IllegalStateException.class) + .hasMessage("Interrupted while waiting for SCM-Manager") + .hasCauseInstanceOf(InterruptedException.class); + assertThat(Thread.currentThread().isInterrupted()).isTrue(); + } finally { + Thread.interrupted(); + } + } + + @Test + void prepareBootstrapRepositoriesAfterScmManagerDeploymentInitializesClusterResourcesRepository() + throws GitAPIException, IOException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepo); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + workspace, + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + scmManagerSetup.prepareBootstrapRepositoriesAfterScmManagerDeployment(); + + verify(centralProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for basic cluster-resources", + false + ); + + verify(clusterResourcesRepo).initLocalRepoIfNeeded(); + verify(clusterResourcesRepo).checkoutRemoteMainIfLocalMainMissing(); + verify(clusterResourcesRepo, never()).commitAndPush(anyString()); + } + + @Test + void pushBootstrapRepositoriesAfterScmManagerDeploymentPushesClusterResourcesRepository() + throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace(clusterResourcesRepo); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + workspace, + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + scmManagerSetup.pushBootstrapRepositoriesAfterScmManagerDeployment(); + + verify(clusterResourcesRepo).commitAndPush("Bootstrap cluster-resources repository after SCM-Manager deployment"); + } + + @Test + void prepareBootstrapRepositoriesAfterScmManagerDeploymentInitializesBothRepositoriesInDedicatedMode() + throws GitAPIException, IOException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepo, + tenantBootstrapRepo + ); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + workspace, + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + scmManagerSetup.prepareBootstrapRepositoriesAfterScmManagerDeployment(); + + verify(centralProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for basic cluster-resources", + false + ); + verify(tenantProvider).createRepository( + "argocd/cluster-resources", + "GitOps repo for tenant bootstrap resources", + false + ); + + verify(clusterResourcesRepo).initLocalRepoIfNeeded(); + verify(clusterResourcesRepo).checkoutRemoteMainIfLocalMainMissing(); + verify(clusterResourcesRepo, never()).commitAndPush(anyString()); + + verify(tenantBootstrapRepo).initLocalRepoIfNeeded(); + verify(tenantBootstrapRepo).checkoutRemoteMainIfLocalMainMissing(); + verify(tenantBootstrapRepo, never()).commitAndPush(anyString()); + } + + @Test + void pushBootstrapRepositoriesAfterScmManagerDeploymentPushesBothRepositoriesInDedicatedMode() + throws GitAPIException { + RepositoryWorkspace workspace = new RepositoryWorkspace( + clusterResourcesRepo, + tenantBootstrapRepo + ); + + ScmManagerSetup scmManagerSetup = new ScmManagerSetup( + scmManager, + deployer, + new ContextBuilder(config).build(), + workspace, + fileSystemUtils, + new ScmManagerToolConfigMapper(config).map(new ContextBuilder(config).build()), + k8sClient + ); + + scmManagerSetup.pushBootstrapRepositoriesAfterScmManagerDeployment(); + + verify(clusterResourcesRepo).commitAndPush("Bootstrap cluster-resources repository after SCM-Manager deployment"); + verify(tenantBootstrapRepo).commitAndPush("Bootstrap tenant repository after SCM-Manager deployment"); + } + + private static void invokePrivateInstallScmmPlugins(ScmManagerSetup scmManagerSetup) + throws ReflectiveOperationException { + Method method = ScmManagerSetup.class.getDeclaredMethod("installScmmPlugins"); + method.setAccessible(true); + method.invoke(scmManagerSetup); + } + + private static void invokePrivateAddDefaultUsers(ScmManagerSetup scmManagerSetup) + throws ReflectiveOperationException { + Method method = ScmManagerSetup.class.getDeclaredMethod("addDefaultUsers"); + method.setAccessible(true); + method.invoke(scmManagerSetup); + } + + private static String createTempDir(String prefix) throws IOException { + return Files.createTempDirectory(prefix).toFile().getCanonicalPath(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfigMapperTest.java b/src/test/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfigMapperTest.java new file mode 100644 index 000000000..a7f93bd83 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/tools/core/scmmanager/ScmManagerToolConfigMapperTest.java @@ -0,0 +1,132 @@ +package com.cloudogu.gitops.tools.core.scmmanager; + +import com.cloudogu.gitops.application.context.DeploymentContext; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.config.scm.ScmTenantSchema; +import com.cloudogu.gitops.config.scm.util.ScmProviderType; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.cloudogu.gitops.tools.common.ImagePullSecretConfig; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class ScmManagerToolConfigMapperTest { + + @Test + void mapsAllRelevantValuesFromDeploymentContextAndConfig() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getApplication().setLocalHelmChartFolder("/charts"); + config.getRegistry().setCreateImagePullSecrets(true); + config.getRegistry().setProxyUrl("proxy.example.org"); + config.getRegistry().setUrl("registry.example.org"); + config.getRegistry().setProxyUsername("proxy-user"); + config.getRegistry().setReadOnlyUsername("read-only-user"); + config.getRegistry().setUsername("registry-user"); + config.getRegistry().setProxyPassword("proxy-password"); + config.getRegistry().setReadOnlyPassword("read-only-password"); + config.getRegistry().setPassword("registry-password"); + config.getJenkins().setActive(true); + config.getJenkins().setUrlForScm("http://jenkins.automation.svc"); + config.getFeatures().getCertManager().setActive(true); + config.getFeatures().getCertManager().setIssuer("production-issuer"); + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setInternal(true); + scmManager.setNamespace("source-control"); + scmManager.setIngress("scm.example.org"); + scmManager.setUsername("scm-user"); + scmManager.setPassword("scm-password"); + scmManager.setGitOpsUsername("gitops-user"); + scmManager.setSkipPlugins(true); + scmManager.setSkipRestart(true); + scmManager.setScmmImage("scm-manager:custom"); + scmManager.getHelm().setRepoURL("https://scm-chart.example.org"); + scmManager.getHelm().setChart("scm-chart"); + scmManager.getHelm().setVersion("8.9.10"); + scmManager.getHelm().setValues(Map.of("replicas", 2)); + config.getScm().setScmManager(scmManager); + + ScmManagerToolConfig actual = new ScmManagerToolConfigMapper(config).map(context()); + + assertThat(actual).isEqualTo(ScmManagerToolConfig.builder() + .active(true) + .multiTenant(true) + .namePrefix("test-") + .namespace("test-source-control") + .releaseName("test-scmm") + .ingress("scm.example.org") + .gitOpsUsername("gitops-user") + .skipPlugins(true) + .skipRestart(true) + .jenkinsActive(true) + .jenkinsUrl("http://jenkins.automation.svc") + .helm(HelmChartConfig.builder() + .repoURL("https://scm-chart.example.org") + .chart("scm-chart") + .version("8.9.10") + .values(Map.of("replicas", 2)) + .localHelmChartFolder("/charts") + .build()) + .imagePullSecret(ImagePullSecretConfig.builder() + .create(true) + .proxyUrl( + "proxy.example.org") + .url( + "registry.example.org") + .proxyUsername( + "proxy-user") + .readOnlyUsername( + "read-only-user") + .username("registry-user") + .proxyPassword( + "proxy-password") + .readOnlyPassword( + "read-only-password") + .password( + "registry-password") + .build()) + .templateConfig(Map.of( + "features", + Map.of( + "certManager", Map.of( + "active", true, + "issuer", "production-issuer" + ) + ), + "registry", + Map.of("createImagePullSecrets", true), + "scm", + Map.of( + "scmManager", + Map.of("scmmImage", "scm-manager:custom") + ) + )) + .build()); + } + + @Test + void doesNotAddTheApplicationPrefixTwice() { + Config config = new Config(); + config.getApplication().setNamePrefix("test-"); + config.getScm().setScmProviderType(ScmProviderType.SCM_MANAGER); + ScmTenantSchema.ScmManagerTenantConfig scmManager = new ScmTenantSchema.ScmManagerTenantConfig(); + scmManager.setNamespace("test-source-control"); + config.getScm().setScmManager(scmManager); + + ScmManagerToolConfig actual = new ScmManagerToolConfigMapper(config).map(context()); + + assertThat(actual.namespace()).isEqualTo("test-source-control"); + } + + private static DeploymentContext context() { + return new DeploymentContext( + DeploymentContext.TenantMode.MULTI_TENANT, + DeploymentContext.ScmManagerDeploymentMode.INTERNAL, + false, + DeploymentContext.ClusterDistribution.KUBERNETES + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/AirGappedUtilsTest.java b/src/test/java/com/cloudogu/gitops/utils/AirGappedUtilsTest.java new file mode 100644 index 000000000..29685a57f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/AirGappedUtilsTest.java @@ -0,0 +1,276 @@ +package com.cloudogu.gitops.utils; + +import com.cloudogu.gitops.application.orchestration.GitHandler; +import com.cloudogu.gitops.config.Config; +import com.cloudogu.gitops.infrastructure.git.GitRepo; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.Permission; +import com.cloudogu.gitops.infrastructure.git.providers.scmmanager.api.Repository; +import com.cloudogu.gitops.infrastructure.helm.HelmClient; +import com.cloudogu.gitops.testhelper.git.GitHandlerForTests; +import com.cloudogu.gitops.testhelper.git.ScmManagerProviderMock; +import com.cloudogu.gitops.testhelper.git.TestGitRepoFactory; +import com.cloudogu.gitops.testhelper.git.TestScmManagerApiClient; +import com.cloudogu.gitops.tools.common.HelmChartConfig; +import com.fasterxml.jackson.core.type.TypeReference; +import com.fasterxml.jackson.dataformat.yaml.YAMLMapper; +import org.eclipse.jgit.api.Git; +import org.eclipse.jgit.api.errors.GitAPIException; +import org.eclipse.jgit.lib.Ref; +import org.eclipse.jgit.revwalk.RevCommit; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class AirGappedUtilsTest { + + private static final TypeReference> YAML_MAP_TYPE = new TypeReference<>() { + }; + private static final YAMLMapper YAML_MAPPER = new YAMLMapper(); + + private Path rootChartsFolder; + private Config config; + private HelmChartConfig helmConfig; + private TestGitRepoFactory gitRepoFactory; + private FileSystemUtils fileSystemUtils; + private TestScmManagerApiClient scmmApiClient; + private HelmClient helmClient; + private GitHandler gitHandler; + + @BeforeEach + void setUp() throws IOException { + rootChartsFolder = Files.createTempDirectory(getClass().getSimpleName()); + + Map configMap = new LinkedHashMap<>(); + configMap.put( + "application", Map.of( + "gitName", "Cloudogu", + "gitEmail", "hello@cloudogu.com" + ) + ); + configMap.put( + "scm", Map.of( + "scmManager", Map.of("url", "") + ) + ); + config = Config.fromMap(configMap); + + helmConfig = HelmChartConfig.builder() + .chart("kube-prometheus-stack") + .repoURL("https://kube-prometheus-stack-repo-url") + .version("58.2.1") + .localHelmChartFolder(rootChartsFolder.toString()) + .build(); + + fileSystemUtils = new FileSystemUtils(); + gitRepoFactory = new TestGitRepoFactory(config, fileSystemUtils); + scmmApiClient = new TestScmManagerApiClient(config); + helmClient = mock(HelmClient.class); + gitHandler = new GitHandlerForTests(new ScmManagerProviderMock()); + + var response = TestScmManagerApiClient.mockSuccessfulResponse(201); + when(scmmApiClient.getRepositoryApi().create(any(Repository.class), anyBoolean())).thenReturn(response); + when(scmmApiClient.getRepositoryApi().createPermission(anyString(), anyString(), any(Permission.class))) + .thenReturn(response); + } + + @Test + void preparesReposForAirGappedUse() throws IOException, GitAPIException { + setupForAirgappedUse(); + + String actualRepoNamespaceAndName = createAirGappedUtils().mirrorHelmRepoToGit(helmConfig); + + assertThat(actualRepoNamespaceAndName) + .isEqualTo(GitRepo.NAMESPACE_3RD_PARTY_DEPENDENCIES + "/kube-prometheus-stack"); + assertAirGapped(); + verify(helmClient).template("kube-prometheus-stack", rootChartsFolder + "/kube-prometheus-stack"); + } + + @Test + void failsWhenUnableToResolveVersionOfDependencies() throws IOException { + setupForAirgappedUse(Collections.emptyMap()); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> createAirGappedUtils().mirrorHelmRepoToGit(helmConfig) + ); + + assertThat(exception.getMessage()).isEqualTo( + "Unable to determine proper version for dependency grafana (version: 7.3.*) " + + "from repo 3rd-party-dependencies/kube-prometheus-stack" + ); + } + + @Test + void alsoWorksForChartsWithoutDependencies() throws IOException { + setupForAirgappedUse(null, Collections.emptyList()); + createAirGappedUtils().mirrorHelmRepoToGit(helmConfig); + + GitRepo prometheusRepo = gitRepoFactory.getRepos().get("3rd-party-dependencies/kube-prometheus-stack"); + Map actualPrometheusChartYaml = YAML_MAPPER.readValue( + Path.of(prometheusRepo.getAbsoluteLocalRepoTmpDir(), "Chart.yaml").toFile(), + YAML_MAP_TYPE + ); + + Object dependencies = actualPrometheusChartYaml.get("dependencies"); + assertThat(dependencies).isNull(); + } + + @Test + void failsForInvalidHelmCharts() throws IOException { + setupForAirgappedUse(); + + RuntimeException expectedException = new RuntimeException(); + doThrow(expectedException).when(helmClient).template(anyString(), anyString()); + + RuntimeException exception = assertThrows( + RuntimeException.class, + () -> createAirGappedUtils().mirrorHelmRepoToGit(helmConfig) + ); + + assertThat(exception.getMessage()) + .isEqualTo("Helm chart in folder " + rootChartsFolder + "/kube-prometheus-stack seems invalid."); + assertThat(exception.getCause()).isSameAs(expectedException); + } + + protected void setupForAirgappedUse() throws IOException { + setupForAirgappedUse(null, null); + } + + protected void setupForAirgappedUse(Map chartLock) throws IOException { + setupForAirgappedUse(chartLock, null); + } + + protected void setupForAirgappedUse( + Map chartLock, + List> dependencies + ) throws IOException { + Path sourceChart = rootChartsFolder.resolve("kube-prometheus-stack"); + Files.createDirectories(sourceChart); + + Map prometheusChartYaml = new LinkedHashMap<>(); + prometheusChartYaml.put("version", "1.2.3"); + prometheusChartYaml.put("name", "kube-prometheus-stack-chart"); + prometheusChartYaml.put( + "dependencies", List.of( + Map.of( + "condition", "crds.enabled", + "name", "crds", + "repository", "", + "version", "0.0.0" + ), + Map.of( + "condition", "grafana.enabled", + "name", "grafana", + "repository", "https://grafana-repo-url", + "version", "7.3.*" + ) + ) + ); + + if (dependencies != null) { + if (dependencies.isEmpty()) { + prometheusChartYaml.remove("dependencies"); + } else { + prometheusChartYaml.put("dependencies", dependencies); + } + } + + fileSystemUtils.writeYaml(prometheusChartYaml, sourceChart.resolve("Chart.yaml").toFile()); + + if (chartLock == null) { + chartLock = Map.of( + "dependencies", List.of( + Map.of( + "name", "crds", + "repository", "", + "version", "0.0.0" + ), + Map.of( + "name", "grafana", + "repository", "https://grafana.github.io/helm-charts", + "version", "7.3.9" + ) + ) + ); + } + fileSystemUtils.writeYaml(chartLock, sourceChart.resolve("Chart.lock").toFile()); + } + + @SuppressWarnings("unchecked") + protected void assertAirGapped() throws IOException, GitAPIException { + GitRepo prometheusRepo = gitRepoFactory.getRepos().get("3rd-party-dependencies/kube-prometheus-stack"); + assertThat(prometheusRepo).isNotNull(); + assertThat(Path.of(prometheusRepo.getAbsoluteLocalRepoTmpDir(), "Chart.lock")).doesNotExist(); + + Map actualPrometheusChartYaml = YAML_MAPPER.readValue( + Path.of(prometheusRepo.getAbsoluteLocalRepoTmpDir(), "Chart.yaml").toFile(), + YAML_MAP_TYPE + ); + assertThat(actualPrometheusChartYaml.get("name")).isEqualTo("kube-prometheus-stack-chart"); + + List> dependencies = + (List>) actualPrometheusChartYaml.get("dependencies"); + assertThat(dependencies).hasSize(2); + assertThat(dependencies.get(0).get("name")).isEqualTo("crds"); + assertThat(dependencies.get(0).get("version")).isEqualTo("0.0.0"); + assertThat(dependencies.get(0).get("repository")).isEqualTo(""); + assertThat(dependencies.get(1).get("name")).isEqualTo("grafana"); + assertThat(dependencies.get(1).get("version")).isEqualTo("7.3.9"); + assertThat(dependencies.get(1).get("repository")).isEqualTo(""); + + assertHelmRepoCommits( + prometheusRepo, + "1.2.3", + "Chart kube-prometheus-stack-chart, version: 1.2.3\n\n" + + "Source: https://kube-prometheus-stack-repo-url\n" + + "Dependencies localized to run in air-gapped environments" + ); + + verify(prometheusRepo).createRepositoryAndSetPermission( + eq("Mirror of Helm chart kube-prometheus-stack from https://kube-prometheus-stack-repo-url"), + eq(false) + ); + } + + void assertHelmRepoCommits(GitRepo repo, String expectedTag, String expectedCommitMessage) + throws IOException, GitAPIException { + Iterable commitIterable = Git.open(new File(repo.getAbsoluteLocalRepoTmpDir())) + .log() + .setMaxCount(1) + .all() + .call(); + List commits = new ArrayList<>(); + commitIterable.forEach(commits::add); + + assertThat(commits.size()).isEqualTo(1); + assertThat(commits.get(0).getFullMessage()).isEqualTo(expectedCommitMessage); + + List tags = Git.open(new File(repo.getAbsoluteLocalRepoTmpDir())).tagList().call(); + assertThat(tags.size()).isEqualTo(1); + assertThat(tags.get(0).getName()).isEqualTo("refs/tags/" + expectedTag); + } + + AirGappedUtils createAirGappedUtils() { + return new AirGappedUtils(gitRepoFactory, fileSystemUtils, helmClient, gitHandler); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/AllowlistFreemarkerObjectWrapperTest.java b/src/test/java/com/cloudogu/gitops/utils/AllowlistFreemarkerObjectWrapperTest.java new file mode 100644 index 000000000..de627707f --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/AllowlistFreemarkerObjectWrapperTest.java @@ -0,0 +1,104 @@ +package com.cloudogu.gitops.utils; + +import freemarker.core.InvalidReferenceException; +import freemarker.template.Configuration; +import freemarker.template.TemplateException; +import freemarker.template.TemplateModelException; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Map; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class AllowlistFreemarkerObjectWrapperTest { + + @Test + void shouldAllowAccessToWhitelistedStaticModels() throws TemplateModelException { + var wrapper = new AllowListFreemarkerObjectWrapper( + Configuration.VERSION_2_3_32, + Set.of("com.cloudogu.gitops.utils.DockerImageParser") + ); + var staticModels = wrapper.getStaticModels(); + + assertNotNull(staticModels.get("com.cloudogu.gitops.utils.DockerImageParser")); + assertNull(staticModels.get("java.lang.Integer")); + assertNull(staticModels.get("java.lang.String")); + } + + @Test + void shouldDenyAccessToNonWhitelistedStaticModels() throws TemplateModelException { + var wrapper = new AllowListFreemarkerObjectWrapper( + Configuration.VERSION_2_3_32, + Set.of("java.lang.String") + ); + var staticModels = wrapper.getStaticModels(); + + assertNull(staticModels.get("java.lang.Integer")); + assertNotNull(staticModels.get("java.lang.String")); + assertNull(staticModels.get("com.cloudogu.gitops.utils.DockerImageParser")); + } + + @Test + void shouldReturnTrueForIsEmptyWhenAllowlistIsEmpty() throws TemplateModelException { + var wrapper = new AllowListFreemarkerObjectWrapper(Configuration.VERSION_2_3_32, Set.of()); + var staticModels = wrapper.getStaticModels(); + + assertTrue(staticModels.isEmpty()); + } + + @Test + void templatingOnlyWorksForWhitelistedStatics() throws IOException { + String templateText = """ + <#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> + <#assign imageObject = DockerImageParser.parse('test:latest')> + <#assign staticsTests=statics['System']> + <#assign imageObject = staticsTests.exit()> + """; + + Map model = Map.of( + "statics", + new AllowListFreemarkerObjectWrapper( + Configuration.VERSION_2_3_32, + Set.of("com.cloudogu.gitops.utils.DockerImageParser") + ).getStaticModels() + ); + File tempInputFile = File.createTempFile("test", ".ftl.yaml"); + Files.writeString(tempInputFile.toPath(), templateText); + + InvalidReferenceException exception = assertThrows( + InvalidReferenceException.class, + () -> new TemplatingEngine().replaceTemplates(tempInputFile, model) + ); + + assertTrue(exception.getMessage().contains("System"), "Exception message should mention 'System'"); + } + + @Test + void templatingInFtlFilesWorksCorrectlyWithWhitelistedStaticModels() throws IOException, TemplateException { + String templateText = """ + <#assign DockerImageParser=statics['com.cloudogu.gitops.utils.DockerImageParser']> + <#assign imageObject = DockerImageParser.parse('test:latest')> + <#assign staticsTests=statics['java.lang.Math']> + <#assign number = staticsTests.round(3.14)> + """; + + Map model = Map.of( + "statics", + new AllowListFreemarkerObjectWrapper( + Configuration.VERSION_2_3_32, + Set.of("java.lang.Math", "com.cloudogu.gitops.utils.DockerImageParser") + ).getStaticModels() + ); + File tempInputFile = File.createTempFile("test", ".ftl.yaml"); + Files.writeString(tempInputFile.toPath(), templateText); + + new TemplatingEngine().replaceTemplates(tempInputFile, model); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/ClusterResourcesCopyFilterTest.java b/src/test/java/com/cloudogu/gitops/utils/ClusterResourcesCopyFilterTest.java new file mode 100644 index 000000000..e0b88bfde --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/ClusterResourcesCopyFilterTest.java @@ -0,0 +1,84 @@ +package com.cloudogu.gitops.utils; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.File; +import java.io.FileFilter; +import java.io.IOException; +import java.nio.file.Files; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +class ClusterResourcesCopyFilterTest { + + @TempDir + File tempDir; + + @Test + void forSubDirIncludesSelectedSubdirAndTraversalParentsOnly() throws IOException { + File root = createClusterResourcesRoot(); + + FileFilter filter = ClusterResourcesCopyFilter.forSubDir(root.getPath(), "apps/monitoring"); + + assertThat(filter.accept(new File(root, "apps"))).isTrue(); + assertThat(filter.accept(new File(root, "apps/monitoring"))).isTrue(); + assertThat(filter.accept(new File( + root, + "apps/monitoring/misc/dashboard/prometheus-dashboard.ftl.yaml" + ))).isTrue(); + assertThat(filter.accept(new File(root, "apps/ingress/values.yaml"))).isFalse(); + } + + @Test + void forSubDirsExcludesToolTemplateDirectoriesExceptArgoCDHelmTemplates() throws IOException { + File root = createClusterResourcesRoot(); + + FileFilter filter = ClusterResourcesCopyFilter.forSubDirs( + root.getPath(), + List.of("apps/monitoring", "apps/argocd") + ); + + assertThat(filter.accept(new File( + root, + "apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml" + ))).isFalse(); + assertThat(filter.accept(new File(root, "apps/argocd/templates/project.ftl.yaml"))).isFalse(); + assertThat(filter.accept(new File(root, "apps/argocd/argocd/templates/allow-namespaces.ftl.yaml"))).isTrue(); + } + + @Test + void forSubDirsAllowsEverythingWhenNoSubdirsAreProvided() throws IOException { + File root = createClusterResourcesRoot(); + + FileFilter filter = ClusterResourcesCopyFilter.forSubDirs(root.getPath(), List.of()); + + assertThat(filter.accept(new File( + root, + "apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml" + ))).isTrue(); + assertThat(filter.accept(new File(root, "apps/ingress/values.yaml"))).isTrue(); + } + + private File createClusterResourcesRoot() throws IOException { + File root = new File(tempDir, "cluster-resources"); + + List paths = List.of( + "apps/monitoring/misc/dashboard/prometheus-dashboard.ftl.yaml", + "apps/monitoring/templates/prometheus-stack-helm-values.ftl.yaml", + "apps/argocd/templates/project.ftl.yaml", + "apps/argocd/argocd/templates/allow-namespaces.ftl.yaml", + "apps/jenkins/templates/values.ftl.yaml", + "apps/ingress/values.yaml" + ); + + for (String path : paths) { + File file = new File(root, path); + Files.createDirectories(file.toPath().getParent()); + Files.writeString(file.toPath(), "test"); + } + + return root; + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/CommandExecutorForTest.java b/src/test/java/com/cloudogu/gitops/utils/CommandExecutorForTest.java new file mode 100644 index 000000000..a5b6ba0a6 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/CommandExecutorForTest.java @@ -0,0 +1,89 @@ +package com.cloudogu.gitops.utils; + +import lombok.Getter; + +import java.util.ArrayList; +import java.util.LinkedList; +import java.util.List; +import java.util.Queue; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; + +public class CommandExecutorForTest extends CommandExecutor { + + @Getter + private final List actualCommands = new ArrayList<>(); + + private final Queue outputs = new LinkedList<>(); + + // This is actually only set when an env is passed to CommandExecutor + @Getter + private List environment = new ArrayList<>(); + + public void enqueueOutput(Output output) { + outputs.add(output); + } + + public void enqueueOutputs(Queue outputsQueue) { + outputs.addAll(outputsQueue); + } + + @Override + protected Output getOutput(Process proc, String command, boolean failOnError) { + actualCommands.add(command); + Output output = outputs.poll(); + if (output == null) { + output = new Output("", "", 0); + } + + if (failOnError && output.getExitCode() > 0) { + throw new RuntimeException("Executing command failed: " + command); + } + + return output; + } + + @Override + protected Process doExecute(String command) { + return mock(Process.class); + } + + @Override + protected Process doExecute(String[] command) { + return mock(Process.class); + } + + @Override + protected Process doExecute(String command, List envp) { + environment = envp; + return mock(Process.class); + } + + public String assertExecuted(String commandStartsWith) { + String actualCommand = actualCommands.stream() + .filter(command -> command.startsWith(commandStartsWith)) + .findFirst() + .orElse(null); + + assertThat(actualCommand) + .as( + "Expected command to have been executed, but was not:\n%s.\nActual commands:\n%s", + commandStartsWith, + String.join("\n", actualCommands) + ) + .isNotNull(); + return actualCommand; + } + + public void assertNotExecuted(String commandStartsWith) { + String actualCommand = actualCommands.stream() + .filter(command -> command.startsWith(commandStartsWith)) + .findFirst() + .orElse(null); + + assertThat(actualCommand) + .as("Expected command to have been executed, but was not: %s", commandStartsWith) + .isNull(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/CommandExecutorTest.java b/src/test/java/com/cloudogu/gitops/utils/CommandExecutorTest.java new file mode 100644 index 000000000..d69ff3406 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/CommandExecutorTest.java @@ -0,0 +1,23 @@ +package com.cloudogu.gitops.utils; + +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class CommandExecutorTest { + + private final CommandExecutorForTest commandExecutor = new CommandExecutorForTest(); + + @Test + void aggregatesEnvironment() { + Map additionalEnv = Map.of("someKey", "someValue"); + commandExecutor.execute("command", additionalEnv); + + assertThat(commandExecutor.getActualCommands().get(0)).isEqualTo("command"); + assertThat(commandExecutor.getEnvironment().toString()).contains("someKey=someValue"); + // Make sure there are other env vars present and not solely the one we passed + assertThat(commandExecutor.getEnvironment().size()).isGreaterThan(additionalEnv.size()); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/DockerImageParserTest.java b/src/test/java/com/cloudogu/gitops/utils/DockerImageParserTest.java new file mode 100644 index 000000000..1d3555887 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/DockerImageParserTest.java @@ -0,0 +1,34 @@ +package com.cloudogu.gitops.utils; + +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class DockerImageParserTest { + + @Test + void parsesSimpleImageString() { + DockerImageParser.Image result = DockerImageParser.parse("grafana/grafana:latest"); + + assertThat(result.getRegistry()).isEqualTo(""); + assertThat(result.getRepository()).isEqualTo("grafana/grafana"); + assertThat(result.getRegistryAndRepositoryAsString()).isEqualTo("grafana/grafana"); + assertThat(result.getTag()).isEqualTo("latest"); + } + + @Test + void parsesImageStringWithPort() { + DockerImageParser.Image result = DockerImageParser.parse("localhost:5000/grafana/grafana:latest"); + + assertThat(result.getRegistry()).isEqualTo("localhost:5000"); + assertThat(result.getRepository()).isEqualTo("grafana/grafana"); + assertThat(result.getRegistryAndRepositoryAsString()).isEqualTo("localhost:5000/grafana/grafana"); + assertThat(result.getTag()).isEqualTo("latest"); + } + + @Test + void throwsWhenThereIsNoColon() { + assertThrows(RuntimeException.class, () -> DockerImageParser.parse("grafana/grafana")); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/FileSystemUtilsTest.java b/src/test/java/com/cloudogu/gitops/utils/FileSystemUtilsTest.java new file mode 100644 index 000000000..61cda59a1 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/FileSystemUtilsTest.java @@ -0,0 +1,98 @@ +package com.cloudogu.gitops.utils; + +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class FileSystemUtilsTest { + + private final FileSystemUtils fileSystemUtils = new FileSystemUtils(); + + @Test + void copiesToTempDir() throws IOException { + String expectedText = "someText"; + + File someFile = File.createTempFile(getClass().getSimpleName(), ""); + Files.writeString(someFile.toPath(), expectedText + System.lineSeparator()); + Path tmpFile = fileSystemUtils.copyToTempDir(someFile.getAbsolutePath()); + + assertThat(tmpFile.toAbsolutePath().toString()).isNotEqualTo(someFile.getAbsoluteFile()); + assertThat(Files.readString(tmpFile).trim()).isEqualTo(expectedText); + } + + @Test + void makesReadOnlyFoldersWritableRecursively() throws IOException { + Path parentDir = Files.createTempDirectory(getClass().getSimpleName()); + + File regularFile = new File(parentDir.toFile(), "regularFile.txt"); + regularFile.createNewFile(); + + File nestedDir = new File(parentDir.toFile(), "nestedDir"); + nestedDir.mkdir(); + + File readOnlyFile = new File(nestedDir, "readOnlyFile.txt"); + readOnlyFile.createNewFile(); + readOnlyFile.setWritable(false); + + File anotherReadOnlyFile = new File(parentDir.toFile(), "anotherReadOnlyFile.txt"); + anotherReadOnlyFile.createNewFile(); + anotherReadOnlyFile.setWritable(false); + + assertThat(readOnlyFile.canWrite()).isFalse(); + assertThat(anotherReadOnlyFile.canWrite()).isFalse(); + + FileSystemUtils.makeWritable(parentDir.toFile()); + + assertThat(regularFile.canWrite()).isTrue(); + assertThat(readOnlyFile.canWrite()).isTrue(); + assertThat(anotherReadOnlyFile.canWrite()).isTrue(); + + org.apache.commons.io.FileUtils.deleteDirectory(parentDir.toFile()); + } + + @Test + void readsAndWritesYaml() { + Path tmpFile = fileSystemUtils.createTempFile(); + Map yaml = Map.of( + "foo", "bar", + "nested", Map.of("a", 1, "b", 2) + ); + + fileSystemUtils.writeYaml(yaml, tmpFile.toFile()); + Map result = fileSystemUtils.readYaml(tmpFile); + + assertThat(result).isEqualTo(yaml); + } + + @Test + void readYamlFallsBackToClasspath() { + Map result = fileSystemUtils.readYaml(Path.of("testMainConfig.yaml")); + + assertThat(nestedValue(result, "registry", "internalPort")).isEqualTo(30000); + } + + @Test + void readYamlFallsBackToClasspathAndRemovesSrcMainResources() { + Map result = fileSystemUtils.readYaml(Path.of("src/main/resources/application-minimal.yaml")); + + assertThat(nestedValue(result, "application", "yes")).isEqualTo(true); + } + + @Test + void readYamlReturnsEmptyMapIfNotFound() { + Map result = fileSystemUtils.readYaml(Path.of("non-existent.yaml")); + + assertThat(result).isEmpty(); + } + + @SuppressWarnings("unchecked") + private Object nestedValue(Map source, String parentKey, String childKey) { + return ((Map) source.get(parentKey)).get(childKey); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/K8sClientForTest.java b/src/test/java/com/cloudogu/gitops/utils/K8sClientForTest.java new file mode 100644 index 000000000..1b5d5171a --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/K8sClientForTest.java @@ -0,0 +1,13 @@ +package com.cloudogu.gitops.utils; + +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import io.fabric8.kubernetes.client.server.mock.KubernetesMockServer; + +public class K8sClientForTest extends K8sClient { + + public K8sClientForTest() { + super(); + setClient(new KubernetesMockServer().createClient()); + sleepTimeMillis = 1; + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/NetworkingUtilsTest.java b/src/test/java/com/cloudogu/gitops/utils/NetworkingUtilsTest.java new file mode 100644 index 000000000..cc229b134 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/NetworkingUtilsTest.java @@ -0,0 +1,60 @@ +package com.cloudogu.gitops.utils; + +import com.cloudogu.gitops.infrastructure.kubernetes.api.K8sClient; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +class NetworkingUtilsTest { + + private final K8sClient k8sClient = mock(K8sClient.class); + private final CommandExecutorForTest commandExecutor = new CommandExecutorForTest(); + private final NetworkingUtils networkingUtils = new NetworkingUtils(k8sClient, commandExecutor); + + @Test + void clusterBindAddressReturnsBindAddressForExternalCluster() { + String internalNodeIp = "1.2.3.4"; + String localIp = "5.6.7.8"; + when(k8sClient.waitForInternalNodeIp()).thenReturn(internalNodeIp); + commandExecutor.enqueueOutput(new CommandExecutor.Output( + "", + "1.0.0.0 via w.x.y.z dev someDevice src " + localIp + " uid 1000", + 0 + )); + + String actualBindAddress = networkingUtils.findClusterBindAddress(); + + assertThat(actualBindAddress).isEqualTo(internalNodeIp); + } + + @Test + void clusterBindAddressReturnsLocalhostWhenNodeIpAndLocalIpAreEqual() { + String internalNodeIp = networkingUtils.getLocalAddress(); + assertThat(internalNodeIp).isNotEmpty(); + + when(k8sClient.waitForInternalNodeIp()).thenReturn(internalNodeIp); + + String actualBindAddress = networkingUtils.findClusterBindAddress(); + + assertThat(actualBindAddress).isEqualTo("localhost"); + } + + @Test + void clusterBindAddressFailsWhenNoPotentialBindAddress() { + when(k8sClient.waitForInternalNodeIp()).thenReturn(""); + commandExecutor.enqueueOutput(new CommandExecutor.Output( + "", + "1.0.0.0 via w.x.y.z dev someDevice src 1.2.3.4 uid 1000", + 0 + )); + + RuntimeException exception = assertThrows(RuntimeException.class, networkingUtils::findClusterBindAddress); + + assertThat(exception.getMessage()).isEqualTo( + "Could not connect to kubernetes cluster: no cluster bind address" + ); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/TemplatingEngineTest.java b/src/test/java/com/cloudogu/gitops/utils/TemplatingEngineTest.java new file mode 100644 index 000000000..c3c455a60 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/TemplatingEngineTest.java @@ -0,0 +1,113 @@ +package com.cloudogu.gitops.utils; + +import freemarker.template.TemplateException; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class TemplatingEngineTest { + + private File tmpDir; + + @BeforeEach + void before() throws IOException { + tmpDir = Files.createTempDirectory("gitops-playground-tests-templatingengine").toFile(); + tmpDir.deleteOnExit(); + } + + @Test + void replacesTwoTemplatesInDifferentFolders() throws IOException, TemplateException { + File fooTemplate = new File(tmpDir.getAbsolutePath(), "foo.ftl.txt"); + Files.writeString( + fooTemplate.toPath(), """ + this is the template + I can embed ${string} + <#if display> + and use ifs + <#else> + and use elses + + """ + ); + + File tmpDir2 = Files.createTempDirectory("gitops-playground-tests-templatingengine").toFile(); + tmpDir2.deleteOnExit(); + File barTemplate = new File(tmpDir2.getAbsolutePath(), "bar.ftl.txt"); + Files.writeString(barTemplate.toPath(), "Hello ${name}"); + + TemplatingEngine engine = new TemplatingEngine(); + engine.replaceTemplate(barTemplate, Map.of("name", "Playground")); + + assertThat(Files.readString(new File(tmpDir2.getAbsolutePath(), "bar.txt").toPath())).isEqualTo( + "Hello Playground"); + assertThat(barTemplate).doesNotExist(); + } + + @Test + void keepsTemplateFile() throws IOException, TemplateException { + File barTemplate = new File(tmpDir.getAbsolutePath(), "bar.ftl.txt"); + File barTarget = new File(tmpDir.getAbsolutePath(), "bar.txt"); + Files.writeString(barTemplate.toPath(), "Hello ${name}"); + + TemplatingEngine engine = new TemplatingEngine(); + engine.template(barTemplate, barTarget, Map.of("name", "Playground")); + + assertThat(Files.readString(barTarget.toPath())).isEqualTo("Hello Playground"); + assertThat(barTemplate).exists(); + } + + @Test + void templatesFromFileToString() throws IOException, TemplateException { + File fooTemplate = new File(tmpDir.getAbsolutePath(), "foo.ftl.txt"); + Files.writeString(fooTemplate.toPath(), "Hello ${name}"); + + TemplatingEngine engine = new TemplatingEngine(); + String result = engine.template(fooTemplate, Map.of("name", "Playground")); + + assertThat(result).isEqualTo("Hello Playground"); + } + + @Test + void templatesFromStringToString() throws IOException, TemplateException { + String fooTemplate = "Hello ${name}"; + + TemplatingEngine engine = new TemplatingEngine(); + String result = engine.template(fooTemplate, Map.of("name", "Playground")); + + assertThat(result).isEqualTo("Hello Playground"); + } + + @Test + void ignoresTemplatesWithoutVariables() throws IOException, TemplateException { + String fooTemplate = "Hello name"; + + TemplatingEngine engine = new TemplatingEngine(); + String result = engine.template(fooTemplate, Map.of()); + + assertThat(result).isEqualTo("Hello name"); + } + + @Test + void replacesYamlTemplates() throws IOException, TemplateException { + File barTemplate = new File(tmpDir.getAbsolutePath() + File.separator + "subdirectory", "result.ftl.yaml"); + Files.createDirectories(barTemplate.getParentFile().toPath()); + Files.writeString(barTemplate.toPath(), "foo: ${prefix}suffix"); + File barTarget = new File(tmpDir.getAbsolutePath(), "subdirectory/keep-this-way.yaml"); + Files.writeString(barTarget.toPath(), "thiswont: ${prefix}-be-replaced"); + + TemplatingEngine engine = new TemplatingEngine(); + engine.replaceTemplates(tmpDir, Map.of("prefix", "myteam-")); + + assertThat(Files.readString(new File(tmpDir, "subdirectory/result.yaml").toPath())).isEqualTo( + "foo: myteam-suffix"); + assertThat(Files.readString(new File(tmpDir, "subdirectory/keep-this-way.yaml").toPath())) + .isEqualTo("thiswont: ${prefix}-be-replaced"); + assertThat(new File(tmpDir, "subdirectory/result.ftl.yaml")).doesNotExist(); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/YamlUtilsTest.java b/src/test/java/com/cloudogu/gitops/utils/YamlUtilsTest.java new file mode 100644 index 000000000..8f3e15e42 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/YamlUtilsTest.java @@ -0,0 +1,36 @@ +package com.cloudogu.gitops.utils; + +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class YamlUtilsTest { + + @Test + void parsesYamlMapWithoutGroovyRuntimeParser() { + Map result = YamlUtils.parseYamlMap(""" + name: gop + nested: + enabled: true + """); + + assertThat(result.get("name")).isEqualTo("gop"); + assertThat(result.get("nested")).isEqualTo(Map.of("enabled", true)); + } + + @Test + void rejectsYamlWithNonMapRoot() { + IllegalArgumentException exception = assertThrows( + IllegalArgumentException.class, () -> + YamlUtils.parseYamlMap(""" + - one + - two + """) + ); + + assertThat(exception.getMessage()).isEqualTo("Could not parse YAML as map: [one, two]"); + } +} diff --git a/src/test/java/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProviderTest.java b/src/test/java/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProviderTest.java new file mode 100644 index 000000000..39112b012 --- /dev/null +++ b/src/test/java/com/cloudogu/gitops/utils/jgit/helpers/InsecureCredentialProviderTest.java @@ -0,0 +1,56 @@ +package com.cloudogu.gitops.utils.jgit.helpers; + +import org.eclipse.jgit.errors.UnsupportedCredentialItem; +import org.eclipse.jgit.transport.CredentialItem; +import org.eclipse.jgit.transport.URIish; +import org.junit.jupiter.api.Test; + +import java.net.URISyntaxException; + +import static org.assertj.core.api.Assertions.assertThat; + +class InsecureCredentialProviderTest { + + @Test + void ignoresIrrelevantItems() { + InsecureCredentialProvider provider = new InsecureCredentialProvider(); + + assertThat(provider.supports(new CredentialItem.Username(), new CredentialItem.Password())).isFalse(); + assertThat(provider.supports( + new CredentialItem.InformationalMessage("This is not a relevant message"), + new CredentialItem.YesNoType("This prompt is irrelevant as well") + )) + .isFalse(); + } + + @Test + void confirmsInsecureHttpsProcessing() throws UnsupportedCredentialItem, URISyntaxException { + InsecureCredentialProvider provider = new InsecureCredentialProvider(); + + CredentialItem.InformationalMessage message = new CredentialItem.InformationalMessage( + "A secure connection to https://192.168.178.37/scm/repo/argocd/cluster-resources could not be established because the server's certificate could not be validated.\n" + + "SSL reported: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n" + + "Do you want to skip SSL verification for this server?"); + CredentialItem.YesNoType skipSingle = new CredentialItem.YesNoType( + "Skip SSL verification for this single git operation"); + CredentialItem.YesNoType skipRepository = new CredentialItem.YesNoType( + "Skip SSL verification for git operations for repository /tmp/groovy-generated-tmpdir-2746077697650757929/.git"); + CredentialItem.YesNoType skipAlways = new CredentialItem.YesNoType( + "Always skip SSL verification for this server from now on"); + + assertThat(provider.supports(message, skipSingle, skipRepository, skipAlways)).isTrue(); + + assertThat(provider.get( + new URIish("https://192.168.178.37/scm/repo/argocd/cluster-resources"), + message, + skipSingle, + skipRepository, + skipAlways + )) + .isTrue(); + + assertThat(skipSingle.getValue()).isTrue(); + assertThat(skipRepository.getValue()).isTrue(); + assertThat(skipAlways.getValue()).isFalse(); + } +} diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/.gitattributes b/src/test/resources/com/cloudogu/gitops/utils/data/.gitattributes similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/.gitattributes rename to src/test/resources/com/cloudogu/gitops/utils/data/.gitattributes diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/Jenkinsfile b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/Jenkinsfile similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/Jenkinsfile rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/Jenkinsfile diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/copyRepo1 b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/copyRepo1 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/copyRepo1 rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/copyRepo1 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo1/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/copyRepo2 b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/copyRepo2 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/copyRepo2 rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/copyRepo2 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/copyRepo2/subPath/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/folderBasedRepo1 b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/folderBasedRepo1 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/folderBasedRepo1 rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/folderBasedRepo1 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/some.yaml.ftl b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/some.yaml.ftl similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/some.yaml.ftl rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/common/repo/some.yaml.ftl diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a1/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a1/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a1/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a1/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a2/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a2/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a2/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1a/repo1a2/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b1/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b1/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b1/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b1/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b2/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b2/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b2/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo1/ns1b/repo1b2/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/folderBasedRepo2 b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/folderBasedRepo2 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/folderBasedRepo2 rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/folderBasedRepo2 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/someOther.yaml.ftl b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/someOther.yaml.ftl similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/someOther.yaml.ftl rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/common/repo/someOther.yaml.ftl diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a1/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a1/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a1/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a1/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a2/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a2/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a2/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2a/repo2a2/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b1/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b1/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b1/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b1/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b2/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b2/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b2/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/folderBasedRepo2/subPath/ns2b/repo2b2/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/Jenkinsfile b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/Jenkinsfile similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/Jenkinsfile rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/Jenkinsfile diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/file b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/file similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/file rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/file diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/mirrorRepo1 b/src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/mirrorRepo1 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/mirrorRepo1 rename to src/test/resources/com/cloudogu/gitops/utils/data/contentRepos/mirrorRepo1/mirrorRepo1 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/HEAD b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/HEAD similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/HEAD rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/HEAD diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/config b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/config similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/config rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/config diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/description b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/description similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/description rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/description diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/23/cb5a712ce9ea3dc4770a350fc8ef5f51789d14 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/23/cb5a712ce9ea3dc4770a350fc8ef5f51789d14 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/23/cb5a712ce9ea3dc4770a350fc8ef5f51789d14 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/23/cb5a712ce9ea3dc4770a350fc8ef5f51789d14 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/8d/ca2f88bcfeb5fb3ecb832c4170ea85ef7be25c b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/8d/ca2f88bcfeb5fb3ecb832c4170ea85ef7be25c similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/8d/ca2f88bcfeb5fb3ecb832c4170ea85ef7be25c rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/8d/ca2f88bcfeb5fb3ecb832c4170ea85ef7be25c diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/d6/cb0108457ad5fac4b9c64bc7f1e14fdcef8c2c b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/d6/cb0108457ad5fac4b9c64bc7f1e14fdcef8c2c similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/d6/cb0108457ad5fac4b9c64bc7f1e14fdcef8c2c rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/objects/d6/cb0108457ad5fac4b9c64bc7f1e14fdcef8c2c diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/refs/heads/different b/src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/refs/heads/different similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/refs/heads/different rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repo-different-default-branch/refs/heads/different diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/HEAD b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/HEAD similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/HEAD rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/HEAD diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/config b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/config similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/config rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/config diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/description b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/description similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/description rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/description diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/info/exclude b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/info/exclude similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/info/exclude rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/info/exclude diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/15/5e9388fc29687b92a4ae2470458a5e08be9a81 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/15/5e9388fc29687b92a4ae2470458a5e08be9a81 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/15/5e9388fc29687b92a4ae2470458a5e08be9a81 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/15/5e9388fc29687b92a4ae2470458a5e08be9a81 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/26/6196b548e131009716575da17635832977d634 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/26/6196b548e131009716575da17635832977d634 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/26/6196b548e131009716575da17635832977d634 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/26/6196b548e131009716575da17635832977d634 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/38/0acb8eb2bba214e1437a79ec95927d4d2fd55f b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/38/0acb8eb2bba214e1437a79ec95927d4d2fd55f similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/38/0acb8eb2bba214e1437a79ec95927d4d2fd55f rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/38/0acb8eb2bba214e1437a79ec95927d4d2fd55f diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5a/7ad14ea366dd80f864b7c6334be5450814883d b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5a/7ad14ea366dd80f864b7c6334be5450814883d similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5a/7ad14ea366dd80f864b7c6334be5450814883d rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5a/7ad14ea366dd80f864b7c6334be5450814883d diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5b/cf50f0537bf4d2719a82e9b0950fbac92b3ecc b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5b/cf50f0537bf4d2719a82e9b0950fbac92b3ecc similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5b/cf50f0537bf4d2719a82e9b0950fbac92b3ecc rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/5b/cf50f0537bf4d2719a82e9b0950fbac92b3ecc diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/75/4f05b8621db74073ee38d5c4c755ee55291f3a b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/75/4f05b8621db74073ee38d5c4c755ee55291f3a similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/75/4f05b8621db74073ee38d5c4c755ee55291f3a rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/75/4f05b8621db74073ee38d5c4c755ee55291f3a diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/8b/c1d1165468359b16d9771d4a9a3df26afc03e8 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/8b/c1d1165468359b16d9771d4a9a3df26afc03e8 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/8b/c1d1165468359b16d9771d4a9a3df26afc03e8 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/8b/c1d1165468359b16d9771d4a9a3df26afc03e8 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/ba/2906d0666cf726c7eaadd2cd3db615dedfdf3a b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/ba/2906d0666cf726c7eaadd2cd3db615dedfdf3a similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/ba/2906d0666cf726c7eaadd2cd3db615dedfdf3a rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/ba/2906d0666cf726c7eaadd2cd3db615dedfdf3a diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/cc/1d71b4c47b0009c1ea1b0bcd4a22e1c78e81b3 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/cc/1d71b4c47b0009c1ea1b0bcd4a22e1c78e81b3 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/cc/1d71b4c47b0009c1ea1b0bcd4a22e1c78e81b3 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/cc/1d71b4c47b0009c1ea1b0bcd4a22e1c78e81b3 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/main b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/main similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/main rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/main diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/someBranch b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/someBranch similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/someBranch rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/heads/someBranch diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/tags/someTag b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/tags/someTag similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/tags/someTag rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository-with-branches-tags/refs/tags/someTag diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/HEAD b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/HEAD similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/HEAD rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/HEAD diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/config b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/config similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/config rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/config diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/description b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/description similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/description rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/description diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/info/exclude b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/info/exclude similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/info/exclude rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/info/exclude diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/2a/d7497fd7f4420e35982f13ae5b0faccd570522 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/4b/825dc642cb6eb9a060e54bf8d69288fbee4904 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/56/d2e3f4b7c95d5645c823f7be8ea6f8a853ac40 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/objects/f9/3e3a1a1525fb5b91020da86e44810c87a2d7bc diff --git a/src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/refs/heads/main b/src/test/resources/com/cloudogu/gitops/utils/data/git-repository/refs/heads/main similarity index 100% rename from src/test/groovy/com/cloudogu/gitops/utils/data/git-repository/refs/heads/main rename to src/test/resources/com/cloudogu/gitops/utils/data/git-repository/refs/heads/main diff --git a/src/test/resources/logback-test.xml b/src/test/resources/logback-test.xml index b807f873f..c561b46f9 100644 --- a/src/test/resources/logback-test.xml +++ b/src/test/resources/logback-test.xml @@ -1,13 +1,13 @@ - true - %cyan(%d{HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}) - %msg%n + %cyan(%d{HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}) - %msg%n + - + diff --git a/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker new file mode 100644 index 000000000..fdbd0b157 --- /dev/null +++ b/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker @@ -0,0 +1 @@ +mock-maker-subclass diff --git a/src/test/resources/testMainConfig.yaml b/src/test/resources/testMainConfig.yaml index 1a23479ff..7cac40000 100644 --- a/src/test/resources/testMainConfig.yaml +++ b/src/test/resources/testMainConfig.yaml @@ -20,15 +20,17 @@ jenkins: password: "admin" metricsUsername: "metrics" metricsPassword: "metrics" + jenkinsImage: "" mavenCentralMirror: "" helm: - values: {} + values: { } scm: scmProviderType: "SCM_MANAGER" scmManager: url: "http://172.18.0.2:9091/scm" username: "admin" password: "admin" + scmmImage: "" helm: chart: "scm-manager" repoURL: "https://packages.scm-manager.org/repository/helm-v2-releases/" @@ -71,7 +73,7 @@ features: chart: "kube-prometheus-stack" repoURL: "https://prometheus-community.github.io/helm-charts" version: "58.2.1" - values: {} + values: { } grafanaImage: "" grafanaSidecarImage: "" prometheusImage: "" @@ -92,15 +94,15 @@ features: helm: chart: "vault" repoURL: "https://helm.releases.hashicorp.com" - version: "0.25.0" + version: "0.34.1" image: "" ingress: active: false helm: chart: "traefik" repoURL: "https://traefik.github.io/charts" - version: "39.0.0" - values: {} + version: "39.0.9" + values: { } image: "" certManager: active: false @@ -108,7 +110,7 @@ features: chart: "cert-manager" repoURL: "https://charts.jetstack.io" version: "1.16.1" - values: {} + values: { } image: "" webhookImage: "" cainjectorImage: ""