Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making necessary changes in the running-functional-tests document. Also doing some cleanup. #8087

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+253 −101
Open

Making necessary changes in the running-functional-tests document. Also doing some cleanup. #8087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 13 additions & 19 deletions .github/workflows/functional-test-cloud.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,16 +86,16 @@ env:
# Private Git repository where terraform module for testing is stored.
TF_RECIPE_PRIVATE_GIT_SOURCE: "git::https://github.com/radius-project/terraform-private-modules//kubernetes-redis"
# bicep-types ACR url for uploading Radius Bicep types
BICEP_TYPES_REGISTRY: 'biceptypes.azurecr.io'
BICEP_TYPES_REGISTRY: "biceptypes.azurecr.io"
ytimocin marked this conversation as resolved.
Show resolved Hide resolved

jobs:
build:
name: Build Radius for test
runs-on: ubuntu-latest
if: github.event_name == 'repository_dispatch' ||
(github.event_name == 'schedule' && github.repository == 'radius-project/radius') ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
github.event_name == 'workflow_dispatch'
(github.event_name == 'schedule' && github.repository == 'radius-project/radius') ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
github.event_name == 'workflow_dispatch'
env:
DE_IMAGE: "ghcr.io/radius-project/deployment-engine"
DE_TAG: "latest"
Expand Down Expand Up @@ -343,11 +343,11 @@ jobs:
append: true
message: |
:hourglass: Publishing Bicep Recipes for functional tests...

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '18'
node-version: "18"

- name: Generate Bicep extensibility types from OpenAPI specs
run: |
Expand All @@ -360,7 +360,7 @@ jobs:
path: ./hack/bicep-types-radius/generated
if-no-files-found: error

- name: 'Login via Azure CLI'
- name: "Login via Azure CLI"
uses: azure/login@v2
with:
client-id: ${{ secrets.BICEPTYPES_CLIENT_ID }}
Expand All @@ -373,8 +373,8 @@ jobs:
chmod +x ./bicep
sudo mv ./bicep /usr/local/bin/bicep
bicep --version
- name: Publish bicep types

ytimocin marked this conversation as resolved.
Show resolved Hide resolved
- name: Publish bicep types
run: |
bicep publish-extension ./hack/bicep-types-radius/generated/index.json --target br:${{ env.BICEP_TYPES_REGISTRY }}/test/radius:${{ env.REL_VERSION == 'edge' && 'latest' || env.REL_VERSION }} --force

Expand Down Expand Up @@ -432,14 +432,14 @@ jobs:
append: true
message: |
:x: Test recipe publishing failed

tests:
name: Run ${{ matrix.name }} functional tests
needs: [build]
if: github.event_name == 'repository_dispatch' ||
(github.event_name == 'schedule' && github.repository == 'radius-project/radius') ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
github.event_name == 'workflow_dispatch'
(github.event_name == 'schedule' && github.repository == 'radius-project/radius') ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
github.event_name == 'workflow_dispatch'
strategy:
fail-fast: true
matrix:
Expand Down Expand Up @@ -725,9 +725,6 @@ jobs:

# Populate the following test environment variables from JSON secret.
# AZURE_COSMOS_MONGODB_ACCOUNT_ID
# AZURE_MSSQL_RESOURCE_ID
# AZURE_MSSQL_USERNAME
# AZURE_MSSQL_PASSWORD
Comment on lines -728 to -730
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see these are not being used anymore.

eval "export $(echo "${{ secrets.FUNCTEST_PREPROVISIONED_RESOURCE_JSON }}" | jq -r 'to_entries | map("\(.key)=\(.value)") | @sh')"

make test-functional-${{ matrix.name }}
Expand All @@ -737,9 +734,6 @@ jobs:
RADIUS_CONTAINER_LOG_PATH: ${{ github.workspace }}/${{ env.RADIUS_CONTAINER_LOG_BASE }}
AWS_REGION: ${{ env.AWS_REGION }}
AWS_ACCOUNT_ID: ${{ secrets.FUNCTEST_AWS_ACCOUNT_ID }}
RADIUS_SAMPLES_REPO_ROOT: ${{ github.workspace }}/samples
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only being used in the noncloud tests.

# Test_MongoDB_Recipe_Parameters is using the following environment variable.
INTEGRATION_TEST_RESOURCE_GROUP_NAME: ${{ env.AZURE_TEST_RESOURCE_GROUP }}
BICEP_RECIPE_REGISTRY: ${{ env.BICEP_RECIPE_REGISTRY }}
BICEP_RECIPE_TAG_VERSION: ${{ env.BICEP_RECIPE_TAG_VERSION }}
GH_TOKEN: ${{ steps.get_installation_token.outputs.token }}
Expand Down
19 changes: 8 additions & 11 deletions .github/workflows/functional-test-noncloud.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ env:
# Local Docker registry port
LOCAL_REGISTRY_PORT: "5000"
# bicep-types ACR url for uploading Radius Bicep types
BICEP_TYPES_REGISTRY: 'biceptypes.azurecr.io'
BICEP_TYPES_REGISTRY: "biceptypes.azurecr.io"

jobs:
build:
Expand Down Expand Up @@ -119,7 +119,7 @@ jobs:
# Set output variables to be used in the other jobs
echo "REL_VERSION=pr-${UNIQUE_ID}" >> $GITHUB_OUTPUT
echo "DE_IMAGE=${{ env.DE_IMAGE }}" >> $GITHUB_OUTPUT
echo "DE_TAG=${{ env.DE_TAG }}" >> $GITHUB_OUTPUT
echo "DE_TAG=${{ env.DE_TAG }}" >> $GITHUB_OUTPUT
tests:
name: Run ${{ matrix.name }} functional tests
needs: build
Expand Down Expand Up @@ -201,7 +201,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '18'
node-version: "18"

- name: Generate Bicep extensibility types from OpenAPI specs
run: |
Expand All @@ -212,7 +212,7 @@ jobs:
with:
name: ${{ matrix.name }}_radius_bicep_types
path: ./hack/bicep-types-radius/generated
if-no-files-found: error
if-no-files-found: error

- name: Create a secure local registry
id: create-local-registry
Expand Down Expand Up @@ -312,9 +312,9 @@ jobs:
sudo mv ./bicep /usr/local/bin/bicep
bicep --version

- name: Publish bicep types
run: |
bicep publish-extension ./hack/bicep-types-radius/generated/index.json --target br:${{ env.LOCAL_REGISTRY_SERVER }}:${{ env.LOCAL_REGISTRY_PORT }}/radius:${{ env.REL_VERSION == 'edge' && 'latest' || env.REL_VERSION }} --force
- name: Publish bicep types
run: |
bicep publish-extension ./hack/bicep-types-radius/generated/index.json --target br:${{ env.LOCAL_REGISTRY_SERVER }}:${{ env.LOCAL_REGISTRY_PORT }}/radius:${{ env.REL_VERSION == 'edge' && 'latest' || env.REL_VERSION }} --force

- name: Generate test bicepconfig.json
run: |
Expand Down Expand Up @@ -343,9 +343,7 @@ jobs:
env:
BICEP_RECIPE_REGISTRY: "${{ env.LOCAL_REGISTRY_SERVER }}:${{ env.LOCAL_REGISTRY_PORT }}"
BICEP_RECIPE_TAG_VERSION: ${{ env.REL_VERSION }}
TEMP_CERT_DIR: ${{ steps.create-local-registry.outputs.temp-cert-dir }}
SSL_CERT_FILE: ${{ steps.create-local-registry.outputs.temp-cert-dir }}/certs/${{ env.LOCAL_REGISTRY_SERVER }}/client.crt
Comment on lines -346 to -347
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not being used.



- name: Run functional tests
run: |
# Ensure rad cli is in path before running tests.
Expand All @@ -364,7 +362,6 @@ jobs:
RADIUS_SAMPLES_REPO_ROOT: ${{ github.workspace }}/samples
BICEP_RECIPE_REGISTRY: "${{ env.LOCAL_REGISTRY_NAME }}:${{ env.LOCAL_REGISTRY_PORT }}"
BICEP_RECIPE_TAG_VERSION: ${{ env.BICEP_RECIPE_TAG_VERSION }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ytimocin marked this conversation as resolved.
Show resolved Hide resolved
GOTESTSUM_OPTS: "--junitfile ./dist/functional_test/results.xml"

- name: Process Functional Test Results
Expand Down
6 changes: 0 additions & 6 deletions .github/workflows/long-running-azure.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,9 +451,6 @@ jobs:

# Populate the following test environment variables from JSON secret.
# AZURE_COSMOS_MONGODB_ACCOUNT_ID
# AZURE_MSSQL_RESOURCE_ID
# AZURE_MSSQL_USERNAME
# AZURE_MSSQL_PASSWORD
eval "export $(echo "${{ secrets.FUNCTEST_PREPROVISIONED_RESOURCE_JSON }}" | jq -r 'to_entries | map("\(.key)=\(.value)") | @sh')"

make test-functional-all
Expand All @@ -464,9 +461,6 @@ jobs:
AWS_SECRET_ACCESS_KEY: ${{ secrets.FUNCTEST_AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ env.AWS_REGION }}
RADIUS_SAMPLES_REPO_ROOT: ${{ github.workspace }}/samples
# Test_MongoDB_Recipe_Parameters is using the following environment variable.
INTEGRATION_TEST_RESOURCE_GROUP_NAME: ${{ env.AZURE_TEST_RESOURCE_GROUP }}
FUNC_TEST_OIDC_ISSUER: ${{ env.FUNCTEST_OIDC_ISSUER }}
DOCKER_REGISTRY: ${{ env.CONTAINER_REGISTRY }}
BICEP_RECIPE_REGISTRY: ${{ env.BICEP_RECIPE_REGISTRY }}
BICEP_RECIPE_TAG_VERSION: ${{ env.BICEP_RECIPE_TAG_VERSION }}
Expand Down
20 changes: 12 additions & 8 deletions ...ntributing-code/contributing-code-control-plane/running-controlplane-locally.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Radius consists of a few processes that get deployed inside a Kubernetes cluster.

This includes:
This includes:

- Applications.Core RP / Portable Resources' Providers (applications-rp) - The resource provider that handles processing of core resources as well as recipes.
- Universal Control Plane (ucp) - Acts as a proxy between the other services, also manages deployments of AWS resources.
Expand All @@ -32,7 +32,7 @@ If you need to manually test APIs you can reach them at the following endpoints
4. Install .NET 8.0 SDK - <https://dotnet.microsoft.com/en-us/download/dotnet/8.0>.
5. Install C# VS Code extension - <https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp>.
6. (Optional) Configure any cloud provider credentials you want to use for developing Radius.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should explain in step 1 that the Kubernetes cluster should be created with defaults, i.e., without the port mappings required in the docs because the debugger will attempt to attach to the same ports, and they will already be used by the cluster, resulting in 503 errors when running rad commands.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain more? This works fine for me with the current settings.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our docs say to use the configuration below when creating a Kind cluster, but this configuration causes port conflicts when launching the debugger.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  extraPortMappings:
  - containerPort: 80
    hostPort: 8080
    listenAddress: "0.0.0.0"  
  - containerPort: 443
    hostPort: 8443
    listenAddress: "0.0.0.0"

> 💡 The Bicep deployment engine uses .NET. However you don't need to know C# or .NET to develop locally with Radius.
> 💡 Radius will use your locally configured Azure or AWS credentials. If you are able to use the `az` or `aws` CLI then you don't need to do any additional setup.

Expand All @@ -50,7 +50,6 @@ rad init --full

This will install Radius and configure an environment for you. The database that's used **will NOT** be shared with your debug setup, so it mostly doesn't matter what choices you make.


## Setup Step 2: Modify config.yaml to point to your local RPs

You can use your build of `rad` (or build from source) to test against a local copy of the RP by creating a special workspace configuration.
Expand All @@ -71,9 +70,9 @@ workspaces:
scope: /planes/radius/local/resourceGroups/default
```

Make a copy of the `default` workspace called `dev` and set it as the default. Then add the `overrides` section from the example below.
Make a copy of the `default` workspace called `dev` and set it as the default. Then add the `overrides` section from the example below.

This example adds a `dev` workspace:
This example adds a `dev` workspace:

```yaml
workspaces:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to get the same result by adding the overrides: section to the existing default workspace, instead of adding a new workspace. It works either way, but do you think that modifying the existing workspace is a simpler way to understand it? In other words, override the ucp property of the current connection instead of creating a new workspace, like this. (I intend this comment as a discussion rather than a strong opinion on it.)

workspaces:
    default: default
    items:
        default:
            connection:
                context: kind-kind
                kind: kubernetes
                overrides:
                    ucp: http://localhost:9000
            environment: /planes/radius/local/resourceGroups/default/providers/Applications.Core/environments/default
            scope: /planes/radius/local/resourceGroups/default

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do both ways. This doc was written before, so I am not going to touch that part of the information. We can update later if necessary.

Expand Down Expand Up @@ -105,7 +104,7 @@ Run this command to create the namespace that will be used to store data.
kubectl create namespace radius-testing
```

## Setup Step 3: Create Resource Group and Environment
## Setup Step 4: Create Resource Group and Environment

At this point Radius is working but you don't have a resource group or environment. You can launch Radius and then use the CLI to create these.

Expand All @@ -126,6 +125,10 @@ rad env create default

At this point you're done with setup! Feel free to stop the debugger.

## Running Functional Tests

For further instructions on running functional tests, please refer to the [Running Functional Tests](../contributing-code-tests/running-functional-tests.md) document.

## Debugging

Now you can launch the Radius locally through the VSCode menu.
Expand Down Expand Up @@ -168,10 +171,11 @@ If you run into issues here, please re-read the prerequisites related to install
### I got a "InvalidTemplate" error when deploying a bicep file

> sample error message:

```json
{
"code": "InvalidTemplate",
"message": "Deployment template validation failed: 'The template language version '2.1-experimental' is not recognized.'.",
"message": "Deployment template validation failed: 'The template language version '2.1-experimental' is not recognized.'."
}
```

Expand All @@ -188,4 +192,4 @@ Build deployment-engine project
dotnet build
```

After building the Deployment Engine, build the radius project and redeploy bicep file by running steps from [Debugging](#debugging).
After building the Deployment Engine, build the radius project and redeploy bicep file by running steps from [Debugging](#debugging).
62 changes: 46 additions & 16 deletions ...tributing/contributing-code/contributing-code-tests/running-functional-tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,42 +33,72 @@ As much as possible, the tests use product functionality such as the Radius CLI

## Running the tests locally

**Please note** that tests may be cached if you are running them a few times in a row. Please run `go clean -testcache` to make sure that the cache is invalidated before running a test.

### Prerequisites

1. Place `rad` on your path
2. Make sure `rad-bicep` is downloaded (`rad bicep download`)
3. Make sure your [local dev environment is setup](../contributing-code-control-plane/running-controlplane-locally.md)
4. Log into your Github account and [Generate PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
5. Log-in to the container registry of your Github organization.

`export CR_PAT=<your_pat>`

`echo $CR_PAT | docker login ghcr.io -u <your_username> --password-stdin`

1. `export CR_PAT=<your_pat>`
2. `echo $CR_PAT | docker login ghcr.io -u <your_username> --password-stdin`
6. Publish Bicep test recipes by running `BICEP_RECIPE_REGISTRY=<registry-name> make publish-test-bicep-recipes`
7. Publish Terraform test recipes by running `make publish-test-terraform-recipes`
8. Change the visibility of the published packages to 'public'

> ⚠️ The tests assume the Kubernetes namespace in use is `default`. If your environment is set up differently you will see
> test failures.

> ⚠️ The tests assume the Kubernetes namespace in use is `default`. If your environment is set up differently you will see test failures.
> ⚠️ If you set environment variables for functional tests you may need to restart VS Code or other editors for them to take effect.

### Run
### Run Non-Cloud Functional Tests

1. Required environment variables:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are the values for these found so that they can be set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added descriptions for these environment variables.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all of these env variables required or some are optional?

1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running make publish-test-terraform-recipes is a prerequisite so let's make that explicit. Feel free to modify the suggested phrasing. Also, let's call it out in the prerequisite above to copy the output for later use.

Suggested change
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. You can find this URL in the output of `make publish-test-terraform-recipes` from step 7 of prerequisite above.

2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required or do we have a default that we can point the users to?

3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes.
Comment on lines +58 to +59
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these URLs or registry names?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, is this required or optional? Do we have a default in either case?

4. **RADIUS_SAMPLES_REPO_ROOT**: This should point to the root directory of the Samples repository in your local, if you want to run the functional tests for the samples.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you want to run the functional tests for the samples

What happens if this is not set?

2. Run:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's expand on this to give user some context on what's going to happen when they run this script.


```sh
.{workspace}/radius/test/execute-functional-tests-noncloud.sh
```

When you're running locally with this configuration, the script is going to create a new Radius group and environment that the functional tests need to run. The same script is also going to make sure that the necessary environment variables are set. If everything is set, the script will run the commands:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: indentation


```sh
make test-functional-cli-noncloud
make test-functional-corerp-noncloud
make test-functional-daprrp-noncloud
make test-functional-datastoresrp-noncloud
make test-functional-kubernetes-noncloud
make test-functional-msgrp-noncloud
make test-functional-samples-noncloud
make test-functional-ucp-noncloud
```

### Run Cloud Functional Tests

1. Run:
1. Required environment variables:
1. **TF_RECIPE_MODULE_SERVER_URL**
2. **DOCKER_REGISTRY**
3. **BICEP_RECIPE_REGISTRY**
4. **AZURE_COSMOS_MONGODB_ACCOUNT_ID**
5. **AWS_ACCESS_KEY_ID**
6. **AWS_SECRET_ACCESS_KEY**
7. **AWS_REGION**
2. You also need to create AWS and Azure Credentials. Please refer to: <https://docs.radapp.io/reference/cli/rad_credential_register/>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should come before step 1 because we are asking the user to set values for these in the previous step.

3. Run:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above regarding providing some context.


```sh
.{workspace}/radius/test/executeFunctionalTest.sh
.{workspace}/radius/test/execute-functional-tests-cloud.sh
```

When you're running locally with this configuration, the tests will use your locally selected Radius Environment and your local copy of `rad`. The executeFunctionalTest.sh scripts creates the azure resources and exports the values to be used in the functional test and runs:
When you're running locally with this configuration, the script is going to create a new Radius group and environment that the functional tests need to run. The same script is also going to make sure that the necessary environment variables are set. If everything is set, the script will run the commands:

```sh
make test-functional-corerp
make test-functional-msgrp
make test-functional-daprrp
make test-functional-datastoresrp
make test-functional-corerp-cloud
make test-functional-ucp-cloud
```

You can also run/debug individual tests from VSCode.
Comment on lines -70 to 104
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also run/debug individual tests from VSCode.

This seems a bit abrupt and confusing at the end after user has already run the tests through command line. Should this option be provided before they run the execute script or earlier?

Expand Down
1 change: 0 additions & 1 deletion pkg/cli/clients/management.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,7 +569,6 @@ func (amc *UCPApplicationsManagementClient) CreateOrUpdateEnvironment(ctx contex
}

return nil

}

// DeleteEnvironment deletes an environment and all of its resources by its name (in the configured scope) or resource ID.
Expand Down
Loading
Loading