Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making necessary changes in the running-functional-tests document. Also doing some cleanup. #8087

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ytimocin
Copy link
Contributor

Description

Making necessary changes in the running-functional-tests document. Also doing some cleanup:

  1. Removed unnecessary environment variables from the workflows.

Type of change

  • This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional).
    Fixes: #issue_number

Contributor checklist

Please verify that the PR meets the following requirements, where applicable:

  • An overview of proposed schema changes is included in a linked GitHub issue.
  • A design document PR is created in the design-notes repository, if new APIs are being introduced.
  • If applicable, design document has been reviewed and approved by Radius maintainers/approvers.
  • A PR for the samples repository is created, if existing samples are affected by the changes in this PR.
  • A PR for the documentation repository is created, if the changes in this PR affect the documentation or any user facing updates are made.
  • A PR for the recipes repository is created, if existing recipes are affected by the changes in this PR.

Copy link

github-actions bot commented Nov 25, 2024

Unit Tests

3 554 tests  ±0   3 550 ✅ ±0   5m 14s ⏱️ -1s
  292 suites ±0       4 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 1ee4a57. ± Comparison against base commit 845f88d.

♻️ This comment has been updated with latest results.

Copy link
Contributor

@kachawla kachawla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had left some comments on previous version of this PR here: #8069

@ytimocin
Copy link
Contributor Author

What's the motivation for changing file name style?

I think the naming convention for sh files is underscores, dashes, or all words together. We use camel case for now. We use both underscores and dashes.

Ref: https://google.github.io/styleguide/shellguide.html#source-filenames.

I was creating new sh files for cloud and non-cloud, so I wanted to give them new names that would also be consistent with other names in our codebase.

That being said, we need to decide if we want to use underscores or dashes because we have both. We only had one camel-case and I changed that.

@ytimocin ytimocin force-pushed the ytimocin/test-doc-update branch from 7aeb2b6 to f8a1e10 Compare November 26, 2024 18:23
@ytimocin ytimocin marked this pull request as ready for review November 26, 2024 18:23
@ytimocin ytimocin requested review from a team as code owners November 26, 2024 18:23
@radius-functional-tests
Copy link

radius-functional-tests bot commented Nov 26, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref f8a1e10
Unique ID funcefc4ddb0b6
Image tag pr-funcefc4ddb0b6
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcefc4ddb0b6
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcefc4ddb0b6
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funcefc4ddb0b6
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcefc4ddb0b6
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcefc4ddb0b6
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ corerp-cloud functional tests succeeded
✅ ucp-cloud functional tests succeeded

Comment on lines -719 to -730
# AZURE_MSSQL_RESOURCE_ID
# AZURE_MSSQL_USERNAME
# AZURE_MSSQL_PASSWORD
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see these are not being used anymore.

@@ -729,9 +726,6 @@ jobs:
AWS_ACCESS_KEY_ID: ${{ secrets.FUNCTEST_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.FUNCTEST_AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ env.AWS_REGION }}
RADIUS_SAMPLES_REPO_ROOT: ${{ github.workspace }}/samples
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only being used in the noncloud tests.

Comment on lines -350 to -347
TEMP_CERT_DIR: ${{ steps.create-local-registry.outputs.temp-cert-dir }}
SSL_CERT_FILE: ${{ steps.create-local-registry.outputs.temp-cert-dir }}/certs/${{ env.LOCAL_REGISTRY_SERVER }}/client.crt
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not being used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reformat

@ytimocin ytimocin force-pushed the ytimocin/test-doc-update branch from f8a1e10 to 223c606 Compare December 3, 2024 01:45
@ytimocin ytimocin temporarily deployed to functional-tests December 3, 2024 01:45 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Dec 3, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 223c606
Unique ID funcec1695f50d
Image tag pr-funcec1695f50d
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcec1695f50d
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcec1695f50d
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funcec1695f50d
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcec1695f50d
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcec1695f50d
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

Copy link
Member

@brooke-hamilton brooke-hamilton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to get all five services running with a debugger following these instructions. 🎆

A helpful note in troubleshooting would be to say that after launching the debugger, five items should be visible in the VS Code call stack panel (in the debugging tab):

  • Launch Deployment Engine
  • Launch Dynamic RP
  • Launch Applications RP
  • Launch Controller
  • Launch UCP

Initially, I had a failure because I was deploying a kind cluster using the instructions in our docs site to map ports, but those ports were conflicting with the ports the debugger was attaching to, causing 503 errors. This became apparent when I realized that the Launch Applications RP debugging configuration was failing and would disappear from the list of processes.

@@ -32,7 +32,7 @@ If you need to manually test APIs you can reach them at the following endpoints
4. Install .NET 6.0 SDK - <https://dotnet.microsoft.com/en-us/download/dotnet/6.0>.
5. Install C# VS Code extension - <https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp>.
6. (Optional) Configure any cloud provider credentials you want to use for developing Radius.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should explain in step 1 that the Kubernetes cluster should be created with defaults, i.e., without the port mappings required in the docs because the debugger will attempt to attach to the same ports, and they will already be used by the cluster, resulting in 503 errors when running rad commands.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain more? This works fine for me with the current settings.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our docs say to use the configuration below when creating a Kind cluster, but this configuration causes port conflicts when launching the debugger.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  extraPortMappings:
  - containerPort: 80
    hostPort: 8080
    listenAddress: "0.0.0.0"  
  - containerPort: 443
    hostPort: 8443
    listenAddress: "0.0.0.0"


This example adds a `dev` workspace:
This example adds a `dev` workspace:

```yaml
workspaces:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to get the same result by adding the overrides: section to the existing default workspace, instead of adding a new workspace. It works either way, but do you think that modifying the existing workspace is a simpler way to understand it? In other words, override the ucp property of the current connection instead of creating a new workspace, like this. (I intend this comment as a discussion rather than a strong opinion on it.)

workspaces:
    default: default
    items:
        default:
            connection:
                context: kind-kind
                kind: kubernetes
                overrides:
                    ucp: http://localhost:9000
            environment: /planes/radius/local/resourceGroups/default/providers/Applications.Core/environments/default
            scope: /planes/radius/local/resourceGroups/default

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do both ways. This doc was written before, so I am not going to touch that part of the information. We can update later if necessary.

### Run
### Run Non-Cloud Functional Tests

1. Required environment variables:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are the values for these found so that they can be set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added descriptions for these environment variables.

Copy link

codecov bot commented Jan 2, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.75%. Comparing base (845f88d) to head (1ee4a57).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8087      +/-   ##
==========================================
+ Coverage   59.73%   59.75%   +0.02%     
==========================================
  Files         590      590              
  Lines       39487    39487              
==========================================
+ Hits        23588    23597       +9     
+ Misses      14152    14146       -6     
+ Partials     1747     1744       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

.github/workflows/functional-test-cloud.yaml Show resolved Hide resolved
.github/workflows/functional-test-cloud.yaml Show resolved Hide resolved
Comment on lines +30 to +47
rad group create default
rad env create kind-radius
rad group create kind-radius

# Check if TF_RECIPE_MODULE_SERVER_URL environment variable is set
if [ -z "$TF_RECIPE_MODULE_SERVER_URL" ]; then
echo "Error: TF_RECIPE_MODULE_SERVER_URL environment variable is not set."
exit 1
fi

# Check if DOCKER_REGISTRY environment variable is set
if [ -z "$DOCKER_REGISTRY" ]; then
echo "Error: DOCKER_REGISTRY environment variable is not set."
exit 1
fi

# Check if BICEP_RECIPE_REGISTRY environment variable is set
if [ -z "$BICEP_RECIPE_REGISTRY" ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remind where is this handled currently?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not being handled anywhere currently in the helper script.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How's everyone running functional tests today? Is this documented somewhere for manual setup?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to update the document with this PR.

Comment on lines 19 to 23
# Make sure that you have default environment in the default group.
# The way to check that is to run the following commands:
# `rad group switch default`
# `rad env list` and check if there is an environment named `default`.
# If not, create one using `rad env create default`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script seems to be creating group and env below L30-L32, so why is this manual check needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just being extra defensive.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems unnecessary here.

test/execute-functional-tests-cloud.sh Show resolved Hide resolved
test/execute-functional-tests-cloud.sh Show resolved Hide resolved
Comment on lines +58 to +59
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images.
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these URLs or registry names?

# and then switch to the environment using `rad env switch kind-radius`.

rad group create default
rad env create kind-radius
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kind-radius environment is created in kind-radius group here https://github.com/radius-project/radius/blob/main/.github/workflows/functional-test-noncloud.yaml#L293-L299. Any reason to create it in the default group locally?

Comment on lines +19 to +22
# Make sure that you have default environment in the default group.
# The way to check that is to run the following commands:
# `rad group switch default`
# `rad env list` and check if there is an environment named `default`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is expected to be already existing, then the manual group creation on L30 shouldn't be needed

### Run Non-Cloud Functional Tests

1. Required environment variables:
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running make publish-test-terraform-recipes is a prerequisite so let's make that explicit. Feel free to modify the suggested phrasing. Also, let's call it out in the prerequisite above to copy the output for later use.

Suggested change
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. You can find this URL in the output of `make publish-test-terraform-recipes` from step 7 of prerequisite above.


1. Required environment variables:
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required or do we have a default that we can point the users to?

1. Required environment variables:
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images.
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, is this required or optional? Do we have a default in either case?

1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command.
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images.
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes.
4. **RADIUS_SAMPLES_REPO_ROOT**: This should point to the root directory of the Samples repository in your local, if you want to run the functional tests for the samples.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you want to run the functional tests for the samples

What happens if this is not set?

### Run
### Run Non-Cloud Functional Tests

1. Required environment variables:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all of these env variables required or some are optional?

2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images.
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes.
4. **RADIUS_SAMPLES_REPO_ROOT**: This should point to the root directory of the Samples repository in your local, if you want to run the functional tests for the samples.
2. Run:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's expand on this to give user some context on what's going to happen when they run this script.

.{workspace}/radius/test/execute-functional-tests-noncloud.sh
```

When you're running locally with this configuration, the script is going to create a new Radius group and environment that the functional tests need to run. The same script is also going to make sure that the necessary environment variables are set. If everything is set, the script will run the commands:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: indentation

5. **AWS_ACCESS_KEY_ID**
6. **AWS_SECRET_ACCESS_KEY**
7. **AWS_REGION**
2. You also need to create AWS and Azure Credentials. Please refer to: <https://docs.radapp.io/reference/cli/rad_credential_register/>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should come before step 1 because we are asking the user to set values for these in the previous step.

6. **AWS_SECRET_ACCESS_KEY**
7. **AWS_REGION**
2. You also need to create AWS and Azure Credentials. Please refer to: <https://docs.radapp.io/reference/cli/rad_credential_register/>.
3. Run:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above regarding providing some context.

Comment on lines -70 to 104
make test-functional-daprrp
make test-functional-datastoresrp
make test-functional-corerp-cloud
make test-functional-ucp-cloud
```

You can also run/debug individual tests from VSCode.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also run/debug individual tests from VSCode.

This seems a bit abrupt and confusing at the end after user has already run the tests through command line. Should this option be provided before they run the execute script or earlier?

# `make test-functional-all-noncloud`

usage() {
echo -e "$0 requires <resourcegroup_name>\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is resourcegroup_name defined?

Comment on lines +76 to +78
# You can run them one by one or all at once.
# The way to run them all at once is to run the following command:
# `make test-functional-all-noncloud`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should a user skip/update to follow this? Also should be cloud instead of noncloud.

Comment on lines +58 to +60
# You can run them one by one or all at once.
# The way to run them all at once is to run the following command:
# `make test-functional-all-noncloud`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Comment on lines +62 to +63
# Make sure that you are in the root directory of the repository.
cd ../
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed in the cloud tests script as well?

make test-functional-datastoresrp-noncloud
make test-functional-kubernetes-noncloud
# make test-functional-msgrp-noncloud
make test-functional-samples-noncloud
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need users to run samples in local testing?

echo "Running Radius non-cloud functional tests."

make test-functional-cli-noncloud
# make test-functional-corerp-noncloud
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't be commented out?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants