-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Making necessary changes in the running-functional-tests document. Also doing some cleanup. #8087
base: main
Are you sure you want to change the base?
Conversation
67cae74
to
7aeb2b6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had left some comments on previous version of this PR here: #8069
I think the naming convention for sh files is underscores, dashes, or all words together. We use camel case for now. We use both underscores and dashes. Ref: https://google.github.io/styleguide/shellguide.html#source-filenames. I was creating new sh files for cloud and non-cloud, so I wanted to give them new names that would also be consistent with other names in our codebase. That being said, we need to decide if we want to use underscores or dashes because we have both. We only had one camel-case and I changed that. |
7aeb2b6
to
f8a1e10
Compare
Radius functional test overview
Click here to see the list of tools in the current test run
Test Status⌛ Building Radius and pushing container images for functional tests... |
# AZURE_MSSQL_RESOURCE_ID | ||
# AZURE_MSSQL_USERNAME | ||
# AZURE_MSSQL_PASSWORD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see these are not being used anymore.
@@ -729,9 +726,6 @@ jobs: | |||
AWS_ACCESS_KEY_ID: ${{ secrets.FUNCTEST_AWS_ACCESS_KEY_ID }} | |||
AWS_SECRET_ACCESS_KEY: ${{ secrets.FUNCTEST_AWS_SECRET_ACCESS_KEY }} | |||
AWS_REGION: ${{ env.AWS_REGION }} | |||
RADIUS_SAMPLES_REPO_ROOT: ${{ github.workspace }}/samples |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only being used in the noncloud tests.
TEMP_CERT_DIR: ${{ steps.create-local-registry.outputs.temp-cert-dir }} | ||
SSL_CERT_FILE: ${{ steps.create-local-registry.outputs.temp-cert-dir }}/certs/${{ env.LOCAL_REGISTRY_SERVER }}/client.crt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not being used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reformat
f8a1e10
to
223c606
Compare
Radius functional test overview
Click here to see the list of tools in the current test run
Test Status⌛ Building Radius and pushing container images for functional tests... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to get all five services running with a debugger following these instructions. 🎆
A helpful note in troubleshooting would be to say that after launching the debugger, five items should be visible in the VS Code call stack panel (in the debugging tab):
- Launch Deployment Engine
- Launch Dynamic RP
- Launch Applications RP
- Launch Controller
- Launch UCP
Initially, I had a failure because I was deploying a kind cluster using the instructions in our docs site to map ports, but those ports were conflicting with the ports the debugger was attaching to, causing 503 errors. This became apparent when I realized that the Launch Applications RP debugging configuration was failing and would disappear from the list of processes.
@@ -32,7 +32,7 @@ If you need to manually test APIs you can reach them at the following endpoints | |||
4. Install .NET 6.0 SDK - <https://dotnet.microsoft.com/en-us/download/dotnet/6.0>. | |||
5. Install C# VS Code extension - <https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp>. | |||
6. (Optional) Configure any cloud provider credentials you want to use for developing Radius. | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should explain in step 1 that the Kubernetes cluster should be created with defaults, i.e., without the port mappings required in the docs because the debugger will attempt to attach to the same ports, and they will already be used by the cluster, resulting in 503 errors when running rad
commands.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you explain more? This works fine for me with the current settings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Our docs say to use the configuration below when creating a Kind cluster, but this configuration causes port conflicts when launching the debugger.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 80
hostPort: 8080
listenAddress: "0.0.0.0"
- containerPort: 443
hostPort: 8443
listenAddress: "0.0.0.0"
|
||
This example adds a `dev` workspace: | ||
This example adds a `dev` workspace: | ||
|
||
```yaml | ||
workspaces: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to get the same result by adding the overrides:
section to the existing default workspace, instead of adding a new workspace. It works either way, but do you think that modifying the existing workspace is a simpler way to understand it? In other words, override the ucp
property of the current connection instead of creating a new workspace, like this. (I intend this comment as a discussion rather than a strong opinion on it.)
workspaces:
default: default
items:
default:
connection:
context: kind-kind
kind: kubernetes
overrides:
ucp: http://localhost:9000
environment: /planes/radius/local/resourceGroups/default/providers/Applications.Core/environments/default
scope: /planes/radius/local/resourceGroups/default
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do both ways. This doc was written before, so I am not going to touch that part of the information. We can update later if necessary.
### Run | ||
### Run Non-Cloud Functional Tests | ||
|
||
1. Required environment variables: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where are the values for these found so that they can be set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added descriptions for these environment variables.
223c606
to
10ec437
Compare
10ec437
to
8a17468
Compare
8a17468
to
a617d63
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #8087 +/- ##
==========================================
+ Coverage 59.73% 59.75% +0.02%
==========================================
Files 590 590
Lines 39487 39487
==========================================
+ Hits 23588 23597 +9
+ Misses 14152 14146 -6
+ Partials 1747 1744 -3 ☔ View full report in Codecov by Sentry. |
rad group create default | ||
rad env create kind-radius | ||
rad group create kind-radius | ||
|
||
# Check if TF_RECIPE_MODULE_SERVER_URL environment variable is set | ||
if [ -z "$TF_RECIPE_MODULE_SERVER_URL" ]; then | ||
echo "Error: TF_RECIPE_MODULE_SERVER_URL environment variable is not set." | ||
exit 1 | ||
fi | ||
|
||
# Check if DOCKER_REGISTRY environment variable is set | ||
if [ -z "$DOCKER_REGISTRY" ]; then | ||
echo "Error: DOCKER_REGISTRY environment variable is not set." | ||
exit 1 | ||
fi | ||
|
||
# Check if BICEP_RECIPE_REGISTRY environment variable is set | ||
if [ -z "$BICEP_RECIPE_REGISTRY" ]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you remind where is this handled currently?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not being handled anywhere currently in the helper script.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How's everyone running functional tests today? Is this documented somewhere for manual setup?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤷♂️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am trying to update the document with this PR.
# Make sure that you have default environment in the default group. | ||
# The way to check that is to run the following commands: | ||
# `rad group switch default` | ||
# `rad env list` and check if there is an environment named `default`. | ||
# If not, create one using `rad env create default`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This script seems to be creating group and env below L30-L32, so why is this manual check needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just being extra defensive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems unnecessary here.
a617d63
to
24a4e7b
Compare
24a4e7b
to
f72fa70
Compare
…so doing some cleanup. Signed-off-by: ytimocin <[email protected]>
f72fa70
to
1ee4a57
Compare
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images. | ||
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these URLs or registry names?
# and then switch to the environment using `rad env switch kind-radius`. | ||
|
||
rad group create default | ||
rad env create kind-radius |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The kind-radius
environment is created in kind-radius
group here https://github.com/radius-project/radius/blob/main/.github/workflows/functional-test-noncloud.yaml#L293-L299. Any reason to create it in the default group locally?
# Make sure that you have default environment in the default group. | ||
# The way to check that is to run the following commands: | ||
# `rad group switch default` | ||
# `rad env list` and check if there is an environment named `default`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is expected to be already existing, then the manual group creation on L30 shouldn't be needed
### Run Non-Cloud Functional Tests | ||
|
||
1. Required environment variables: | ||
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Running make publish-test-terraform-recipes
is a prerequisite so let's make that explicit. Feel free to modify the suggested phrasing. Also, let's call it out in the prerequisite above to copy the output for later use.
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command. | |
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. You can find this URL in the output of `make publish-test-terraform-recipes` from step 7 of prerequisite above. |
|
||
1. Required environment variables: | ||
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command. | ||
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this required or do we have a default that we can point the users to?
1. Required environment variables: | ||
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command. | ||
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images. | ||
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here, is this required or optional? Do we have a default in either case?
1. **TF_RECIPE_MODULE_SERVER_URL**: This is the URL for the Terraform Recipe Module Server. If you have run `make publish-test-terraform-recipes` you will see this URL at the end of that command. | ||
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images. | ||
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes. | ||
4. **RADIUS_SAMPLES_REPO_ROOT**: This should point to the root directory of the Samples repository in your local, if you want to run the functional tests for the samples. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if you want to run the functional tests for the samples
What happens if this is not set?
### Run | ||
### Run Non-Cloud Functional Tests | ||
|
||
1. Required environment variables: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are all of these env variables required or some are optional?
2. **DOCKER_REGISTRY**: This is the container registry that you would be using for storing the test related images. | ||
3. **BICEP_RECIPE_REGISTRY**: This is the container registry that you would be using for storing the Bicep recipes. | ||
4. **RADIUS_SAMPLES_REPO_ROOT**: This should point to the root directory of the Samples repository in your local, if you want to run the functional tests for the samples. | ||
2. Run: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's expand on this to give user some context on what's going to happen when they run this script.
.{workspace}/radius/test/execute-functional-tests-noncloud.sh | ||
``` | ||
|
||
When you're running locally with this configuration, the script is going to create a new Radius group and environment that the functional tests need to run. The same script is also going to make sure that the necessary environment variables are set. If everything is set, the script will run the commands: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: indentation
5. **AWS_ACCESS_KEY_ID** | ||
6. **AWS_SECRET_ACCESS_KEY** | ||
7. **AWS_REGION** | ||
2. You also need to create AWS and Azure Credentials. Please refer to: <https://docs.radapp.io/reference/cli/rad_credential_register/>. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should come before step 1 because we are asking the user to set values for these in the previous step.
6. **AWS_SECRET_ACCESS_KEY** | ||
7. **AWS_REGION** | ||
2. You also need to create AWS and Azure Credentials. Please refer to: <https://docs.radapp.io/reference/cli/rad_credential_register/>. | ||
3. Run: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above regarding providing some context.
make test-functional-daprrp | ||
make test-functional-datastoresrp | ||
make test-functional-corerp-cloud | ||
make test-functional-ucp-cloud | ||
``` | ||
|
||
You can also run/debug individual tests from VSCode. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can also run/debug individual tests from VSCode.
This seems a bit abrupt and confusing at the end after user has already run the tests through command line. Should this option be provided before they run the execute script or earlier?
# `make test-functional-all-noncloud` | ||
|
||
usage() { | ||
echo -e "$0 requires <resourcegroup_name>\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is resourcegroup_name
defined?
# You can run them one by one or all at once. | ||
# The way to run them all at once is to run the following command: | ||
# `make test-functional-all-noncloud` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What should a user skip/update to follow this? Also should be cloud instead of noncloud.
# You can run them one by one or all at once. | ||
# The way to run them all at once is to run the following command: | ||
# `make test-functional-all-noncloud` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here
# Make sure that you are in the root directory of the repository. | ||
cd ../ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this needed in the cloud tests script as well?
make test-functional-datastoresrp-noncloud | ||
make test-functional-kubernetes-noncloud | ||
# make test-functional-msgrp-noncloud | ||
make test-functional-samples-noncloud |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need users to run samples in local testing?
echo "Running Radius non-cloud functional tests." | ||
|
||
make test-functional-cli-noncloud | ||
# make test-functional-corerp-noncloud |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't be commented out?
Description
Making necessary changes in the running-functional-tests document. Also doing some cleanup:
Type of change
Fixes: #issue_number
Contributor checklist
Please verify that the PR meets the following requirements, where applicable: