Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unskipping tea tests and adding protection around possible nil models #8053

Merged
merged 1 commit into from
Jan 27, 2025

Conversation

ytimocin
Copy link
Contributor

Description

Un-skipping tea tests and adding protection around possible nil models.

Type of change

  • This pull request is a minor refactor, code cleanup, test improvement, or other maintenance task and doesn't change the functionality of Radius (issue link optional).
    Fixes: #issue_number

Contributor checklist

Please verify that the PR meets the following requirements, where applicable:

  • An overview of proposed schema changes is included in a linked GitHub issue.
  • A design document PR is created in the design-notes repository, if new APIs are being introduced.
  • If applicable, design document has been reviewed and approved by Radius maintainers/approvers.
  • A PR for the samples repository is created, if existing samples are affected by the changes in this PR.
  • A PR for the documentation repository is created, if the changes in this PR affect the documentation or any user facing updates are made.
  • A PR for the recipes repository is created, if existing recipes are affected by the changes in this PR.

Copy link

github-actions bot commented Nov 12, 2024

Unit Tests

3 557 tests  +5   3 555 ✅ +7   5m 27s ⏱️ +9s
  292 suites ±0       2 💤  - 2 
    1 files   ±0       0 ❌ ±0 

Results for commit 8d67f86. ± Comparison against base commit c155a90.

♻️ This comment has been updated with latest results.

Copy link

codecov bot commented Nov 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.97%. Comparing base (c155a90) to head (8d67f86).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8053      +/-   ##
==========================================
+ Coverage   59.82%   59.97%   +0.14%     
==========================================
  Files         590      590              
  Lines       39513    39513              
==========================================
+ Hits        23639    23696      +57     
+ Misses      14114    14055      -59     
- Partials     1760     1762       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@radius-functional-tests
Copy link

radius-functional-tests bot commented Nov 13, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 8636b76
Unique ID funcf18396758b
Image tag pr-funcf18396758b
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcf18396758b
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcf18396758b
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funcf18396758b
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcf18396758b
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcf18396758b
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@@ -50,6 +48,11 @@ func Test_summaryModel(t *testing.T) {
waitForEmpty := func(t *testing.T, reader io.Reader) string {
normalized := ""
teatest.WaitFor(t, reader, func(bts []byte) bool {
if bts == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a theory about why this would happen? Why is this the right fix?

tm.WaitFinished(t, teatest.WithFinalTimeout(waitTimeout))

finalModel := tm.FinalModel(t)
require.NotNil(t, finalModel, "Final model should not be nil")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain why this is a good fix given the failure pattern from the original issue? #7670

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that there is sometimes a very short delay of model being updated. I am still in the process of proving this. The reason for my thinking is that we get a nil pointer exception, and we don't get it all the time. I think, but am not sure yet, that waiting for the model to be completely finished may fix the issue.

As mentioned above, I am working on learning more about the tea library and proving my point. :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that there is sometimes a very short delay of model being updated

That's my theory as well. The test is flaky, which means the root cause is probably a data-race.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@radius-functional-tests
Copy link

radius-functional-tests bot commented Nov 15, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 5194103
Unique ID func24303fddb5
Image tag pr-func24303fddb5
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr: 1.12.0
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func24303fddb5
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func24303fddb5
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func24303fddb5
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func24303fddb5
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func24303fddb5
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@radius-functional-tests
Copy link

radius-functional-tests bot commented Dec 15, 2024

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 39f59c8
Unique ID func16427ecbb5
Image tag pr-func16427ecbb5
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func16427ecbb5
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func16427ecbb5
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func16427ecbb5
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func16427ecbb5
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func16427ecbb5
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 13, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 7cfad64
Unique ID func4bb2fbc716
Image tag pr-func4bb2fbc716
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func4bb2fbc716
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func4bb2fbc716
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func4bb2fbc716
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func4bb2fbc716
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func4bb2fbc716
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

Comment on lines -50 to -59
waitForEmpty := func(t *testing.T, reader io.Reader) string {
normalized := ""
teatest.WaitFor(t, reader, func(bts []byte) bool {
normalized = stripansi.Strip(strings.ReplaceAll(string(bts), "\r\n", "\n"))
return !strings.Contains(normalized, strings.Trim(summaryFooter, "\n"))
}, teatest.WithDuration(waitTimeout))

return normalized
}

Copy link
Contributor

@vishwahiremat vishwahiremat Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to understand why is this part of the code not needed anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tm.Quit is taking care of this part now.

@ytimocin ytimocin force-pushed the ytimocin/tea-tes/nil-issue branch from 7cfad64 to 5ca6cd5 Compare January 15, 2025 17:21
@ytimocin ytimocin force-pushed the ytimocin/tea-tes/nil-issue branch from 5ca6cd5 to d92554d Compare January 24, 2025 05:54
@ytimocin ytimocin temporarily deployed to functional-tests January 24, 2025 05:54 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 24, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref d92554d
Unique ID func3ac3a94808
Image tag pr-func3ac3a94808
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func3ac3a94808
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func3ac3a94808
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func3ac3a94808
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func3ac3a94808
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func3ac3a94808
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@ytimocin ytimocin force-pushed the ytimocin/tea-tes/nil-issue branch from d92554d to 3db66ba Compare January 27, 2025 04:49
@ytimocin ytimocin temporarily deployed to functional-tests January 27, 2025 04:49 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 27, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 3db66ba
Unique ID funcc8cfab7cc9
Image tag pr-funcc8cfab7cc9
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funcc8cfab7cc9
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funcc8cfab7cc9
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funcc8cfab7cc9
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funcc8cfab7cc9
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funcc8cfab7cc9
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@ytimocin ytimocin force-pushed the ytimocin/tea-tes/nil-issue branch from 3db66ba to 06b865a Compare January 27, 2025 18:47
@ytimocin ytimocin temporarily deployed to functional-tests January 27, 2025 18:47 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 27, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 06b865a
Unique ID funca9e6b8e0b4
Image tag pr-funca9e6b8e0b4
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funca9e6b8e0b4
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funca9e6b8e0b4
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funca9e6b8e0b4
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funca9e6b8e0b4
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funca9e6b8e0b4
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ corerp-cloud functional tests succeeded
✅ ucp-cloud functional tests succeeded

@ytimocin ytimocin force-pushed the ytimocin/tea-tes/nil-issue branch from 06b865a to 7d451c4 Compare January 27, 2025 19:22
@ytimocin ytimocin temporarily deployed to functional-tests January 27, 2025 19:22 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 27, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 7d451c4
Unique ID func1a0c853547
Image tag pr-func1a0c853547
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func1a0c853547
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func1a0c853547
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func1a0c853547
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-func1a0c853547
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func1a0c853547
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@ytimocin ytimocin force-pushed the ytimocin/tea-tes/nil-issue branch from 7d451c4 to 8d67f86 Compare January 27, 2025 20:37
@ytimocin ytimocin temporarily deployed to functional-tests January 27, 2025 20:38 — with GitHub Actions Inactive
@radius-functional-tests
Copy link

radius-functional-tests bot commented Jan 27, 2025

Radius functional test overview

🔍 Go to test action run

Name Value
Repository radius-project/radius
Commit ref 8d67f86
Unique ID funca067da3091
Image tag pr-funca067da3091
Click here to see the list of tools in the current test run
  • gotestsum 1.12.0
  • KinD: v0.20.0
  • Dapr:
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-funca067da3091
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-funca067da3091
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-funca067da3091
  • controller test image location: ghcr.io/radius-project/dev/controller:pr-funca067da3091
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-funca067da3091
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ corerp-cloud functional tests succeeded
✅ ucp-cloud functional tests succeeded

@ytimocin ytimocin merged commit a1782fc into main Jan 27, 2025
32 checks passed
@ytimocin ytimocin deleted the ytimocin/tea-tes/nil-issue branch January 27, 2025 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants