Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misc sdk changes #6018

Merged
merged 9 commits into from
Jan 31, 2025
Merged

Misc sdk changes #6018

merged 9 commits into from
Jan 31, 2025

Conversation

ehsandeep
Copy link
Member

@ehsandeep ehsandeep commented Jan 31, 2025

Summary by CodeRabbit

Release Notes

  • New Features

    • Enhanced TLS configuration for integration testing.
    • Improved URL handling in offline HTTP request processing.
    • Added flexible parser initialization with custom cache support.
    • New feedback mechanism in integration tests for successful completion.
  • Improvements

    • More precise URL extraction for HTTP requests.
    • Updated dependency version for improved stability.
    • Modified template validation commands for enhanced execution parameters.

The release introduces subtle enhancements to testing, HTTP request handling, and template parsing mechanisms, providing more flexibility and control in various scenarios.

Copy link
Contributor

coderabbitai bot commented Jan 31, 2025

Walkthrough

The pull request introduces modifications across five different files: cmd/integration-test/ssl.go, pkg/protocols/offlinehttp/request.go, pkg/templates/parser.go, go.mod, and Makefile. The changes involve enhancing TLS configuration in an integration test, improving raw input handling in an HTTP request processing method, adding a new initialization method for a template parser with a pre-defined cache, updating a dependency version in the Go module file, and modifying the template-validate target in the Makefile to include additional parameters for the nuclei binary.

Changes

File Change Summary
cmd/integration-test/ssl.go Updated TLS server configuration with explicit MinVersion and MaxVersion, and modified expected results count from 1 to 2.
pkg/protocols/offlinehttp/request.go - Added inputString parameter to executeRawInput method.
- Introduced getURLFromRequest function.
- Updated URL handling in output event construction to use reqURL.
pkg/templates/parser.go Added NewParserWithParsedCache method to create a Parser with a pre-defined parsed templates cache.
go.mod Updated dependency version of github.com/projectdiscovery/yamldoc-go from v1.0.4 to v1.0.5, and noted retraction of version v3.2.0 due to a broken JS protocol issue.
Makefile Modified template-validate target to include -et http/technologies flag in the commands for the nuclei binary.

Suggested reviewers

  • dogancanbakir

Poem

🐰 Hopping through code, what a sight,
TLS and URLs set just right!
New caches for templates, oh what fun,
Validations run, and all tests done!
With each little change, we grow and play,
In the world of code, we leap and sway! 🚀


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e11ce5a and 04cc426.

📒 Files selected for processing (1)
  • Makefile (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: Tests (macOS-latest)
  • GitHub Check: Tests (windows-latest)
  • GitHub Check: Tests (ubuntu-latest)
🔇 Additional comments (1)
Makefile (1)

140-141: Verify the impact of excluding HTTP technology templates during validation.

The addition of -et http/technologies flag excludes HTTP technology templates from validation. Please clarify:

  1. The rationale behind excluding these templates
  2. Whether this affects template validation coverage
  3. If there are any potential risks in skipping validation for HTTP technology templates

Let's check the impact of this change by running the following script:

✅ Verification successful

Template exclusion during validation is an intended feature

The -et http/technologies flag is a valid template exclusion mechanism used to focus validation on specific template subsets. This practice:

  • Allows for targeted template validation
  • Doesn't affect the scanner's detection capabilities
  • Is a documented and supported feature of nuclei
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Analyze the impact of excluding HTTP technology templates.

# Count total templates vs excluded templates
echo "Analyzing template counts..."
echo "Total templates:"
fd -e yaml -e yml . | wc -l

echo "HTTP technology templates that will be excluded:"
fd -e yaml -e yml . -p "http/technologies" | wc -l

# Check if any existing templates depend on HTTP technology templates
echo "Checking for template dependencies..."
rg -l "http/technologies" -g "*.y*ml"

Length of output: 531


Script:

#!/bin/bash
# Description: Improved analysis of HTTP technology templates

# Check if there's a templates/http/technologies directory
echo "Looking for HTTP technology templates directory..."
fd -t d "technologies" -p "http"

# Search for technology detection patterns in templates
echo -e "\nAnalyzing templates with technology detection patterns..."
rg -l "(?i)(tech|technology|stack|framework|cms|server)" -g "*.y*ml" -A 5

# Check for template documentation
echo -e "\nChecking documentation about template organization..."
fd -g "README*" -X grep -l "template" {} \;

Length of output: 1557


Script:

#!/bin/bash
# Description: Check documentation for template organization and exclusion flags

echo "Checking main README for template organization info..."
cat README.md | grep -A 5 -i "template"

echo -e "\nChecking for template exclusion documentation..."
rg -i "(-et|exclude.*template|template.*exclude)" -g "*.md"

Length of output: 21464


🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@Ice3man543 Ice3man543 marked this pull request as ready for review January 31, 2025 10:27
@auto-assign auto-assign bot requested a review from dwisiswant0 January 31, 2025 10:27
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
pkg/protocols/offlinehttp/request.go (1)

134-139: Consider adding URL validation.

While the URL construction is correct, adding validation could prevent potential issues with malformed URLs.

Consider enhancing the function:

 func getURLFromRequest(req *http.Request) string {
+	if req == nil || req.URL == nil {
+		return ""
+	}
 	if req.URL.Scheme == "" {
 		req.URL.Scheme = "https"
 	}
-	return fmt.Sprintf("%s://%s%s", req.URL.Scheme, req.Host, req.URL.Path)
+	url := fmt.Sprintf("%s://%s%s", req.URL.Scheme, req.Host, req.URL.Path)
+	if _, err := http.ParseRequestURI(url); err != nil {
+		return ""
+	}
+	return url
 }
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 052fd8b and 9908926.

📒 Files selected for processing (3)
  • cmd/integration-test/ssl.go (1 hunks)
  • pkg/protocols/offlinehttp/request.go (6 hunks)
  • pkg/templates/parser.go (1 hunks)
🧰 Additional context used
🪛 GitHub Check: Lint
cmd/integration-test/ssl.go

[failure] 127-127:
SA1019: tls.VersionSSL30 has been deprecated since Go 1.13 because it shouldn't be used: SSLv3 is cryptographically broken, and is no longer supported by this package. See golang.org/issue/32716. (staticcheck)

🪛 golangci-lint (1.62.2)
cmd/integration-test/ssl.go

127-127: SA1019: tls.VersionSSL30 has been deprecated since Go 1.13 because it shouldn't be used: SSLv3 is cryptographically broken, and is no longer supported by this package. See golang.org/issue/32716.

(staticcheck)

🪛 GitHub Actions: 🔨 Tests
cmd/integration-test/ssl.go

[error] 127-127: tls.VersionSSL30 has been deprecated since Go 1.13 because it shouldn't be used: SSLv3 is cryptographically broken, and is no longer supported by this package. See golang.org/issue/32716

🔇 Additional comments (2)
pkg/templates/parser.go (1)

36-41: LGTM! Well-structured constructor function.

The new constructor function enhances flexibility by allowing cache reuse while maintaining clean separation of concerns.

pkg/protocols/offlinehttp/request.go (1)

95-95: LGTM! Enhanced URL handling.

The changes improve URL handling by separating input data from URL string and adding fallback URL construction.

Also applies to: 116-119

cmd/integration-test/ssl.go Show resolved Hide resolved
cmd/integration-test/ssl.go Dismissed Show dismissed Hide dismissed
cmd/integration-test/ssl.go Dismissed Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
cmd/integration-test/integration-test.go (1)

118-121: Consider enhancing the success message and making it platform-agnostic.

While the changes improve CI feedback, consider these enhancements:

  1. Include the number of successful tests in the message for better visibility.
  2. Make the group markers conditional based on the CI platform to support other CI systems.
 } else {
-    fmt.Println("::group::All tests passed")
-    fmt.Println("::endgroup::")
+    successMsg := fmt.Sprintf("All %d tests passed successfully", len(protocolTests))
+    if ci.IsGitHubActions() {
+        fmt.Println("::group::" + successMsg)
+        fmt.Println("::endgroup::")
+    } else {
+        fmt.Println(successMsg)
+    }
     os.Exit(0)
 }
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8d7e21b and e11ce5a.

📒 Files selected for processing (1)
  • cmd/integration-test/integration-test.go (1 hunks)

Copy link
Member Author

@ehsandeep ehsandeep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally all test are passing!

@ehsandeep ehsandeep merged commit 53748c4 into dev Jan 31, 2025
20 checks passed
@ehsandeep ehsandeep deleted the misc-sdk-changes branch January 31, 2025 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants