fix(deps): replace go-tree-sitter with pure-Go goja parser#1510
fix(deps): replace go-tree-sitter with pure-Go goja parser#1510dalledajay-coder wants to merge 2 commits intoprojectdiscovery:devfrom
Conversation
This commit removes the CGO dependency on go-tree-sitter by replacing BishopFox's jsluice with a pure-Go implementation using dop251/goja parser. Changes: - Replace jsluice dependency with dop251/goja (pure-Go JavaScript parser) - Rewrite ExtractJsluiceEndpoints using goja's AST walker - Remove platform-specific build constraints (parser_nojs.go deleted) - Enable jsluice functionality on all platforms (Windows, 32-bit, darwin/arm64) Benefits: - No CGO required (CGO_ENABLED=0 builds work) - Simplified cross-platform compilation - Works on darwin/arm64 without cross-compilers - All existing tests pass Fixes projectdiscovery#1367
WalkthroughReplaces the jsluice dependency with a goja/parser-based JavaScript AST extractor (with a regex fallback), removes Windows/386-specific parser file and the Changes
Sequence DiagramsequenceDiagram
participant Input as JavaScript Input
participant Parser as goja Parser
participant AST as AST Traverser
participant Extractor as Endpoint Extractor
participant Regex as Regex Fallback
participant Output as Endpoints
Input->>Parser: Parse JavaScript code
alt Parse Success
Parser->>AST: Provide AST nodes
AST->>Extractor: Traverse nodes (strings, templates, calls, constructors)
Extractor->>Extractor: Extract, synthesize, deduplicate endpoints
Extractor->>Output: Return endpoints
else Parse Failure
Parser->>Regex: Fallback regex scan
Regex->>Output: Return regex-extracted endpoints
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@pkg/utils/jsluice.go`:
- Around line 679-697: The regexes in endpointExtractor.extractWithRegex are
compiled on every call; move their compilation to package initialization by
creating a package-level variable (e.g., fallbackURLPatterns []*regexp.Regexp)
containing the four regexp.MustCompile(...) entries, and then update
extractWithRegex to iterate over fallbackURLPatterns instead of recompiling
patterns; ensure the new var name (fallbackURLPatterns) is used in
extractWithRegex and that the package imports regexp remains intact.
🧹 Nitpick comments (1)
pkg/utils/jsluice_test.go (1)
103-119: Consider verifying endpoint count to catch false positives.The current test only verifies that expected URLs are present but doesn't check for unexpected URLs being extracted. This could mask regressions where the extractor starts extracting spurious endpoints.
♻️ Suggested improvement
for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { endpoints := ExtractJsluiceEndpoints(tt.input) // Create a map of found URLs for easier checking foundURLs := make(map[string]bool) for _, ep := range endpoints { foundURLs[ep.Endpoint] = true } // Check that all expected URLs are found for _, wantURL := range tt.wantURLs { if !foundURLs[wantURL] { t.Errorf("ExtractJsluiceEndpoints() missing expected URL %q, got %v", wantURL, endpoints) } } + + // Check that no unexpected URLs were extracted + if len(endpoints) != len(tt.wantURLs) { + t.Errorf("ExtractJsluiceEndpoints() returned %d endpoints, want %d", len(endpoints), len(tt.wantURLs)) + } }) }
- Move fallback regex patterns to package-level initialization - Add endpoint count verification in tests to catch false positives - Fix window.open extraction to not capture HTTP methods from xhr.open Co-Authored-By: Claude Opus 4.5 <[email protected]>
|
@dalledajay-coder Please make sure you to add necessary details, this is the PR template: Proposed changesProofChecklist
I've updated the description. |
|
@dalledajay-coder Reminder for updating the PR description according to the PR Template. |
dogancanbakir
left a comment
There was a problem hiding this comment.
- merge conflict
- UPDATE docs
Proposed changes
This commit removes the CGO dependency on go-tree-sitter by replacing BishopFox's jsluice with a pure-Go implementation using dop251/goja parser.
Changes:
Benefits:
/claim #1367
Proof
Checklist
Summary by CodeRabbit
New Features
Improvements
Removals
Tests
Chores