diff --git a/docs/modules.md b/docs/modules.md index 7b5836d8..30e736ed 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -115,6 +115,42 @@ http: each payload creates a separate request for each path. +payloads can also be **named sets**, one per fuzzing position, which lets a +single module vary more than one place in the request at once. write a mapping +instead of a list, and reference each set by its name: + +```yaml +http: + paths: + - "{{BaseURL}}/?user={{user}}&role={{role}}" + + payloads: + user: + - "admin" + - "root" + role: + - "1" + - "2" +``` + +set order is the order you declare them, which is what `attack` pairs on. a +plain list is still accepted and desugars to one set named `payload`, so +`{{payload}}` keeps working unchanged. + +a set whose value is a single string is read from that file, one payload per +line, so a large wordlist does not have to live inside the module: + +```yaml + payloads: + user: "/usr/share/sif/wordlists/users.txt" +``` + +requests are generated lazily, one combination at a time, so a module crossing +several large sets does not build the whole product in memory first. +`-fuzz-max-requests` (default 25000, 0 for unlimited) caps how many requests one +fuzzing module may send per target, so a combinatorial blowup stops itself +instead of running until the scan is killed. + #### attack how paths and payloads combine into requests. @@ -126,6 +162,9 @@ http: - `clusterbomb` (default) - every path is tried with every payload - `pitchfork` - path and payload are paired by index, stopping at the shorter list +- `batteringram` - one value at a time from the first set is broadcast into every + named position, stopping at that set's length. use it when the same value has + to appear in several places of one request at once #### wordlist @@ -310,6 +349,48 @@ both 32-bit forms are accepted, so values from shodan or any favicon-hash tool drop in without conversion. pair it with a `status: 200` matcher so an error page served for `/favicon.ico` is not hashed. a finding fires when the body hashes to any listed value. +### dsl matcher + +evaluate one or more boolean expressions against the response. the syntax and +variable names are nuclei's, so an expression written for a nuclei template +pastes in unchanged. + +```yaml +matchers: + - type: dsl + dsl: + - "status_code == 200 && contains(body, 'admin')" + - "content_length > 1024" +``` + +the variables bound for every expression: + +| variable | type | value | +|----------|------|-------| +| `status_code` | int | response status code | +| `body` | string | response body, after the 5 MB cap | +| `content_length` | int | length of `body` in bytes | +| `header` / `all_headers` | string | the response headers, one `Name: value` per line | +| `duration` | float | round-trip time in seconds | +| `host` | string | `host[:port]` of the request url | + +named extractor values are bound too, so an expression can test something an +extractor pulled out of the same response; in a request chain it also sees the +variables earlier steps extracted. an extractor whose name collides with a +builtin shadows it, matching nuclei's mutation order. + +helper functions are an allowlist, not a blocklist: string inspection and +transforms, `regex`/`regex_all`/`regex_any`, base64/hex/url/html encode and +decode, `md5`/`sha1`/`sha256`/`mmh3`, and numeric conversion. anything else +fails at load, so a dependency bump cannot quietly introduce a helper that reads +files, makes its own requests, or allocates without bound. expressions are also +capped at 4096 bytes. + +expressions are compiled and checked when the module loads, so a typo fails the +module up front instead of silently never matching. at match time an expression +that errors or yields a non-boolean counts as a miss. multiple expressions +combine with AND by default, or with `condition: or`. + ### combining matchers multiple matchers are combined with AND logic by default. diff --git a/docs/usage.md b/docs/usage.md index ec5f9642..e2573816 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -50,6 +50,22 @@ sizes: `small`, `medium`, `large` ./sif -u https://example.com -dirlist medium ``` +#### fuzzing budget + +`-fuzz-max-requests ` - cap the requests a single fuzzing module may send per +target (default 25000, `0` for unlimited). a module crossing several payload sets +multiplies them out, so this is the stop that keeps one module from consuming the +whole scan. + +```bash +./sif -u https://example.com -all-modules -fuzz-max-requests 5000 +``` + +`-fuzz-global-max-requests ` - total fuzz requests across every module and +every target in the run (default 100000, `0` for unlimited). the per-module cap +above bounds one module; this bounds the whole scan, so a target list full of +fuzzing modules cannot multiply past it. + #### response filters modern apps serve a catch-all 200 for unknown routes, so a naive scan reports diff --git a/go.mod b/go.mod index 4cb0bbdf..79c455f4 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,9 @@ require ( github.com/charmbracelet/log v1.0.0 github.com/gocolly/colly/v2 v2.3.0 github.com/likexian/whois v1.15.7 + github.com/projectdiscovery/dsl v0.8.20 github.com/projectdiscovery/goflags v0.1.74 + github.com/projectdiscovery/govaluate v0.0.0-20260615100919-5ee2581bbf7e github.com/projectdiscovery/nuclei/v3 v3.11.0 github.com/projectdiscovery/retryabledns v1.0.115 github.com/projectdiscovery/utils v0.11.1 @@ -272,7 +274,6 @@ require ( github.com/projectdiscovery/blackrock v0.0.1 // indirect github.com/projectdiscovery/cdncheck v1.2.42 // indirect github.com/projectdiscovery/clistats v0.1.4 // indirect - github.com/projectdiscovery/dsl v0.8.20 // indirect github.com/projectdiscovery/fastdialer v0.5.11 // indirect github.com/projectdiscovery/fasttemplate v0.0.2 // indirect github.com/projectdiscovery/freeport v0.0.7 // indirect @@ -282,7 +283,6 @@ require ( github.com/projectdiscovery/goja_nodejs v0.0.0-20260618132410-8519f75f703d // indirect github.com/projectdiscovery/gologger v1.1.71 // indirect github.com/projectdiscovery/gostruct v0.0.2 // indirect - github.com/projectdiscovery/govaluate v0.0.0-20260615100919-5ee2581bbf7e // indirect github.com/projectdiscovery/gozero v0.1.1-0.20260530071156-fa1dad563d76 // indirect github.com/projectdiscovery/hmap v0.0.101 // indirect github.com/projectdiscovery/httpx v1.9.0 // indirect diff --git a/internal/config/config.go b/internal/config/config.go index dabc82f0..512a089a 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -21,75 +21,77 @@ import ( ) type Settings struct { - Dirlist string - DirMatchCodes string // -mc dirlist: status codes to keep - DirFilterCodes string // -fc dirlist: status codes to drop - DirFilterSizes string // -fs dirlist: body sizes to drop - DirFilterWords string // -fw dirlist: word counts to drop - DirFilterRegex string // -fr dirlist: regex; body match drops response - Calibrate bool // -ac auto-calibrate the soft-404 baseline (dirlist, sql) - DirWordlist string // -w dirlist: custom wordlist (file path or url) - DirExtensions string // -e dirlist: extensions appended to each word - Dnslist string - Resolvers string // -resolvers dnslist: comma list overriding the bundled pool - Debug bool - LogDir string - NoScan bool - Ports string - Dorking bool - Git bool - Whois bool - Threads int - Concurrency int - Nuclei bool - JavaScript bool - Timeout time.Duration - URLs goflags.StringSlice - File string - ApiMode bool - Template string - CMS bool - Headers bool - SecurityHeaders bool - CloudStorage bool - SubdomainTakeover bool - Shodan bool - SecurityTrails bool - SQL bool - LFI bool - JWT bool - OpenAPI bool - Favicon bool - CORS bool - Redirect bool - XSS bool - Framework bool - Crawl bool - CrawlDepth int - TLSCert bool - TLSCertPort int - Passive bool - Probe bool - SARIF string // path to write a sarif 2.1.0 report to ("" = off) - Markdown string // path to write a markdown report to ("" = off) - JSONReport string // path to write a json findings report to ("" = off) - Silent bool // route chrome to stderr, print one finding per line to stdout - Diff bool // surface only findings added/removed vs the last snapshot - Store string // snapshot dir for diff mode ("" = default state dir) - Modules string // Comma-separated list of module IDs to run - ModuleTags string // Run modules matching these tags - AllModules bool // Run all loaded modules - ListModules bool // List available modules and exit - Proxy string - Header goflags.StringSlice // custom request headers ("Key: Value") - Cookie string - RateLimit int - MaxRetries int // -max-retries: retries on 429/503 (0 = off) - Notify bool // -notify: ship findings to configured providers - NotifySeverity string // -notify-severity: minimum severity to send (info..critical) - NotifyConfig string // -notify-config: path to a notify-compatible yaml file - ConfigFile string // -config: path to a yaml config file ("" = default ~/.config/sif/config.yaml) - Profile string // -profile: named profile overlay from the config file + Dirlist string + DirMatchCodes string // -mc dirlist: status codes to keep + DirFilterCodes string // -fc dirlist: status codes to drop + DirFilterSizes string // -fs dirlist: body sizes to drop + DirFilterWords string // -fw dirlist: word counts to drop + DirFilterRegex string // -fr dirlist: regex; body match drops response + Calibrate bool // -ac auto-calibrate the soft-404 baseline (dirlist, sql) + DirWordlist string // -w dirlist: custom wordlist (file path or url) + DirExtensions string // -e dirlist: extensions appended to each word + Dnslist string + Resolvers string // -resolvers dnslist: comma list overriding the bundled pool + Debug bool + LogDir string + NoScan bool + Ports string + Dorking bool + Git bool + Whois bool + Threads int + Concurrency int + Nuclei bool + JavaScript bool + Timeout time.Duration + URLs goflags.StringSlice + File string + ApiMode bool + Template string + CMS bool + Headers bool + SecurityHeaders bool + CloudStorage bool + SubdomainTakeover bool + Shodan bool + SecurityTrails bool + SQL bool + LFI bool + JWT bool + OpenAPI bool + Favicon bool + CORS bool + Redirect bool + XSS bool + Framework bool + Crawl bool + CrawlDepth int + TLSCert bool + TLSCertPort int + Passive bool + Probe bool + SARIF string // path to write a sarif 2.1.0 report to ("" = off) + Markdown string // path to write a markdown report to ("" = off) + JSONReport string // path to write a json findings report to ("" = off) + Silent bool // route chrome to stderr, print one finding per line to stdout + Diff bool // surface only findings added/removed vs the last snapshot + Store string // snapshot dir for diff mode ("" = default state dir) + Modules string // Comma-separated list of module IDs to run + ModuleTags string // Run modules matching these tags + AllModules bool // Run all loaded modules + ListModules bool // List available modules and exit + Proxy string + Header goflags.StringSlice // custom request headers ("Key: Value") + Cookie string + RateLimit int + MaxRetries int // -max-retries: retries on 429/503 (0 = off) + FuzzMaxRequests int // -fuzz-max-requests: cap per fuzzing module per target (0 = unlimited) + FuzzGlobalMaxRequests int // -fuzz-global-max-requests: scan-wide cap shared by every fuzzing module (0 = unlimited) + Notify bool // -notify: ship findings to configured providers + NotifySeverity string // -notify-severity: minimum severity to send (info..critical) + NotifyConfig string // -notify-config: path to a notify-compatible yaml file + ConfigFile string // -config: path to a yaml config file ("" = default ~/.config/sif/config.yaml) + Profile string // -profile: named profile overlay from the config file } // minThreads is the floor for the worker count. Threads feeds wg.Add across the @@ -194,6 +196,8 @@ func registerFlags(settings *Settings) *goflags.FlagSet { flagSet.StringVar(&settings.Cookie, "cookie", "", "Cookie header to send with every request"), flagSet.IntVar(&settings.RateLimit, "rate-limit", 0, "Max requests per second (0 = unlimited)"), flagSet.IntVar(&settings.MaxRetries, "max-retries", 2, "Retries on 429/503 with Retry-After backoff (0 = off)"), + flagSet.IntVar(&settings.FuzzMaxRequests, "fuzz-max-requests", 25000, "Max requests a single fuzzing module may send per target (0 = unlimited)"), + flagSet.IntVar(&settings.FuzzGlobalMaxRequests, "fuzz-global-max-requests", 100000, "Max total fuzz requests across every module and target in the scan (0 = unlimited)"), ) flagSet.CreateGroup("output", "Output", diff --git a/internal/modules/attack_modes_test.go b/internal/modules/attack_modes_test.go index 6b8911f8..4b848fde 100644 --- a/internal/modules/attack_modes_test.go +++ b/internal/modules/attack_modes_test.go @@ -59,7 +59,7 @@ func TestGenerateHTTPRequestsAttack(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - cfg := &HTTPConfig{Paths: tt.paths, Payloads: tt.payloads, Attack: tt.attack} + cfg := &HTTPConfig{Paths: tt.paths, Payloads: legacyPayloads(tt.payloads), Attack: tt.attack} reqs, err := generateHTTPRequests(target, cfg) if err != nil { t.Fatalf("generateHTTPRequests: %v", err) @@ -75,12 +75,12 @@ func TestGenerateHTTPRequestsAttack(t *testing.T) { } func TestValidateAttack(t *testing.T) { - for _, ok := range []string{"", "clusterbomb", "pitchfork", "Pitchfork", "CLUSTERBOMB"} { + for _, ok := range []string{"", "clusterbomb", "pitchfork", "Pitchfork", "CLUSTERBOMB", "batteringram", "BatteringRam"} { if err := validateAttack(ok); err != nil { t.Errorf("validateAttack(%q) = %v, want nil", ok, err) } } - for _, bad := range []string{"sniper", "batteringram", "bogus"} { + for _, bad := range []string{"sniper", "bogus"} { if err := validateAttack(bad); err == nil { t.Errorf("validateAttack(%q) = nil, want error", bad) } @@ -127,7 +127,7 @@ func TestExecuteHTTPModulePitchfork(t *testing.T) { HTTP: &HTTPConfig{ Attack: "pitchfork", Paths: []string{"{{BaseURL}}/a?x={{payload}}", "{{BaseURL}}/b?x={{payload}}"}, - Payloads: []string{"1", "2"}, + Payloads: legacyPayloads([]string{"1", "2"}), Matchers: []Matcher{{Type: "word", Part: "body", Words: []string{"ok"}}}, }, } diff --git a/internal/modules/dsl.go b/internal/modules/dsl.go new file mode 100644 index 00000000..ba5dc603 --- /dev/null +++ b/internal/modules/dsl.go @@ -0,0 +1,182 @@ +/* +·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· +: : +: █▀ █ █▀▀ · Blazing-fast pentesting suite : +: ▄█ █ █▀ · BSD 3-Clause License : +: : +: (c) 2022-2026 vmfunc, xyzeva, : +: lunchcat alumni & contributors : +: : +·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· +*/ + +package modules + +import ( + "fmt" + "net/http" + "net/url" + "strings" + "sync" + + "github.com/projectdiscovery/dsl" + "github.com/projectdiscovery/govaluate" +) + +// maxDSLExprLen bounds a single dsl expression. govaluate's Evaluate takes no +// context and cannot be interrupted mid-helper, so capping the input is the +// reliable defense against a pathological expression. +const maxDSLExprLen = 4096 + +// allowedDSLHelpers is the set of helper functions a dsl expression may call, +// keyed by the underscore-stripped name so both alias forms (to_lower and +// tolower) resolve. It is an allowlist so a future projectdiscovery/dsl bump +// cannot silently reintroduce a side-effecting helper (llm_prompt, public_ip, +// wait_for, the gadget generators) or an unbounded-allocation one (repeat, the +// rand_* and faker families): anything unnamed here fails to compile. +var allowedDSLHelpers = func() map[string]bool { + names := []string{ + // string inspection / comparison + "contains", "contains_all", "contains_any", "starts_with", "ends_with", + "line_starts_with", "line_ends_with", "equals_any", "len", "index", + // string transforms + "to_lower", "to_upper", "trim", "trim_left", "trim_right", "trim_space", + "trim_prefix", "trim_suffix", "split", "join", "replace", "replace_regex", + "concat", "reverse", + // regex (RE2, linear-time) + "regex", "regex_all", "regex_any", + // encode / decode + "base64", "base64_decode", "hex_encode", "hex_decode", + "url_encode", "url_decode", "html_escape", "html_unescape", + // hashing / fingerprint + "md5", "sha1", "sha256", "mmh3", + // numeric / conversion + "to_number", "to_string", + } + m := make(map[string]bool, len(names)) + for _, n := range names { + m[stripUnderscore(n)] = true + } + return m +}() + +func stripUnderscore(s string) string { return strings.ReplaceAll(s, "_", "") } + +// dslHelpers is the curated govaluate function map: every entry of +// dsl.HelperFunctions() whose (underscore-stripped) name is allowlisted. +var dslHelpers = func() map[string]govaluate.ExpressionFunction { + all := dsl.HelperFunctions() + out := make(map[string]govaluate.ExpressionFunction, len(all)) + for name, fn := range all { + if allowedDSLHelpers[stripUnderscore(name)] { + out[name] = fn + } + } + return out +}() + +// dslCache memoizes compiled expressions by source string. A compiled +// *EvaluableExpression is safe to Evaluate concurrently (value receiver, pooled +// scratch state), so sharing one across goroutines is fine. +var dslCache sync.Map // string -> *govaluate.EvaluableExpression + +// hasNonEmptyDSL reports whether exprs holds at least one non-empty expression. +func hasNonEmptyDSL(exprs []string) bool { + for _, e := range exprs { + if strings.TrimSpace(e) != "" { + return true + } + } + return false +} + +// dslCompile compiles expr against the curated helpers, caching the result, so +// an over-length expression, a syntax error or a non-allowlisted function is +// rejected at module load rather than silently missing at match time. +func dslCompile(expr string) (*govaluate.EvaluableExpression, error) { + if len(expr) > maxDSLExprLen { + return nil, fmt.Errorf("dsl expression exceeds %d bytes", maxDSLExprLen) + } + if cached, ok := dslCache.Load(expr); ok { + return cached.(*govaluate.EvaluableExpression), nil + } + compiled, err := govaluate.NewEvaluableExpressionWithFunctions(expr, dslHelpers) + if err != nil { + return nil, fmt.Errorf("dsl expression %q: %w", expr, err) + } + actual, _ := dslCache.LoadOrStore(expr, compiled) + return actual.(*govaluate.EvaluableExpression), nil +} + +// dslVars builds the variable environment a dsl expression evaluates against, +// using nuclei's lowercase names so nuclei dsl expressions paste in unchanged. +// Named extractor values overlay the builtins (matching nuclei's mutation +// order), so an extractor may reference, and on a name clash shadow, a builtin. +func dslVars(mc *MatchContext) map[string]interface{} { + headers := getPart("header", mc.Resp, mc.Body) + vars := map[string]interface{}{ + "status_code": statusCodeOf(mc.Resp), + "body": mc.Body, + "content_length": len(mc.Body), + "all_headers": headers, + "header": headers, + "duration": mc.Duration.Seconds(), + "host": hostOf(mc.URL), + } + for k, v := range mc.Extracted { + vars[k] = v + } + return vars +} + +// hostOf returns the host[:port] of a request URL, matching nuclei's `host` +// variable so a pasted nuclei expression (host == "example.com") behaves as +// expected. On an unparseable URL it falls back to the raw string rather than +// binding an empty host. +func hostOf(rawURL string) string { + if u, err := url.Parse(rawURL); err == nil && u.Host != "" { + return u.Host + } + return rawURL +} + +func statusCodeOf(resp *http.Response) int { + if resp == nil { + return 0 + } + return resp.StatusCode +} + +// evalDSL folds a dsl matcher's expressions under its condition (default AND). +// An expression that errors at eval, or yields a non-bool, counts as false +// (fail-closed), matching the engine's swallow-at-match-time invariant. +func evalDSL(m *Matcher, mc *MatchContext) bool { + if len(m.DSL) == 0 { + return false + } + vars := dslVars(mc) + or := strings.EqualFold(m.Condition, "or") + for _, expr := range m.DSL { + matched := evalOneDSL(expr, vars) + if or && matched { + return true + } + if !or && !matched { + return false + } + } + return !or +} + +func evalOneDSL(expr string, vars map[string]interface{}) bool { + compiled, err := dslCompile(expr) + if err != nil { + return false // unreachable after load validation; fail closed anyway + } + result, err := compiled.Evaluate(vars) + if err != nil { + return false // unbound var / type mismatch -> miss + } + b, ok := result.(bool) + return ok && b +} diff --git a/internal/modules/dsl_test.go b/internal/modules/dsl_test.go new file mode 100644 index 00000000..a2ad3c55 --- /dev/null +++ b/internal/modules/dsl_test.go @@ -0,0 +1,152 @@ +/* +·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· +: : +: █▀ █ █▀▀ · Blazing-fast pentesting suite : +: ▄█ █ █▀ · BSD 3-Clause License : +: : +: (c) 2022-2026 vmfunc, xyzeva, : +: lunchcat alumni & contributors : +: : +·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· +*/ + +package modules + +import ( + "net/http" + "strings" + "testing" +) + +func TestDSLCompile(t *testing.T) { + tests := []struct { + name string + expr string + wantErr bool + }{ + {"valid comparison", "status_code == 200", false}, + {"valid helper", `contains(body, "admin")`, false}, + {"bad syntax", "status_code ==", true}, + {"non-allowlisted side-effect helper", "wait_for(1)", true}, + {"non-allowlisted network helper", "public_ip()", true}, + {"non-allowlisted alloc helper", `repeat("A", 1000000)`, true}, + {"over-length expression", strings.Repeat("a", maxDSLExprLen+1), true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := dslCompile(tt.expr) + if (err != nil) != tt.wantErr { + t.Fatalf("dslCompile(%q) err = %v, wantErr %v", tt.expr, err, tt.wantErr) + } + }) + } +} + +func TestCheckMatcherDSL(t *testing.T) { + const body = "welcome admin dashboard" + resp := fakeResponse(t, 200, http.Header{"X-Powered-By": []string{"nginx"}}) + mc := &MatchContext{Resp: resp, Body: body} + + tests := []struct { + name string + dsl []string + expect bool + }{ + {"status_code true", []string{"status_code == 200"}, true}, + {"status_code false", []string{"status_code == 500"}, false}, + {"body contains", []string{`contains(body, "admin")`}, true}, + {"content_length", []string{"content_length > 5"}, true}, + {"all_headers", []string{`contains(to_lower(all_headers), "nginx")`}, true}, + {"header alias", []string{`contains(to_lower(header), "nginx")`}, true}, + {"unbound var misses", []string{"nonexistent_var == 1"}, false}, + {"non-bool result misses", []string{"len(body)"}, false}, + {"empty list false", nil, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + m := &Matcher{Type: "dsl", DSL: tt.dsl} + if got := checkMatcher(m, mc); got != tt.expect { + t.Errorf("dsl %v = %v, want %v", tt.dsl, got, tt.expect) + } + }) + } +} + +func TestCheckMatcherDSLCondition(t *testing.T) { + const body = "hello world" + mc := &MatchContext{Resp: fakeResponse(t, 200, nil), Body: body} + trueExpr := "status_code == 200" + falseExpr := "status_code == 500" + + tests := []struct { + name string + condition string + dsl []string + expect bool + }{ + {"and both true", "and", []string{trueExpr, `contains(body, "hello")`}, true}, + {"and one false", "and", []string{trueExpr, falseExpr}, false}, + {"empty defaults to and", "", []string{trueExpr, falseExpr}, false}, + {"or one true", "or", []string{falseExpr, trueExpr}, true}, + {"or none true", "or", []string{falseExpr, `contains(body, "absent")`}, false}, + {"AND case-insensitive", "AND", []string{trueExpr, falseExpr}, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + m := &Matcher{Type: "dsl", Condition: tt.condition, DSL: tt.dsl} + if got := checkMatcher(m, mc); got != tt.expect { + t.Errorf("dsl cond %q %v = %v, want %v", tt.condition, tt.dsl, got, tt.expect) + } + }) + } +} + +func TestCheckMatcherDSLNegative(t *testing.T) { + mc := &MatchContext{Resp: fakeResponse(t, 200, nil), Body: "x"} + ms := []Matcher{{Type: "dsl", DSL: []string{"status_code == 200"}, Negative: true}} + if checkMatchers(ms, "", mc) { + t.Error("negative dsl matcher on a matching response should yield false") + } +} + +func TestCheckMatcherDSLHost(t *testing.T) { + // host must bind the hostname (nuclei parity), not the full requested URL, + // so a pasted `host == "..."` expression behaves as a nuclei user expects. + mc := &MatchContext{ + Resp: fakeResponse(t, 200, nil), + Body: "x", + URL: "http://example.com/admin/panel", + } + if m := (&Matcher{Type: "dsl", DSL: []string{`host == "example.com"`}}); !checkMatcher(m, mc) { + t.Error(`host should bind the hostname "example.com", got a miss`) + } + if m := (&Matcher{Type: "dsl", DSL: []string{`contains(host, "/admin")`}}); checkMatcher(m, mc) { + t.Error("host must not contain the URL path") + } +} + +func TestCheckMatcherDSLExtractorVar(t *testing.T) { + mc := &MatchContext{ + Resp: fakeResponse(t, 200, nil), + Body: "x", + Extracted: map[string]string{"version": "1.2.3"}, + } + m := &Matcher{Type: "dsl", DSL: []string{`version == "1.2.3"`}} + if !checkMatcher(m, mc) { + t.Error("dsl matcher should see the named extractor variable") + } +} + +func TestCheckMatcherDSLEvalErrorMisses(t *testing.T) { + mc := &MatchContext{Resp: fakeResponse(t, 200, nil), Body: "x"} + // compiles fine, errors at eval (string vs number comparison) + m := &Matcher{Type: "dsl", DSL: []string{`body > 5`}} + if checkMatcher(m, mc) { + t.Error("an expression that errors at eval must miss, not match") + } + // under or, a bad-eval expr followed by a valid true expr still matches + m2 := &Matcher{Type: "dsl", Condition: "or", DSL: []string{`body > 5`, "status_code == 200"}} + if !checkMatcher(m2, mc) { + t.Error("or with a valid true expr after a bad-eval expr should match") + } +} diff --git a/internal/modules/executor.go b/internal/modules/executor.go index 2e165e04..e7b5a0e8 100644 --- a/internal/modules/executor.go +++ b/internal/modules/executor.go @@ -18,13 +18,17 @@ import ( "errors" "fmt" "io" + "iter" "net/http" "os" "regexp" + "sort" "strings" "sync" + "sync/atomic" "time" + "github.com/charmbracelet/log" "github.com/tidwall/gjson" "github.com/vmfunc/sif/internal/httpx" ) @@ -34,6 +38,40 @@ import ( // from mistaking "not implemented" for "scanned, found nothing". var ErrUnsupportedModuleType = errors.New("unsupported module type") +// FuzzBudget is a scan-wide ceiling on fuzz requests shared by every +// module's producer, on top of each module's own FuzzMaxRequests. A nil +// *FuzzBudget is always unlimited, so callers never need to nil-check. +type FuzzBudget struct { + max int64 + sent atomic.Int64 + warned atomic.Bool +} + +// NewFuzzBudget returns nil (unlimited) for maxRequests <= 0. +func NewFuzzBudget(maxRequests int) *FuzzBudget { + if maxRequests <= 0 { + return nil + } + return &FuzzBudget{max: int64(maxRequests)} +} + +// Reserve claims one request against the budget and reports whether it fit. +// Safe for concurrent use; a nil receiver always reports true. +func (b *FuzzBudget) Reserve() bool { + if b == nil { + return true + } + return b.sent.Add(1) <= b.max +} + +// warnOnce logs the exhaustion line exactly once no matter how many +// producers hit it concurrently. +func (b *FuzzBudget) warnOnce(moduleID, target string) { + if b.warned.CompareAndSwap(false, true) { + log.Warnf("fuzz: scan-wide budget of %d requests exhausted (module %s on %s hit it; further fuzz requests across all modules are skipped)", b.max, moduleID, target) + } +} + // httpRequest represents a generated HTTP request. type httpRequest struct { Method string @@ -89,11 +127,21 @@ func ExecuteHTTPModule(ctx context.Context, target string, def *YAMLModule, opts return executeHTTPChain(ctx, client, target, def) } - // Generate requests based on paths and payloads - requests, err := generateHTTPRequests(target, cfg) + // Resolve paths and payload sets up front (the only failing steps); the + // product itself is streamed and never materialized. + paths, err := resolvePaths(cfg) if err != nil { return nil, err } + sets, err := resolveSets(cfg) + if err != nil { + return nil, err + } + for _, s := range sets { + if len(s.Values) == 0 { + log.Warnf("fuzz: module %s has empty payload set %q on %s; no requests sent", def.ID, s.Name, target) + } + } // Determine thread count threads := cfg.Threads @@ -104,43 +152,66 @@ func ExecuteHTTPModule(ctx context.Context, target string, def *YAMLModule, opts threads = 10 } - // Execute requests concurrently - var wg sync.WaitGroup - var mu sync.Mutex - resultsChan := make(chan Finding, len(requests)) - - // Limit concurrency - sem := make(chan struct{}, threads) + reqCh := make(chan *httpRequest) + resultCh := make(chan Finding) - for _, req := range requests { - select { - case <-ctx.Done(): - return result, ctx.Err() - case sem <- struct{}{}: + // Producer: stream combinations into reqCh, stopping at the request budget + // (0 = unlimited) and logging a single truncation line. Watches ctx so a + // cancelled run stops pulling promptly. + go func() { + defer close(reqCh) + var sent int + budget := opts.FuzzMaxRequests + for req := range streamRequests(target, cfg, paths, sets) { + if ctx.Err() != nil { + return + } + if budget > 0 && sent >= budget { + log.Warnf("fuzz: module %s hit the %d-request budget on %s (further combinations skipped)", def.ID, budget, target) + return + } + if !opts.FuzzGlobalBudget.Reserve() { + opts.FuzzGlobalBudget.warnOnce(def.ID, target) + return + } + select { + case <-ctx.Done(): + return + case reqCh <- req: + sent++ + } } + }() - wg.Add(1) - go func(r *httpRequest) { + // Workers: a fixed pool pulling from reqCh; matches flow to resultCh. + var wg sync.WaitGroup + wg.Add(threads) + for i := 0; i < threads; i++ { + go func() { defer wg.Done() - defer func() { <-sem }() - - finding, ok := executeHTTPRequest(ctx, client, r, cfg, def.Info.Severity) - if ok { - resultsChan <- finding + for r := range reqCh { + if ctx.Err() != nil { + return + } + if finding, ok := executeHTTPRequest(ctx, client, r, cfg, def.Info.Severity); ok { + select { + case <-ctx.Done(): + return + case resultCh <- finding: + } + } } - }(req) + }() } - // Collect results go func() { wg.Wait() - close(resultsChan) + close(resultCh) }() - for finding := range resultsChan { - mu.Lock() + // Collector: single consumer owns result.Findings, so no mutex is needed. + for finding := range resultCh { result.Findings = append(result.Findings, finding) - mu.Unlock() } return result, nil @@ -189,7 +260,9 @@ func executeHTTPChain(ctx context.Context, client *http.Client, target string, d req.Header.Set("User-Agent", defaultUserAgent) } + start := time.Now() resp, err := client.Do(req) + elapsed := time.Since(start) if err != nil { // a transport error breaks the chain; return whatever matched earlier. return result, nil @@ -210,7 +283,14 @@ func executeHTTPChain(ctx context.Context, client *http.Client, target string, d // a step with matchers gates the chain: a match records a finding, a miss // means the precondition failed so the chain stops here. if len(step.Matchers) > 0 { - if !checkMatchers(step.Matchers, step.MatchersCondition, resp, respStr) { + mc := &MatchContext{ + Resp: resp, + Body: respStr, + URL: url, + Duration: elapsed, + Extracted: vars, + } + if !checkMatchers(step.Matchers, step.MatchersCondition, mc) { break } result.Findings = append(result.Findings, Finding{ @@ -238,58 +318,197 @@ func snapshotVars(vars map[string]string) map[string]string { return out } -// generateHTTPRequests creates all requests based on paths and payloads. +// generateHTTPRequests materializes every fuzz request. The streaming path in +// ExecuteHTTPModule does not call this; it remains for tests and any caller that +// wants the full slice. It errors only where path resolution can fail. func generateHTTPRequests(target string, cfg *HTTPConfig) ([]*httpRequest, error) { - var requests []*httpRequest - paths, err := resolvePaths(cfg) if err != nil { return nil, err } + sets, err := resolveSets(cfg) + if err != nil { + return nil, err + } + var requests []*httpRequest + for req := range streamRequests(target, cfg, paths, sets) { + requests = append(requests, req) + } + return requests, nil +} - // Ensure target has no trailing slash - target = strings.TrimSuffix(target, "/") +// resolveSets loads any file-backed payload sets into inline values, returning +// the sets ready for streaming. It is the only place set resolution can fail. +func resolveSets(cfg *HTTPConfig) ([]PayloadSet, error) { + sets := cfg.Payloads.Sets + out := make([]PayloadSet, len(sets)) + for i, s := range sets { + if s.File == "" { + out[i] = s + continue + } + vals, err := loadWordlist(s.File) + if err != nil { + return nil, fmt.Errorf("payloads[%s]: %w", s.Name, err) + } + out[i] = PayloadSet{Name: s.Name, Values: vals} + } + return out, nil +} +// streamRequests lazily yields one *httpRequest per fuzz combination, so at +// most one combination exists at a time. clusterbomb crosses paths against +// every set as a nested odometer, rightmost fastest; pitchfork zips them by +// index and stops at the shortest; batteringram broadcasts one value from the +// first set to every position, stopping at that set's length. +func streamRequests(target string, cfg *HTTPConfig, paths []string, sets []PayloadSet) iter.Seq[*httpRequest] { method := cfg.Method if method == "" { method = "GET" } + target = strings.TrimSuffix(target, "/") + + return func(yield func(*httpRequest) bool) { + if len(sets) == 0 { + for _, path := range paths { + if !yield(newFuzzRequest(method, target, path, nil, cfg)) { + return + } + } + return + } - // If no payloads, just use paths directly - if len(cfg.Payloads) == 0 { + if strings.EqualFold(cfg.Attack, "pitchfork") { + n := pitchforkLen(paths, sets) + for i := 0; i < n; i++ { + vars := make(map[string]string, len(sets)) + for _, s := range sets { + vars[s.Name] = s.Values[i] + } + if !yield(newFuzzRequest(method, target, paths[i], vars, cfg)) { + return + } + } + return + } + + if strings.EqualFold(cfg.Attack, "batteringram") { + if len(sets) == 0 || len(sets[0].Values) == 0 { + return + } + for _, path := range paths { + for _, v := range sets[0].Values { + vars := make(map[string]string, len(sets)) + for _, s := range sets { + vars[s.Name] = v + } + if !yield(newFuzzRequest(method, target, path, vars, cfg)) { + return + } + } + } + return + } + + // clusterbomb: an empty set makes the product empty. + for _, s := range sets { + if len(s.Values) == 0 { + return + } + } + idx := make([]int, len(sets)) for _, path := range paths { - url := substituteVariables(path, target, "") - requests = append(requests, &httpRequest{ - Method: method, - URL: url, - Headers: cfg.Headers, - Body: cfg.Body, - Original: path, - }) + for { + vars := make(map[string]string, len(sets)) + for k, s := range sets { + vars[s.Name] = s.Values[idx[k]] + } + if !yield(newFuzzRequest(method, target, path, vars, cfg)) { + return + } + if !advance(idx, sets) { + break + } + } + for k := range idx { + idx[k] = 0 + } } - return requests, nil } +} - // pitchfork pairs path[i] with payload[i] and stops at the shorter list; - // clusterbomb (default) crosses every path with every payload. - if strings.EqualFold(cfg.Attack, "pitchfork") { - n := len(paths) - if len(cfg.Payloads) < n { - n = len(cfg.Payloads) +// advance increments the odometer over set value indices, rightmost fastest, +// and reports whether a next combination exists. +func advance(idx []int, sets []PayloadSet) bool { + for k := len(idx) - 1; k >= 0; k-- { + idx[k]++ + if idx[k] < len(sets[k].Values) { + return true } - for i := 0; i < n; i++ { - requests = append(requests, newPayloadRequest(method, target, paths[i], cfg.Payloads[i], cfg)) + idx[k] = 0 + } + return false +} + +// pitchforkLen is the shortest of the path list and every set, the number of +// index-paired combinations pitchfork emits. +func pitchforkLen(paths []string, sets []PayloadSet) int { + n := len(paths) + for _, s := range sets { + if len(s.Values) < n { + n = len(s.Values) } - return requests, nil } + return n +} - for _, path := range paths { - for _, payload := range cfg.Payloads { - requests = append(requests, newPayloadRequest(method, target, path, payload, cfg)) +// newFuzzRequest builds one request for a combination. It substitutes +// {{BaseURL}}, the legacy {{payload}}/{{Payload}} builtin, and every {{name}} in +// vars into the url, body and each header value. Headers are copied only when a +// substitution could apply, preserving the shared cfg.Headers map otherwise. +func newFuzzRequest(method, target, path string, vars map[string]string, cfg *HTTPConfig) *httpRequest { + pv := vars["payload"] // drives {{payload}}/{{Payload}}; "" when no such set + sub := func(s string) string { return substituteVariablesWithVars(s, target, pv, vars) } + + headers := cfg.Headers + if len(cfg.Headers) > 0 { + headers = make(map[string]string, len(cfg.Headers)) + for k, v := range cfg.Headers { + headers[k] = sub(v) } } + return &httpRequest{ + Method: method, + URL: sub(path), + Headers: headers, + Body: sub(cfg.Body), + Payload: payloadLabel(vars), + Original: path, + } +} - return requests, nil +// payloadLabel renders a combination for the httpRequest.Payload debug field: +// the lone value for a single set (the legacy shape), else name=value pairs in +// name order joined by "&". The field is metadata only; findings key off URL. +func payloadLabel(vars map[string]string) string { + switch len(vars) { + case 0: + return "" + case 1: + for _, v := range vars { + return v + } + } + keys := make([]string, 0, len(vars)) + for k := range vars { + keys = append(keys, k) + } + sort.Strings(keys) + parts := make([]string, len(keys)) + for i, k := range keys { + parts[i] = k + "=" + vars[k] + } + return strings.Join(parts, "&") } // resolvePaths expands a wordlist over any {{word}} path templates so one @@ -345,27 +564,14 @@ func loadWordlist(path string) ([]string, error) { return words, nil } -// newPayloadRequest builds one request with the path and body templates -// substituted for the given payload. -func newPayloadRequest(method, target, path, payload string, cfg *HTTPConfig) *httpRequest { - return &httpRequest{ - Method: method, - URL: substituteVariables(path, target, payload), - Headers: cfg.Headers, - Body: substituteVariables(cfg.Body, target, payload), - Payload: payload, - Original: path, - } -} - -// validateAttack rejects an attack mode that is not "", "clusterbomb", or -// "pitchfork"; an empty value defaults to clusterbomb. +// validateAttack rejects an attack mode that is not "", "clusterbomb", +// "pitchfork", or "batteringram"; an empty value defaults to clusterbomb. func validateAttack(attack string) error { switch strings.ToLower(attack) { - case "", "clusterbomb", "pitchfork": + case "", "clusterbomb", "pitchfork", "batteringram": return nil default: - return fmt.Errorf("invalid attack %q (want \"clusterbomb\" or \"pitchfork\")", attack) + return fmt.Errorf("invalid attack %q (want \"clusterbomb\", \"pitchfork\", or \"batteringram\")", attack) } } @@ -413,7 +619,9 @@ func executeHTTPRequest(ctx context.Context, client *http.Client, r *httpRequest req.Header.Set("User-Agent", defaultUserAgent) } + start := time.Now() resp, err := client.Do(req) + elapsed := time.Since(start) if err != nil { return Finding{}, false } @@ -426,13 +634,22 @@ func executeHTTPRequest(ctx context.Context, client *http.Client, r *httpRequest } bodyStr := string(respBody) - // Check matchers - if !checkMatchers(cfg.Matchers, cfg.MatchersCondition, resp, bodyStr) { - return Finding{}, false + // extract before matching: runExtractors is side-effect-free and the finding + // is only built on a match, so hoisting it is behavior-preserving and lets a + // matcher read extractor values out of the context. + extracted := runExtractors(cfg.Extractors, resp, bodyStr) + + mc := &MatchContext{ + Resp: resp, + Body: bodyStr, + URL: r.URL, + Duration: elapsed, + Extracted: extracted, } - // Extract data - extracted := runExtractors(cfg.Extractors, resp, bodyStr) + if !checkMatchers(cfg.Matchers, cfg.MatchersCondition, mc) { + return Finding{}, false + } // favicon-only matches fire on binary icon bytes; report the hash, not the body. evidence := truncateEvidence(bodyStr) @@ -449,14 +666,14 @@ func executeHTTPRequest(ctx context.Context, client *http.Client, r *httpRequest } // checkMatchers combines matchers with condition "and" (default, all match) or "or" (any). -func checkMatchers(matchers []Matcher, condition string, resp *http.Response, body string) bool { +func checkMatchers(matchers []Matcher, condition string, mc *MatchContext) bool { if len(matchers) == 0 { return false } or := strings.EqualFold(condition, "or") for i := range matchers { - matched := checkMatcher(&matchers[i], resp, body) + matched := checkMatcher(&matchers[i], mc) if matchers[i].Negative { matched = !matched } @@ -483,29 +700,29 @@ func validateMatchersCondition(condition string) error { } // checkMatcher evaluates a single matcher. -func checkMatcher(m *Matcher, resp *http.Response, body string) bool { +func checkMatcher(m *Matcher, mc *MatchContext) bool { switch m.Type { case "status": for _, status := range m.Status { - if resp.StatusCode == status { + if mc.Resp.StatusCode == status { return true } } return false case "word": - return checkWords(getPart(m.Part, resp, body), m.Words, m.Condition, m.CaseInsensitive) + return checkWords(getPart(m.Part, mc.Resp, mc.Body), m.Words, m.Condition, m.CaseInsensitive) case "regex": - return checkRegex(getPart(m.Part, resp, body), m.Regex, m.Condition) + return checkRegex(getPart(m.Part, mc.Resp, mc.Body), m.Regex, m.Condition) case "favicon": - return checkFaviconHash(body, m.Hash) + return checkFaviconHash(mc.Body, m.Hash) case "size": // size matches the response body length against any listed value. for _, n := range m.Size { - if len(body) == n { + if len(mc.Body) == n { return true } } @@ -514,13 +731,16 @@ func checkMatcher(m *Matcher, resp *http.Response, body string) bool { case "range": switch strings.ToLower(m.Source) { case "status": - return inRange(resp.StatusCode, m.Min, m.Max) + return inRange(mc.Resp.StatusCode, m.Min, m.Max) case "size", "": - return inRange(len(body), m.Min, m.Max) + return inRange(len(mc.Body), m.Min, m.Max) default: return false } + case "dsl": + return evalDSL(m, mc) + default: return false } diff --git a/internal/modules/executor_test.go b/internal/modules/executor_test.go index 6a88dd6f..ff961fa3 100644 --- a/internal/modules/executor_test.go +++ b/internal/modules/executor_test.go @@ -129,7 +129,7 @@ func TestExecuteHTTPModulePayloadExpansion(t *testing.T) { Type: TypeHTTP, HTTP: &HTTPConfig{ Paths: []string{"{{BaseURL}}/search?q={{payload}}"}, - Payloads: []string{"safe", "boom"}, + Payloads: legacyPayloads([]string{"safe", "boom"}), Matchers: []Matcher{ {Type: "word", Part: "body", Words: []string{"sql syntax"}}, }, @@ -363,6 +363,40 @@ func TestExecuteHTTPModuleWordlist(t *testing.T) { } } +// drives the full executor with a dsl matcher: fetch a live response, evaluate +// the expression against its bound variables, report exactly one finding. +func TestExecuteHTTPModuleDSL(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Server", "nginx") + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("welcome admin dashboard")) + })) + defer srv.Close() + + def := &YAMLModule{ + ID: "dsl-e2e", + Type: TypeHTTP, + Info: YAMLModuleInfo{Severity: "info"}, + HTTP: &HTTPConfig{ + Method: "GET", + Paths: []string{"{{BaseURL}}/"}, + Matchers: []Matcher{{ + Type: "dsl", + DSL: []string{`status_code == 200 && contains(body, "admin")`}, + }}, + }, + } + opts := Options{Timeout: testTimeout, Client: httpx.Client(testTimeout)} + + result, err := ExecuteHTTPModule(context.Background(), srv.URL, def, opts) + if err != nil { + t.Fatalf("ExecuteHTTPModule: %v", err) + } + if len(result.Findings) != 1 { + t.Fatalf("expected exactly 1 dsl finding, got %d", len(result.Findings)) + } +} + func TestTruncateEvidence(t *testing.T) { short := "short evidence" if got := truncateEvidence(short); got != short { diff --git a/internal/modules/favicon.go b/internal/modules/favicon.go index 7c4b461e..f8a66875 100644 --- a/internal/modules/favicon.go +++ b/internal/modules/favicon.go @@ -68,8 +68,8 @@ func faviconEvidence(matchers []Matcher, body string) (string, bool) { } // validateMatchers fails favicon matchers that would silently never fire (no -// hash, or one out of 32-bit range) and malformed range matchers at load -// rather than at match time. +// hash, or one out of 32-bit range), malformed range matchers, and dsl matchers +// that are empty or do not compile, at load rather than at match time. func validateMatchers(matchers []Matcher) error { for i := range matchers { if matchers[i].Type == "favicon" { @@ -83,6 +83,17 @@ func validateMatchers(matchers []Matcher) error { } } + if matchers[i].Type == "dsl" { + if !hasNonEmptyDSL(matchers[i].DSL) { + return fmt.Errorf("dsl matcher requires at least one non-empty expression") + } + for _, expr := range matchers[i].DSL { + if _, err := dslCompile(expr); err != nil { + return fmt.Errorf("dsl matcher: %w", err) + } + } + } + if matchers[i].Type == "range" { if matchers[i].Min == nil && matchers[i].Max == nil { return fmt.Errorf("range matcher requires min or max") diff --git a/internal/modules/favicon_test.go b/internal/modules/favicon_test.go index 9f23c9a6..aa487fb8 100644 --- a/internal/modules/favicon_test.go +++ b/internal/modules/favicon_test.go @@ -54,7 +54,7 @@ func TestCheckMatcherFavicon(t *testing.T) { t.Run(tt.name, func(t *testing.T) { m := &Matcher{Type: "favicon", Hash: tt.hashes} resp := fakeResponse(t, 200, nil) - if got := checkMatcher(m, resp, body); got != tt.expect { + if got := checkMatcher(m, &MatchContext{Resp: resp, Body: body}); got != tt.expect { t.Errorf("checkMatcher favicon = %v, want %v", got, tt.expect) } }) @@ -197,6 +197,12 @@ func TestValidateMatchers(t *testing.T) { {name: "favicon with no hash", matchers: []Matcher{{Type: "favicon"}}, wantErr: true}, {name: "out-of-range hash", matchers: []Matcher{{Type: "favicon", Hash: []int64{99999999999}}}, wantErr: true}, {name: "non-favicon ignored", matchers: []Matcher{{Type: "word", Words: []string{"x"}}}, wantErr: false}, + {name: "valid dsl allowed", matchers: []Matcher{{Type: "dsl", DSL: []string{"status_code == 200"}}}, wantErr: false}, + {name: "bad dsl syntax rejected", matchers: []Matcher{{Type: "dsl", DSL: []string{"status_code =="}}}, wantErr: true}, + {name: "one bad dsl among good rejected", matchers: []Matcher{{Type: "dsl", DSL: []string{"status_code == 200", "((("}}}, wantErr: true}, + {name: "non-allowlisted dsl helper rejected", matchers: []Matcher{{Type: "dsl", DSL: []string{"wait_for(1)"}}}, wantErr: true}, + {name: "empty dsl rejected", matchers: []Matcher{{Type: "dsl"}}, wantErr: true}, + {name: "only-empty dsl expression rejected", matchers: []Matcher{{Type: "dsl", DSL: []string{""}}}, wantErr: true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -212,7 +218,7 @@ func TestCheckMatcherFaviconNegative(t *testing.T) { signed := int64(fingerprint.FaviconHash(faviconFixture)) matchers := []Matcher{{Type: "favicon", Hash: []int64{signed}, Negative: true}} resp := fakeResponse(t, 200, nil) - if checkMatchers(matchers, "", resp, string(faviconFixture)) { + if checkMatchers(matchers, "", &MatchContext{Resp: resp, Body: string(faviconFixture)}) { t.Error("negative favicon matcher should not match its own hash") } } diff --git a/internal/modules/fuzz_test.go b/internal/modules/fuzz_test.go new file mode 100644 index 00000000..db20cdf0 --- /dev/null +++ b/internal/modules/fuzz_test.go @@ -0,0 +1,526 @@ +/* +·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· +: : +: █▀ █ █▀▀ · Blazing-fast pentesting suite : +: ▄█ █ █▀ · BSD 3-Clause License : +: : +: (c) 2022-2026 vmfunc, xyzeva, : +: lunchcat alumni & contributors : +: : +·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· +*/ + +package modules + +import ( + "context" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "reflect" + "runtime" + "sort" + "sync" + "sync/atomic" + "testing" + "time" + + "gopkg.in/yaml.v3" + + "github.com/vmfunc/sif/internal/httpx" +) + +// legacyPayloads builds the single anonymous set the sequence form desugars to, +// so existing tests that used a flat []string keep their exact meaning. +func legacyPayloads(vals []string) PayloadSets { + if len(vals) == 0 { + return PayloadSets{} + } + return PayloadSets{Sets: []PayloadSet{{Name: "payload", Values: vals}}} +} + +func TestPayloadSetsUnmarshalSequence(t *testing.T) { + var cfg HTTPConfig + if err := yaml.Unmarshal([]byte("payloads: [\"a\", \"b\"]\n"), &cfg); err != nil { + t.Fatalf("unmarshal: %v", err) + } + want := []PayloadSet{{Name: "payload", Values: []string{"a", "b"}}} + if !reflect.DeepEqual(cfg.Payloads.Sets, want) { + t.Errorf("sequence form = %+v, want %+v", cfg.Payloads.Sets, want) + } +} + +func TestPayloadSetsUnmarshalMappingOrdered(t *testing.T) { + // both keys are inline here (not file-backed): a file-backed entry is + // rejected by validate() in this task (see TestPayloadSetsValidation), so + // mixing one in would make this parse fail rather than exercise ordering. + var cfg HTTPConfig + src := "payloads:\n user: [\"admin\", \"root\"]\n role: [\"admin\", \"user\"]\n" + if err := yaml.Unmarshal([]byte(src), &cfg); err != nil { + t.Fatalf("unmarshal: %v", err) + } + want := []PayloadSet{ + {Name: "user", Values: []string{"admin", "root"}}, + {Name: "role", Values: []string{"admin", "user"}}, + } + if !reflect.DeepEqual(cfg.Payloads.Sets, want) { + t.Errorf("mapping form = %+v, want %+v", cfg.Payloads.Sets, want) + } +} + +func TestPayloadSetsValidation(t *testing.T) { + // duplicate set name + if err := yaml.Unmarshal([]byte("payloads:\n x: [\"1\"]\n x: [\"2\"]\n"), &HTTPConfig{}); err == nil { + t.Error("duplicate set name accepted") + } + // reserved builtin name + if err := yaml.Unmarshal([]byte("payloads:\n BaseURL: [\"1\"]\n"), &HTTPConfig{}); err == nil { + t.Error("BaseURL set name accepted") + } + // file-backed set now parses (loaded at resolve time) + if err := yaml.Unmarshal([]byte("payloads:\n p: creds.txt\n"), &HTTPConfig{}); err != nil { + t.Errorf("file-backed set rejected: %v", err) + } +} + +func TestResolveSetsLoadsFile(t *testing.T) { + dir := t.TempDir() + wl := filepath.Join(dir, "creds.txt") + if err := os.WriteFile(wl, []byte("admin\nroot\n\nguest\n"), 0o600); err != nil { + t.Fatal(err) + } + cfg := &HTTPConfig{ + Paths: []string{"{{BaseURL}}/?u={{user}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "user", File: wl}, + }}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + // loadWordlist skips the blank line, leaving 3 words. + want := []string{"http://t/?u=admin", "http://t/?u=guest", "http://t/?u=root"} + urls := reqURLs(got) + if !reflect.DeepEqual(urls, want) { + t.Errorf("file-set urls = %v, want %v", urls, want) + } +} + +func TestResolveSetsMissingFile(t *testing.T) { + cfg := &HTTPConfig{ + Paths: []string{"{{BaseURL}}/?u={{user}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{{Name: "user", File: "/no/such/wordlist"}}}, + } + if _, err := generateHTTPRequests("http://t", cfg); err == nil { + t.Error("missing wordlist accepted") + } +} + +func TestStreamRequestsMultiSetClusterbomb(t *testing.T) { + cfg := &HTTPConfig{ + Paths: []string{"{{BaseURL}}/x?u={{user}}&p={{pass}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "user", Values: []string{"a", "b"}}, + {Name: "pass", Values: []string{"1", "2"}}, + }}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + // paths outer, sets in declaration order, rightmost (pass) fastest. + want := []string{ + "http://t/x?u=a&p=1", "http://t/x?u=a&p=2", + "http://t/x?u=b&p=1", "http://t/x?u=b&p=2", + } + urls := reqURLs(got) // reqURLs sorts; sort want too + sort.Strings(want) + if !reflect.DeepEqual(urls, want) { + t.Errorf("clusterbomb urls = %v, want %v", urls, want) + } +} + +func TestStreamRequestsMultiSetPitchfork(t *testing.T) { + cfg := &HTTPConfig{ + Attack: "pitchfork", + Paths: []string{"{{BaseURL}}/a?u={{user}}&p={{pass}}", "{{BaseURL}}/b?u={{user}}&p={{pass}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "user", Values: []string{"a", "b", "c"}}, + {Name: "pass", Values: []string{"1", "2"}}, + }}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + // zip paths(2) x user(3) x pass(2) -> stop at 2 + want := []string{"http://t/a?u=a&p=1", "http://t/b?u=b&p=2"} + urls := reqURLs(got) + sort.Strings(want) + if !reflect.DeepEqual(urls, want) { + t.Errorf("pitchfork urls = %v, want %v", urls, want) + } +} + +func TestStreamRequestsBatteringram(t *testing.T) { + cfg := &HTTPConfig{ + Attack: "batteringram", + Paths: []string{"{{BaseURL}}/x?u={{user}}&p={{pass}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "user", Values: []string{"a", "b", "c"}}, + {Name: "pass", Values: []string{"1", "2", "3"}}, + }}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + // every position gets the same value per iteration, from the first set, + // and it iterates the full set length rather than crossing user x pass. + want := []string{ + "http://t/x?u=a&p=a", "http://t/x?u=b&p=b", "http://t/x?u=c&p=c", + } + urls := reqURLs(got) + sort.Strings(want) + if !reflect.DeepEqual(urls, want) { + t.Errorf("batteringram urls = %v, want %v", urls, want) + } + if len(got) != 3 { + t.Errorf("batteringram sent %d requests, want 3 (set length, not a cross-product)", len(got)) + } +} + +func TestStreamRequestsBatteringramCrossesPaths(t *testing.T) { + cfg := &HTTPConfig{ + Attack: "batteringram", + Paths: []string{"{{BaseURL}}/a?p={{payload}}", "{{BaseURL}}/b?p={{payload}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "payload", Values: []string{"1", "2"}}, + }}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + want := []string{ + "http://t/a?p=1", "http://t/a?p=2", "http://t/b?p=1", "http://t/b?p=2", + } + urls := reqURLs(got) + sort.Strings(want) + if !reflect.DeepEqual(urls, want) { + t.Errorf("batteringram urls = %v, want %v", urls, want) + } +} + +func TestStreamRequestsBatteringramEmptySet(t *testing.T) { + cfg := &HTTPConfig{ + Attack: "batteringram", + Paths: []string{"{{BaseURL}}/"}, + Payloads: PayloadSets{Sets: []PayloadSet{{Name: "payload", Values: nil}}}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + if len(got) != 0 { + t.Errorf("empty set sent %d requests, want 0", len(got)) + } +} + +func TestStreamRequestsHeaderSubstitution(t *testing.T) { + cfg := &HTTPConfig{ + Paths: []string{"{{BaseURL}}/"}, + Headers: map[string]string{"X-Token": "t-{{payload}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "payload", Values: []string{"abc"}}, + }}, + } + got, err := generateHTTPRequests("http://t", cfg) + if err != nil { + t.Fatalf("generate: %v", err) + } + if len(got) != 1 || got[0].Headers["X-Token"] != "t-abc" { + t.Errorf("header not substituted: %+v", got[0].Headers) + } +} + +func TestExecuteHTTPModuleBudgetTruncates(t *testing.T) { + var hits int64 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt64(&hits, 1) + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + // 10 x 10 = 100 combinations, capped at 15. + users := make([]string, 10) + passes := make([]string, 10) + for i := range users { + users[i] = string(rune('a' + i)) + passes[i] = string(rune('0' + i)) + } + def := &YAMLModule{ + ID: "fz", + Type: TypeHTTP, + HTTP: &HTTPConfig{ + Paths: []string{"{{BaseURL}}/?u={{user}}&p={{pass}}"}, + Payloads: PayloadSets{Sets: []PayloadSet{ + {Name: "user", Values: users}, + {Name: "pass", Values: passes}, + }}, + Matchers: []Matcher{{Type: "word", Part: "body", Words: []string{"ok"}}}, + }, + } + opts := Options{Timeout: testTimeout, Client: httpx.Client(testTimeout), FuzzMaxRequests: 15} + if _, err := ExecuteHTTPModule(context.Background(), srv.URL, def, opts); err != nil { + t.Fatalf("ExecuteHTTPModule: %v", err) + } + if got := atomic.LoadInt64(&hits); got != 15 { + t.Errorf("sent %d requests, want 15 (budget cap)", got) + } +} + +func TestExecuteHTTPModuleBudgetUnlimited(t *testing.T) { + var hits int64 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt64(&hits, 1) + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + def := &YAMLModule{ + ID: "fz", + Type: TypeHTTP, + HTTP: &HTTPConfig{ + Paths: []string{"{{BaseURL}}/?p={{payload}}"}, + Payloads: legacyPayloads([]string{"1", "2", "3", "4", "5"}), + Matchers: []Matcher{{Type: "word", Part: "body", Words: []string{"ok"}}}, + }, + } + opts := Options{Timeout: testTimeout, Client: httpx.Client(testTimeout), FuzzMaxRequests: 0} + if _, err := ExecuteHTTPModule(context.Background(), srv.URL, def, opts); err != nil { + t.Fatalf("ExecuteHTTPModule: %v", err) + } + if got := atomic.LoadInt64(&hits); got != 5 { + t.Errorf("sent %d requests, want 5 (0 = unlimited)", got) + } +} + +// TestExecuteHTTPModuleCancelMidStream is a regression guard for the fan-out +// pool's two send-selects (producer -> reqCh, worker -> resultCh in +// ExecuteHTTPModule). TestExecuteHTTPModuleContextCancel only ever passes an +// already-cancelled context, so the producer returns at its up-front ctx.Err() +// check and never reaches `select { case <-ctx.Done(): case reqCh <- req: }`. +// Here the context is cancelled *while* requests are in flight: the server +// blocks on the request's context so workers sit parked mid-request, the +// producer keeps trying to push a large payload set through an unbuffered +// reqCh with only 2 workers draining it, and cancel has to land on the +// producer's send-select (not just the worker's). If either escape-hatch were +// ever removed, this test would hang and fail via the time.After branch below +// - there is no red phase, since the production code is already correct. +func TestExecuteHTTPModuleCancelMidStream(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-time.After(3 * time.Second): + } + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + payloads := make([]string, 200) + for i := range payloads { + payloads[i] = string(rune('a'+(i%26))) + string(rune('0'+(i%10))) + } + def := &YAMLModule{ + ID: "fz-cancel-mid", + Type: TypeHTTP, + HTTP: &HTTPConfig{ + Paths: []string{"{{BaseURL}}/?p={{payload}}"}, + Payloads: legacyPayloads(payloads), + Matchers: []Matcher{{Type: "word", Part: "body", Words: []string{"ok"}}}, + }, + } + opts := Options{Timeout: testTimeout, Client: httpx.Client(testTimeout), Threads: 2, FuzzMaxRequests: 0} + + before := runtime.NumGoroutine() + + ctx, cancel := context.WithCancel(context.Background()) + go func() { + time.Sleep(75 * time.Millisecond) + cancel() + }() + + done := make(chan struct{}) + go func() { + defer close(done) + if _, err := ExecuteHTTPModule(ctx, srv.URL, def, opts); err != nil { + t.Errorf("ExecuteHTTPModule: %v", err) + } + }() + + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("ExecuteHTTPModule did not return promptly after mid-stream cancel") + } + + // poll for goroutines to drain rather than a single fixed sleep, so a + // slow teardown on a loaded runner does not read as a leak. + deadline := time.Now().Add(2 * time.Second) + for { + runtime.GC() + after := runtime.NumGoroutine() + if after <= before+2 { + break + } + if time.Now().After(deadline) { + t.Errorf("possible goroutine leak: before=%d after=%d", before, after) + break + } + time.Sleep(20 * time.Millisecond) + } +} + +// fuzzBudgetModule builds a distinct HTTP fuzz module with its own payload set. +func fuzzBudgetModule(id string, n int) *YAMLModule { + vals := make([]string, n) + for i := range vals { + vals[i] = string(rune('a'+(i%26))) + string(rune('0'+(i%10))) + string(rune('A'+(i%26))) + } + return &YAMLModule{ + ID: id, + Type: TypeHTTP, + HTTP: &HTTPConfig{ + Paths: []string{"{{BaseURL}}/?p={{payload}}"}, + Payloads: legacyPayloads(vals), + Matchers: []Matcher{{Type: "word", Part: "body", Words: []string{"ok"}}}, + }, + } +} + +// two modules with no per-module cap, run concurrently, must together send +// exactly the global budget's worth of requests, not per module. +func TestFuzzBudgetCapsAcrossConcurrentModules(t *testing.T) { + var hits int64 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt64(&hits, 1) + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + budget := NewFuzzBudget(30) + opts := Options{ + Timeout: testTimeout, + Client: httpx.Client(testTimeout), + FuzzMaxRequests: 0, // unlimited per module: the global budget is the only cap in play + FuzzGlobalBudget: budget, + } + + modA := fuzzBudgetModule("fz-a", 100) + modB := fuzzBudgetModule("fz-b", 100) + + var wg sync.WaitGroup + wg.Add(2) + for _, m := range []*YAMLModule{modA, modB} { + m := m + go func() { + defer wg.Done() + if _, err := ExecuteHTTPModule(context.Background(), srv.URL, m, opts); err != nil { + t.Errorf("ExecuteHTTPModule(%s): %v", m.ID, err) + } + }() + } + wg.Wait() + + if got := atomic.LoadInt64(&hits); got != 30 { + t.Errorf("sent %d requests across both modules, want 30 (shared global budget)", got) + } +} + +// a budget exhausted mid-stream, shared by several concurrent modules, must +// not deadlock; every ExecuteHTTPModule call returns promptly. +func TestFuzzBudgetExhaustionNoDeadlock(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + budget := NewFuzzBudget(5) + opts := Options{ + Timeout: testTimeout, + Client: httpx.Client(testTimeout), + Threads: 4, + FuzzGlobalBudget: budget, + } + + const numModules = 6 + mods := make([]*YAMLModule, numModules) + for i := range mods { + mods[i] = fuzzBudgetModule(string(rune('A'+i)), 50) + } + + done := make(chan struct{}) + go func() { + defer close(done) + var wg sync.WaitGroup + wg.Add(numModules) + for _, m := range mods { + m := m + go func() { + defer wg.Done() + if _, err := ExecuteHTTPModule(context.Background(), srv.URL, m, opts); err != nil { + t.Errorf("ExecuteHTTPModule(%s): %v", m.ID, err) + } + }() + } + wg.Wait() + }() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("modules did not return after global budget exhaustion; possible deadlock") + } +} + +// a caller that never sets Options.FuzzGlobalBudget keeps seeing unlimited. +func TestFuzzBudgetNilIsUnlimited(t *testing.T) { + var b *FuzzBudget + for i := 0; i < 1000; i++ { + if !b.Reserve() { + t.Fatalf("nil budget refused reservation %d, want always true", i) + } + } +} + +// TestNewFuzzBudgetUnlimited pins the nil-means-unlimited contract from both +// ends: the constructor returns nil for a non-positive cap, and a nil budget +// keeps reserving forever. A nil check missing from Reserve would panic on +// every scan run with the flag set to 0. +func TestNewFuzzBudgetUnlimited(t *testing.T) { + for _, max := range []int{0, -1, -1000} { + if b := NewFuzzBudget(max); b != nil { + t.Errorf("NewFuzzBudget(%d) = %+v, want nil (unlimited)", max, b) + } + } + + var unlimited *FuzzBudget + for i := 0; i < 1000; i++ { + if !unlimited.Reserve() { + t.Fatalf("nil budget refused a reservation at attempt %d", i+1) + } + } + + limited := NewFuzzBudget(2) + for i := 1; i <= 2; i++ { + if !limited.Reserve() { + t.Fatalf("budget of 2 refused reservation %d", i) + } + } + if limited.Reserve() { + t.Error("budget of 2 allowed a third reservation") + } +} diff --git a/internal/modules/matchers_condition_test.go b/internal/modules/matchers_condition_test.go index 31a63229..131eac6d 100644 --- a/internal/modules/matchers_condition_test.go +++ b/internal/modules/matchers_condition_test.go @@ -53,7 +53,7 @@ func TestCheckMatchersCondition(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - if got := checkMatchers(tt.matchers, tt.condition, resp, body); got != tt.expect { + if got := checkMatchers(tt.matchers, tt.condition, &MatchContext{Resp: resp, Body: body}); got != tt.expect { t.Errorf("checkMatchers(%q) = %v, want %v", tt.condition, got, tt.expect) } }) diff --git a/internal/modules/matchers_range_test.go b/internal/modules/matchers_range_test.go index f270794f..b2335a24 100644 --- a/internal/modules/matchers_range_test.go +++ b/internal/modules/matchers_range_test.go @@ -51,7 +51,7 @@ func TestCheckMatcherRange(t *testing.T) { t.Run("status source in range", func(t *testing.T) { resp := fakeResponse(t, 503, nil) m := &Matcher{Type: "range", Source: "status", Min: intp(500), Max: intp(599)} - if !checkMatcher(m, resp, "") { + if !checkMatcher(m, &MatchContext{Resp: resp}) { t.Error("expected 503 to be within 500-599") } }) @@ -59,7 +59,7 @@ func TestCheckMatcherRange(t *testing.T) { t.Run("status source out of range", func(t *testing.T) { resp := fakeResponse(t, 200, nil) m := &Matcher{Type: "range", Source: "status", Min: intp(500), Max: intp(599)} - if checkMatcher(m, resp, "") { + if checkMatcher(m, &MatchContext{Resp: resp}) { t.Error("expected 200 to be outside 500-599") } }) @@ -67,7 +67,7 @@ func TestCheckMatcherRange(t *testing.T) { t.Run("size source default", func(t *testing.T) { resp := fakeResponse(t, 200, nil) m := &Matcher{Type: "range", Min: intp(5), Max: intp(20)} - if !checkMatcher(m, resp, "twelve chars") { + if !checkMatcher(m, &MatchContext{Resp: resp, Body: "twelve chars"}) { t.Error("expected body length within bounds to match") } }) @@ -75,7 +75,7 @@ func TestCheckMatcherRange(t *testing.T) { t.Run("size source explicit", func(t *testing.T) { resp := fakeResponse(t, 200, nil) m := &Matcher{Type: "range", Source: "size", Min: intp(1000)} - if checkMatcher(m, resp, "short") { + if checkMatcher(m, &MatchContext{Resp: resp, Body: "short"}) { t.Error("expected short body to miss a high min bound") } }) diff --git a/internal/modules/matchers_test.go b/internal/modules/matchers_test.go index 5217d3f6..7a318ea3 100644 --- a/internal/modules/matchers_test.go +++ b/internal/modules/matchers_test.go @@ -50,7 +50,7 @@ func TestCheckMatcherStatus(t *testing.T) { t.Run(tt.name, func(t *testing.T) { m := &Matcher{Type: "status", Status: tt.want} resp := fakeResponse(t, tt.status, nil) - if got := checkMatcher(m, resp, ""); got != tt.expect { + if got := checkMatcher(m, &MatchContext{Resp: resp, Body: ""}); got != tt.expect { t.Errorf("checkMatcher status = %v, want %v", got, tt.expect) } }) @@ -77,7 +77,7 @@ func TestCheckMatcherWord(t *testing.T) { t.Run(tt.name, func(t *testing.T) { m := &Matcher{Type: "word", Part: "body", Words: tt.words, Condition: tt.condition} resp := fakeResponse(t, 200, nil) - if got := checkMatcher(m, resp, body); got != tt.expect { + if got := checkMatcher(m, &MatchContext{Resp: resp, Body: body}); got != tt.expect { t.Errorf("checkMatcher word = %v, want %v", got, tt.expect) } }) @@ -107,7 +107,7 @@ func TestCheckMatcherRegex(t *testing.T) { t.Run(tt.name, func(t *testing.T) { m := &Matcher{Type: "regex", Part: "body", Regex: tt.patterns, Condition: tt.condition} resp := fakeResponse(t, 200, nil) - if got := checkMatcher(m, resp, body); got != tt.expect { + if got := checkMatcher(m, &MatchContext{Resp: resp, Body: body}); got != tt.expect { t.Errorf("checkMatcher regex = %v, want %v", got, tt.expect) } }) @@ -119,13 +119,13 @@ func TestCheckMatcherHeaderPart(t *testing.T) { resp := fakeResponse(t, 200, header) m := &Matcher{Type: "word", Part: "header", Words: []string{"PHP/8.1"}} - if !checkMatcher(m, resp, "body-content") { + if !checkMatcher(m, &MatchContext{Resp: resp, Body: "body-content"}) { t.Error("expected header-part word matcher to hit on header value") } // the same word lives only in the header, so a body-part matcher must miss. mBody := &Matcher{Type: "word", Part: "body", Words: []string{"PHP/8.1"}} - if checkMatcher(mBody, resp, "body-content") { + if checkMatcher(mBody, &MatchContext{Resp: resp, Body: "body-content"}) { t.Error("body-part matcher should not see header-only value") } } @@ -133,7 +133,7 @@ func TestCheckMatcherHeaderPart(t *testing.T) { func TestCheckMatcherUnknownType(t *testing.T) { m := &Matcher{Type: "size", Part: "body"} resp := fakeResponse(t, 200, nil) - if checkMatcher(m, resp, "anything") { + if checkMatcher(m, &MatchContext{Resp: resp, Body: "anything"}) { t.Error("unknown matcher type should not match") } } @@ -186,7 +186,7 @@ func TestCheckMatchers(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - if got := checkMatchers(tt.matchers, "", resp, body); got != tt.expect { + if got := checkMatchers(tt.matchers, "", &MatchContext{Resp: resp, Body: body}); got != tt.expect { t.Errorf("checkMatchers = %v, want %v", got, tt.expect) } }) @@ -250,6 +250,22 @@ func TestCheckRegex(t *testing.T) { } } +// a dsl matcher is compiled at load, so an uncompilable expression must fail the +// whole module rather than silently never matching at scan time. +func TestParseYAMLModuleDSLMatcher(t *testing.T) { + dir := t.TempDir() + write := func(name, body string) string { return writeModule(t, dir, name, body) } + + badDSL := write("bad-matcher-dsl.yaml", "id: bmd\ntype: http\nhttp:\n paths: [\"/\"]\n matchers:\n - type: dsl\n dsl: [\"status_code ==\"]\n") + if _, err := ParseYAMLModule(badDSL); err == nil { + t.Fatal("uncompilable dsl matcher expression accepted") + } + goodDSL := write("good-matcher-dsl.yaml", "id: gmd\ntype: http\nhttp:\n paths: [\"/\"]\n matchers:\n - type: dsl\n dsl: [\"status_code == 200\"]\n") + if _, err := ParseYAMLModule(goodDSL); err != nil { + t.Fatalf("valid dsl matcher rejected: %v", err) + } +} + func TestGetPart(t *testing.T) { header := http.Header{"Server": []string{"nginx"}} resp := fakeResponse(t, 200, header) @@ -447,7 +463,7 @@ func TestGenerateHTTPRequests(t *testing.T) { cfg := &HTTPConfig{ Method: "POST", Paths: []string{"{{BaseURL}}/q?x={{payload}}"}, - Payloads: []string{"1", "2", "3"}, + Payloads: legacyPayloads([]string{"1", "2", "3"}), Body: "data={{payload}}", } got, err := generateHTTPRequests("http://h", cfg) @@ -476,7 +492,7 @@ func TestGenerateHTTPRequests(t *testing.T) { t.Run("multiple paths times multiple payloads", func(t *testing.T) { cfg := &HTTPConfig{ Paths: []string{"{{BaseURL}}/a", "{{BaseURL}}/b"}, - Payloads: []string{"x", "y"}, + Payloads: legacyPayloads([]string{"x", "y"}), } got, err := generateHTTPRequests("http://h", cfg) if err != nil { @@ -521,7 +537,7 @@ func TestGenerateHTTPRequests(t *testing.T) { cfg := &HTTPConfig{ Paths: []string{"{{BaseURL}}/{{word}}?q={{payload}}"}, Wordlist: list, - Payloads: []string{"1", "2", "3"}, + Payloads: legacyPayloads([]string{"1", "2", "3"}), } got, err := generateHTTPRequests("http://h", cfg) if err != nil { diff --git a/internal/modules/module.go b/internal/modules/module.go index 29d4d3d3..546be237 100644 --- a/internal/modules/module.go +++ b/internal/modules/module.go @@ -57,10 +57,14 @@ type Info struct { // Options for module execution. type Options struct { - Timeout time.Duration - Threads int - LogDir string - Client *http.Client + Timeout time.Duration + Threads int + LogDir string + FuzzMaxRequests int + // FuzzGlobalBudget is a scan-wide request cap shared across every + // module/target; nil means unlimited. + FuzzGlobalBudget *FuzzBudget + Client *http.Client } // Result from module execution. @@ -105,6 +109,23 @@ type Matcher struct { Max *int `yaml:"max,omitempty"` // CaseInsensitive folds word matching to lower-case when set (word matcher only). CaseInsensitive bool `yaml:"case-insensitive,omitempty"` + + // DSL holds one or more boolean expressions evaluated against the response + // (dsl matchers only). Compiled and validated at module load. + DSL []string `yaml:"dsl,omitempty"` +} + +// MatchContext carries everything a matcher can evaluate against one response, +// so a new matcher type can read a field the classic ones ignore without +// changing the signature of the whole engine again. +type MatchContext struct { + Resp *http.Response + Body string + URL string + Duration time.Duration + // Extracted is the running variable set in a request chain, so it also + // carries earlier steps' values. + Extracted map[string]string } // Extractor defines data extraction from responses. diff --git a/internal/modules/yaml.go b/internal/modules/yaml.go index ab34489c..1cba98c8 100644 --- a/internal/modules/yaml.go +++ b/internal/modules/yaml.go @@ -56,10 +56,10 @@ type HTTPConfig struct { Method string `yaml:"method"` Paths []string `yaml:"paths"` Wordlist string `yaml:"wordlist,omitempty"` - Payloads []string `yaml:"payloads,omitempty"` + Payloads PayloadSets `yaml:"payloads,omitempty"` Headers map[string]string `yaml:"headers,omitempty"` Body string `yaml:"body,omitempty"` - Attack string `yaml:"attack,omitempty"` // clusterbomb (default), pitchfork + Attack string `yaml:"attack,omitempty"` // clusterbomb (default), pitchfork, batteringram Threads int `yaml:"threads,omitempty"` DisableRedirects bool `yaml:"disable-redirects,omitempty"` // stop at the first response; don't follow 3xx Matchers []Matcher `yaml:"matchers"` @@ -68,6 +68,79 @@ type HTTPConfig struct { Requests []HTTPStep `yaml:"requests,omitempty"` // ordered request chain; see HTTPStep } +// PayloadSet is one named fuzzing position. Values holds the inline list; when +// the YAML value was a scalar string, File holds that path and Values stays nil +// until resolved at execution time. +type PayloadSet struct { + Name string + Values []string + File string +} + +// PayloadSets is the parsed `payloads:` field: an ordered list of named sets. +// A YAML sequence desugars to one set named "payload" (so {{payload}} keeps +// working); a YAML mapping becomes one set per key in declaration order. +type PayloadSets struct { + Sets []PayloadSet +} + +// UnmarshalYAML accepts the two payloads shapes. A sequence is the legacy single +// anonymous list. A mapping is named sets, order-preserved via the raw node +// (a Go map would scramble order, which clusterbomb/pitchfork depend on). A +// key whose value is a scalar is a file-backed set; a sequence value is inline. +func (p *PayloadSets) UnmarshalYAML(value *yaml.Node) error { + switch value.Kind { + case yaml.SequenceNode: + var vals []string + if err := value.Decode(&vals); err != nil { + return fmt.Errorf("payloads: %w", err) + } + if len(vals) > 0 { + p.Sets = []PayloadSet{{Name: "payload", Values: vals}} + } + case yaml.MappingNode: + sets := make([]PayloadSet, 0, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + name := value.Content[i].Value + v := value.Content[i+1] + set := PayloadSet{Name: name} + switch v.Kind { + case yaml.ScalarNode: + set.File = v.Value + case yaml.SequenceNode: + if err := v.Decode(&set.Values); err != nil { + return fmt.Errorf("payloads[%s]: %w", name, err) + } + default: + return fmt.Errorf("payloads[%s]: want a list or a file path", name) + } + sets = append(sets, set) + } + p.Sets = sets + default: + return fmt.Errorf("payloads: want a list or a map of named lists") + } + return p.validate() +} + +// validate rejects ambiguous or reserved set names. A file-backed set (File +// non-empty) parses cleanly here; resolveSets is where a missing or unreadable +// wordlist actually fails. +func (p PayloadSets) validate() error { + seen := make(map[string]struct{}, len(p.Sets)) + for _, s := range p.Sets { + switch s.Name { + case "BaseURL", "baseurl": + return fmt.Errorf("payloads: set name %q collides with the base-url builtin", s.Name) + } + if _, dup := seen[s.Name]; dup { + return fmt.Errorf("payloads: duplicate set name %q", s.Name) + } + seen[s.Name] = struct{}{} + } + return nil +} + // HTTPStep is one request in a chain. steps run in order and share a variable // map: each step's extractors populate {{name}} references usable in the path, // headers and body of later steps. a step whose matchers don't match halts the diff --git a/sif.go b/sif.go index 481e4fb2..c76b926c 100644 --- a/sif.go +++ b/sif.go @@ -45,9 +45,10 @@ import ( // App represents the main application structure for sif. // It encapsulates the configuration settings, target URLs, and logging information. type App struct { - settings *config.Settings - targets []string - logFiles []string + settings *config.Settings + targets []string + logFiles []string + fuzzBudget *modules.FuzzBudget // scan-wide fuzz request cap, shared read-only across every concurrent scanTarget call } // Version is set by main to the resolved build version and shown on the banner. @@ -87,7 +88,7 @@ func NewModuleResult[T ScanResult](data T) ModuleResult { // // Errors if no targets are supplied through URLs or File. func New(settings *config.Settings) (*App, error) { - app := &App{settings: settings} + app := &App{settings: settings, fuzzBudget: modules.NewFuzzBudget(settings.FuzzGlobalMaxRequests)} // -silent reroutes all chrome to stderr (and suppresses spinners) before the // banner prints, so stdout carries nothing but findings even on the banner. @@ -762,10 +763,12 @@ func (app *App) scanTarget(url, storeDir string, wantReport bool) (targetScan, e // -proxy/-H/-cookie/-rate-limit apply to module scans the same as every // other scanner instead of each module dialing out on a bare client. opts := modules.Options{ - Timeout: app.settings.Timeout, - Threads: app.settings.Threads, - LogDir: app.settings.LogDir, - Client: httpx.Client(app.settings.Timeout), + Timeout: app.settings.Timeout, + Threads: app.settings.Threads, + LogDir: app.settings.LogDir, + Client: httpx.Client(app.settings.Timeout), + FuzzMaxRequests: app.settings.FuzzMaxRequests, + FuzzGlobalBudget: app.fuzzBudget, } for _, m := range toRun {