Add Rapidfire vulnerability scanner parser #11909

skywalke34 · 2025-02-27T05:10:26Z

Description
Rapidfire Scan Parser

Rapidfire CSV Parser

CSV Field Mappings

CSV Field	Finding Field	Parser Line #	Notes
IP Address	endpoints[].host	162-173	Used if hostname not available
Hostname	endpoints[].host	162-173	Primary choice for endpoint host
MAC Address	description	134-136	Added to description with "MAC Address:" prefix
Severity	severity	149	Capitalized and validated against SEVERITIES, defaults to Info
Issue	title	107-110	Direct mapping, stripped of whitespace
Ports	endpoints[].port	165-166	Extracted number before "/" using regex
OID	vuln_id_from_tool	152	Direct mapping
CVE	unsaved_vulnerability_ids	176-177	Split on comma, filtered to valid CVE IDs
Last Detected	date	151	Parsed to datetime using dateutil.parser
Known Exploited Vulnerability	description	131-132	Added to description with prefix
Summary	description	117-118	Added to description with "Summary:" prefix
Vulnerability Detection Result	description	119-120	Added to description with prefix
Solution	mitigation	150	Direct mapping
Vulnerability Insight	impact	82-103	Formatted with CVEs into impact field
Vulnerability Detection Method	description	121-122	Added to description with prefix
References	references	70-124	Formatted into markdown list of links
Known To Be Used In Ransomware Campaigns	description, tags	137-138, 179-180	Adds warning to description and "ransomware" tag

Additional Finding Field Settings

Finding Field	Value	Parser Line #	Notes
test	test parameter	153	Set from test parameter passed to get_findings
dynamic_finding	True	153	Hardcoded to True for all findings
static_finding	False	154	Hardcoded to False for all findings

Processing Notes

Deduplication is performed using combination of title, IP address, hostname and port
For duplicate findings, the existing finding is updated rather than creating a new one
The parser uses csv.DictReader with comma delimiter and quote character
Empty rows are skipped
References are formatted into a readable markdown list with descriptive link text
Impact field is specially formatted to combine vulnerability insight and CVE details
Date parsing handles various formats and falls back to current time
Port extraction handles various formats like "8080/tcp" or "443/tcp (https)"

Test results

13 unit tests successfully executed against example .csv files in unittests/scans/rapidfire directory:

dryrunsecurity · 2025-02-27T05:11:14Z

DryRun Security Summary

The pull request integrates Rapidfire vulnerability scanner support into DefectDojo, adding documentation, configuration, and parser capabilities while revealing several potential security findings in the process.

Expand for full summary

The pull request adds documentation, configuration, and parser support for the Rapidfire vulnerability scanner in DefectDojo, including unit tests and sample scan data.

Security findings include:

External URL exposure (Rapidfire vendor website)
Potential information leakage through detailed CSV field mappings
Sensitive network information exposure in test scan files (internal IP addresses, hostnames, MAC addresses)
Multiple Apache Tomcat vulnerabilities discovered in test scan data
Detailed vulnerability information that could aid potential attackers
Exposure of internal network infrastructure details through test files
Potential reconnaissance information from hostname and port disclosures

View PR in the DryRun Dashboard.

…est scan examples

Failed unit tests looking for ###Sample Scan Data and File Types.

mtesauro

Approved

dogboat

Great work! Just a few questions, mostly related to the format of the scan.

dogboat · 2025-03-18T17:36:14Z

docs/content/en/connecting_your_tools/parsers/file/rapidfire.md

+There are samples for:
+* `one_vuln.csv` - A file containing a single vulnerability finding
+* `many_vulns.csv` - A file containing multiple vulnerability findings
+* `no_vuln.csv` - An empty scan with no vulnerabilities


Don't leave out invalid_date.csv and complex_ports.csv! Those are awesome too!

I may have created more issues by creating the invalid_date.csv and complex_ports.csv early when encountering errors. I have access an original rapidfire .csv output with more than 2 examples, so I will review the accuracy of these unit test .csv files...

dogboat · 2025-03-18T17:42:29Z

unittests/tools/test_rapidfire_parser.py

+class TestRapidFireParser(TestCase):
+    def test_parse_no_findings(self):
+        """Test parsing a RapidFire report with no findings"""
+        with open("unittests/scans/rapidfire/no_vuln.csv", encoding="utf-8") as testfile:


There was a recently-added util method, unittests.dojo_test_case.get_unit_tests_scans_path, that can be used with these open() calls instead of the full path; its use looks like e.g.

with open(get_unit_tests_scans_path("rapidfire") / "no_vuln.csv", encoding="utf-8"):

Any chance you could update the tests to use this?

Yes, absolutely! :)

dogboat · 2025-03-18T18:07:20Z

unittests/scans/rapidfire/invalid_date.csv

@@ -0,0 +1,3 @@
+IP Address,Hostname,MAC Address,Severity,Issue,Ports,OID,CVE,Last Detected,Known Exploited Vulnerability,Summary,Vulnerability Detection Result,Solution,Vulnerability Insight,Vulnerability Detection Method,References,Known to Be Used in Ransomware Campaigns
+192.168.1.1,webserver.local,00:11:22:33:44:55,High,SQL Injection Vulnerability,80,OID123,CVE-2023-1234,05/15/2023,Yes,Critical SQL Injection found,Successful SQL Injection,Update database access layer,Allows unauthorized data access,Penetration testing,https://example.com/sql-injection,Yes
+10.0.0.1,appserver.local,AA:BB:CC:DD:EE:FF,Medium,Cross-Site Scripting (XSS),443,OID456,CVE-2023-5678,06/20/2023,No,XSS vulnerability in login form,Reflected XSS detected,Implement input sanitization,Allows script injection,Automated scanning,https://example.com/xss,No


Sorry if I'm missing something -- are these actually invalid dates? "05/15/2023" and "06/20/2023" both seem to parse correctly... This test has value, I just want to make sure I'm understanding it properly.

No, you are not missing anything! (Thank you!). An early issue I was encountering was different date formats 31-Oct-24 vs 2024-03-31, but after closer review, I now I believe I can simplify with parsing only the 31-Oct-24 format. Going to look deeper at this...

dogboat · 2025-03-18T18:22:06Z

dojo/tools/rapidfire/parser.py

+            return None
+
+        # Regular expression to match port number at start of string
+        port_match = re.match(r"^\d+(?=/|$)", str(ports_str))


I'm wondering if this misses out on some data we could capture. The complex_ports.csv has a few example ports that we lose here, specifically: "80, 443 (https), 8080/tcp (http-alt)" and "22 (ssh)." The latter maps to a port straightforwardly; is the former useful to capture as well? It might require creating multiple endpoints, one for each port (complicating things, I know!) but I'm wondering if it's better to capture all that still.

I'm going to review the original .csv files for more examples. The complex_ports.csv is manipulated and not an original output from rapidfire.

dogboat · 2025-03-18T18:29:03Z

dojo/tools/rapidfire/parser.py

+                    find.unsaved_vulnerability_ids = cves
+
+                # Add tags for ransomware if applicable
+                if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "true":


Same as above comment re: checking "yes" here.

Suggested change

if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "true":

if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "yes":

Same question about the capitalization of "In" in the title as well.

dogboat · 2025-03-18T18:29:28Z

dojo/tools/rapidfire/parser.py

+                    description.append(f"**MAC Address**: {row['MAC Address']}")
+
+                ransomware_warning = "⚠️ **Warning**: This vulnerability is known to be used in ransomware campaigns"
+                if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "true":


It looks like the example CVSs contain yes/no here; might need to check against "yes" rather than "true" if that's what the tool outputs.

Suggested change

if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "true":

if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "yes":

Also possibly not an issue -- some of the example CSVs (complex_ports/invalid_date) use a lowercase 'in' in the title for that column... is that just the test files, or an inconsistency the tools produces that we need to take into account?

Very good catch. You correctly identified an inconsistency strictly within thje complex_ports and invalid_date files. The original .csv example only had 2 findings, and both values for "Known To Be Used In Ransomware Campaigns" were "Unknown". As I don't have direct access to rapidfire to generate more exports, a collegue provided me a 2nd Rapidfire Vulnerability Scan Report (By Device) with more vulnerabilities (this is what I used for the many_vulns.csv). In this file too, every "Known To Be Used In Ransomware Campaigns" value is also "Unknown". So... I can't say if that value would be yes or true... or anything other than "Unknown".
I checked Rapidfiretools.com - Vulscan-external-vulnerability-scan-issues-by-device.pdf - no mention of Ransomware Campaigns.
I also took Rapidfiretools.com painful demo-on-rails trying to see any screenshots or metadata mentioning "Ransomware"... Nothing.
I think without a known value other than "Unknown", I should not parse this data field. I could also parse for every variation of yes/true/no/false... but that's shooting in the dark.

I have not been able to confirm any values for this field other than "Unknown", so going to check for both converted lowercase "yes" and "true" and add a tag if the value is either of those. I am also going to change the value appended to the Description so it isn't using the "Warning" if true, but simply include whatever string value is in the imported data. I will document with comments if we ever find out valid values for that metadata.

…x/2.44.4-2.45.0-dev Release: Merge back 2.44.4 into bugfix from: master-into-bugfix/2.44.4-2.45.0-dev

Co-authored-by: Paul Osinski <[email protected]>

* wiz: handle mitigated timestamps * wiz: handle mitigated timestamps * wiz: handle mitigated timestamps * wiz: handle mitigated timestamps * wiz: failsafe * wiz: failsafe * finalize datetime parsing format

Release: Merge release into master from: release/2.45.0

…x/2.45.0-2.46.0-dev Release: Merge back 2.45.0 into bugfix from: master-into-bugfix/2.45.0-2.46.0-dev

Co-authored-by: Cody Maffucci <[email protected]>

…efectDojo#12129) * 🔨 Rework RustyHog to fix DefectDojo#10584 * Update docs/content/en/connecting_your_tools/parsers/file/rusty_hog.md Co-authored-by: Cody Maffucci <[email protected]> * update --------- Co-authored-by: Cody Maffucci <[email protected]>

* wiz scan: handle more fields * wiz scan: handle more fields

* immuniweb json: domains * immuniweb json * immuniweb json * immuniweb json

* 🎉 Add cisco security advisory to vulnid * fix

* Updated Anchore Enterprise Documentation to fit parser. Note Anchore Engine is EOL replacement is Anchore Enterprise * Fix missing test link in documentation and bumped up chart version * Update helm/defectdojo/Chart.yaml Co-authored-by: Cody Maffucci <[email protected]> * Update anchore_engine.md --------- Co-authored-by: valentijnscholten <[email protected]> Co-authored-by: Cody Maffucci <[email protected]>

* h1: parse main_state * h1: parse main_state * h1: parse main_state * h1: parse main_state * h1: parse main_state

…tDojo#12214)

…2204)

* changelog 2.45.0 * Update docs/content/en/changelog/changelog.md Co-authored-by: Charles Neill <[email protected]> --------- Co-authored-by: Paul Osinski <[email protected]> Co-authored-by: Cody Maffucci <[email protected]> Co-authored-by: Charles Neill <[email protected]>

…ojo#12206)

* 💄 🪲 Fix Aqua parser severity justification * update * update

Release: Merge release into master from: release/2.45.1

dryrunsecurity · 2025-04-14T23:24:06Z

🔴 Risk threshold exceeded.

This pull request contains multiple security concerns, including a sensitive file edit in dojo/finding/views.py, a potential hardcoded encryption key exposure, and vulnerabilities related to access control and information disclosure that could compromise system security if left unaddressed.

⚠️ Configured Codepaths Edit in dojo/finding/views.py

Vulnerability	Configured Codepaths Edit
Description	Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in `.dryrunsecurity.yaml`.

💭 Unconfirmed Findings (5)

Vulnerability	Hardcoded Encryption Key Exposure
Description	Critical vulnerability involving a hardcoded AES256 encryption key in the production installation documentation. The exposed key could potentially allow unauthorized access to encrypted API keys and credentials.

Vulnerability	Access Control Weakness in Image Access
Description	Vulnerability in finding views with simplified token validation logic that could enable unauthorized image access by bypassing size verification mechanisms.

Vulnerability	Potential Information Disclosure
Description	Multiple files contain external URLs and sample scan paths that could provide insights into system configuration and tooling, potentially aiding potential attackers.

Vulnerability	Email Address Exposure
Description	Minor information disclosure of an email address ([email protected]) found in a GitHub PR reminder script.

Vulnerability	Webhook Testing Considerations
Description	Recommendation to use public webhook testing services poses a risk of exposing sensitive webhook payloads to third-party services.

We've notified @mtesauro.

All finding details can be found in the DryRun Security Dashboard.

Add Rapidfire vulnerability scanner parser

8d3ae9e

github-actions bot added settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR docs unittests parser labels Feb 27, 2025

skywalke34 marked this pull request as draft February 27, 2025 05:12

skywalke34 added 13 commits February 27, 2025 09:49

Fix formatting in Rapidfire parser's impact section and ruff lint errors

b913018

Fix Rapidfire parser's ruff lint errors again

5060283

Fix Rapidfire parser's ruff lint errors yet again

643f8e3

Fix Rapidfire parser's ruff lint errors 3rd time a charm

2fdad40

Fix Rapidfire parser's ruff lint errors 4th time

e395078

Fix Rapidfire parser's ruff lint errors 1 missing line!

4d8ab42

Missing title in doc

5f229c6

Merge remote-tracking branch 'upstream/dev' into rapidfire-clean

0735e2a

Merge remote-tracking branch 'upstream/dev' into rapidfire-clean

ab37194

Update documentation details within parser code, unit tests and unitt…

b6b1c96

…est scan examples

Clean Ruff lint errors

d4311d1

Broke unit tests with missing spaces.

5faea6e

Ruff Lint Errors

48650de

skywalke34 marked this pull request as ready for review March 7, 2025 02:49

Maffooch and others added 2 commits March 7, 2025 14:52

Update rapidfire.md

514c3b4

Update rapidfire.md to include missing sections

bf43cab

Failed unit tests looking for ###Sample Scan Data and File Types.

Maffooch approved these changes Mar 14, 2025

View reviewed changes

mtesauro approved these changes Mar 16, 2025

View reviewed changes

hblankenship approved these changes Mar 17, 2025

View reviewed changes

dogboat requested changes Mar 18, 2025

View reviewed changes

DefectDojo release bot and others added 3 commits March 31, 2025 16:01

Update versions in application files

aeca132

Merge pull request DefectDojo#12143 from DefectDojo/master-into-bugfi…

7554a85

…x/2.44.4-2.45.0-dev Release: Merge back 2.44.4 into bugfix from: master-into-bugfix/2.44.4-2.45.0-dev

changelog 2.44.4 (DefectDojo#12150)

a19c511

Co-authored-by: Paul Osinski <[email protected]>

Maffooch and others added 27 commits April 4, 2025 12:18

Vite: Bump to 6.2.5 (DefectDojo#12176)

4dee30e

wiz: handle mitigated timestamps (DefectDojo#12168)

cc5d8ef

* wiz: handle mitigated timestamps * wiz: handle mitigated timestamps * wiz: handle mitigated timestamps * wiz: handle mitigated timestamps * wiz: failsafe * wiz: failsafe * finalize datetime parsing format

Update wording in application versioning (DefectDojo#12174)

08138f9

Update versions in application files

5a93e63

Merge pull request DefectDojo#12187 from DefectDojo/release/2.45.0

2740240

Release: Merge release into master from: release/2.45.0

Update versions in application files

6a1bb84

Merge branch 'bugfix' into master-into-bugfix/2.45.0-2.46.0-dev

80052ac

Merge pull request DefectDojo#12188 from DefectDojo/master-into-bugfi…

3f037d2

…x/2.45.0-2.46.0-dev Release: Merge back 2.45.0 into bugfix from: master-into-bugfix/2.45.0-2.46.0-dev

Bump Django to 5.1.8 (DefectDojo#12191)

e5806c0

Co-authored-by: Cody Maffucci <[email protected]>

README: Point to sample scans for demo (DefectDojo#12162)

5ccbef0

wiz scan: handle more fields and unique_id_from_tool (DefectDojo#12198)

84c6da3

* wiz scan: handle more fields * wiz scan: handle more fields

fix(renovate): Add separateMinorPatch (DefectDojo#12190)

ff97ef9

immuniweb json parser (DefectDojo#12179)

8f0a4a9

* immuniweb json: domains * immuniweb json * immuniweb json * immuniweb json

🎉 Add cisco security advisory to vulnid (DefectDojo#12180)

9134295

* 🎉 Add cisco security advisory to vulnid * fix

Implement HCL vulnids (DefectDojo#12199)

115da96

h1: vulnerability disclosure parser improvements (DefectDojo#12212)

d2265c4

* h1: parse main_state * h1: parse main_state * h1: parse main_state * h1: parse main_state * h1: parse main_state

Update how-to-write-a-parser.md to not contain Ruff violations (Defec…

7098998

…tDojo#12214)

close old findings: don't overwrite mitigated timestamp (DefectDojo#1…

3c8ef5e

…2204)

fix(notif): Product name not rendered correctly (DefectDojo#12203)

ebf0044

🐛 Fix Django template engagement_pdf_report DefectDojo#12201 (DefectD…

aa966d9

…ojo#12206)

💄 🪲 Fix Aqua parser severity justification (DefectDojo#12192)

7b81a16

* 💄 🪲 Fix Aqua parser severity justification * update * update

Update versions in application files

d1eca7c

Merge pull request DefectDojo#12236 from DefectDojo/release/2.45.1

3fd00f0

Release: Merge release into master from: release/2.45.1

Merge branch 'DefectDojo:master' into rapidfire-clean

067ca30

github-actions bot added ui helm labels Apr 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Rapidfire vulnerability scanner parser #11909

Add Rapidfire vulnerability scanner parser #11909

skywalke34 commented Feb 27, 2025 •

edited

Loading

dryrunsecurity bot commented Feb 27, 2025 •

edited

Loading

mtesauro left a comment

dogboat left a comment

dogboat Mar 18, 2025

skywalke34 Mar 21, 2025

dogboat Mar 18, 2025

skywalke34 Mar 21, 2025

dogboat Mar 18, 2025

skywalke34 Mar 21, 2025

dogboat Mar 18, 2025

skywalke34 Mar 21, 2025

dogboat Mar 18, 2025

dogboat Mar 18, 2025

skywalke34 Mar 21, 2025

skywalke34 Apr 15, 2025

dryrunsecurity bot commented Apr 14, 2025

	if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "true":
	if row.get("Known To Be Used In Ransomware Campaigns", "").lower() == "yes":

Add Rapidfire vulnerability scanner parser #11909

Are you sure you want to change the base?

Add Rapidfire vulnerability scanner parser #11909

Conversation

skywalke34 commented Feb 27, 2025 • edited Loading

Rapidfire CSV Parser

CSV Field Mappings

Additional Finding Field Settings

Processing Notes

dryrunsecurity bot commented Feb 27, 2025 • edited Loading

DryRun Security Summary

mtesauro left a comment

Choose a reason for hiding this comment

dogboat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dryrunsecurity bot commented Apr 14, 2025

🔴 Risk threshold exceeded.

skywalke34 commented Feb 27, 2025 •

edited

Loading

dryrunsecurity bot commented Feb 27, 2025 •

edited

Loading