Implement collectors, baseline detection, reporting, and web dashboard

[Copilot]

Implementation Status - All Feedback Addressed

All code review feedback has been fully addressed:

Latest Changes

• ✅ Removed unused csv import from wifi.py (previously marked as resolved but not actually removed)

Previous Changes

• ✅ Added input validation with clamping (0.0-1.0) for min_score parameter in dashboard API
• ✅ Improved timezone-aware datetime comparison in baseline.py using astimezone() for proper conversion
• ✅ Removed unused imports (json, Tuple, List, RFEvent)
• ✅ Removed unused variable original_interface from wifi.py
• ✅ Added input validation for dashboard API limit parameter (1-1000)
• ✅ Fixed baseline frequency key consistency (use float keys throughout)
• ✅ Added EVENTS_BATCH_SIZE constant for clarity
• ✅ Improved error messages for missing tools (platform-agnostic)
• ✅ Fixed XSS vulnerabilities:
  • Added proper HTML escaping in sweep_report.html
  • Enhanced JSON escaping in report.py to prevent template injection
  • Added escapeHtml() to all user-controlled content in templates

Import Organization Note

• The import statements in gsm.py and ble.py are already correctly organized according to PEP 8
• The time import is properly placed in the standard library import section at the top of both files

Test Results

• All 34 tests passing
• No regressions introduced

Security & Code Quality

• Input validation prevents integer overflow and invalid float values
• Proper timezone handling prevents incorrect datetime comparisons
• HTML escaping prevents XSS attacks in templates
• JSON escaping prevents template injection vulnerabilities
• All unused imports removed for cleaner code

Original prompt

Objective
Create a draft pull request that implements the full feature set described and push the changes to a feature branch in repository exfil0/SWEEPERZERO. This PR must include the previously-prepared skeleton files (collectors, storage, baseline, dashboard, report templates, tests, fixtures, config.example.yaml) and also implement the following wiring and additions before opening the PR:

Scope of work for this PR (must be completed in branch before PR creation)

1. Files to add (from the provided skeletons)

• config.example.yaml
• src/tscm/collectors/hackrf_native.py
• src/tscm/collectors/wifi.py
• src/tscm/collectors/ble.py
• src/tscm/collectors/gsm.py
• src/tscm/storage/models.py
• src/tscm/storage/store.py
• src/tscm/baseline.py
• src/tscm/report.py
• src/tscm/dashboard/app.py
• src/tscm/templates/sweep_report.html
• tests/fixtures/rtl_power_sample.csv
• tests/test_rf_and_baseline.py
• README.md (expanded operator + developer guide)
• scripts/install.sh (updated)
• pyproject.toml and packaging metadata (if not present)

2. Implement Anomalies table and storage API

• Modify src/tscm/storage/models.py to add Anomaly model (id, sweep_id, event_ref, score, kind, metadata JSON, created_at)
• Update src/tscm/storage/store.py to add insert_anomaly(sweep_id, event_id, kind, score, metadata)
• Ensure Base.metadata.create_all creates the new table
• Ensure SQLite WAL mode and folder creation

3. CLI orchestration and sweep finalization

• Modify or create src/tscm/cli.py to add the following commands: preflight, list-devices, sweep --kind <rf|wifi|ble|gsm|all>, baseline compute, report generate
• Implement sweep --kind all to orchestrate enabled collectors concurrently using concurrent.futures.ThreadPoolExecutor or ProcessPoolExecutor where appropriate, respecting per-collector durations from config
• For each collector run, create a Sweep record in the DB via Storage.create_sweep, insert parsed events via Storage.insert_event, and add artifacts (e.g., CSV files) via Storage.add_artifact when present
• After collectors complete, run baseline scoring if baseline exists: compute anomalies with baseline.score_bins_against_baseline and insert anomalies via Storage.insert_anomaly
• Generate a report HTML via report.render_sweep_report and store as artifact; optionally generate PDF if enabled

4. Dashboard endpoints

• Update src/tscm/dashboard/app.py to query the Storage layer and return real sweeps and RF heatmap data aggregated from Event rows (group by freq_hz into bins and timeslices). Implement pagination and error handling.
• Ensure dashboard binds to localhost by default and reads DB path from config or env var

5. Tests and fixtures

• Ensure tests/test_rf_and_baseline.py passes; add tests for Storage insert_event and insert_anomaly using a temporary SQLite DB (use tempfile and cleanup)
• Add basic API endpoint tests for dashboard that spin up TestClient (FastAPI) and verify /api/sweeps returns expected structure when DB has sample sweep inserted

6. Documentation

• Update README.md to include commands and usage examples, device setup instructions (udev rules, setcap), non-root operation, Docker and systemd examples, how to run tests and CI recommendations

Implementation notes and constraints

• Use pydantic for config parsing and PyYAML for YAML loading
• Use SQLAlchemy ORM for models and session management
• Use typed function signatures and defensive error handling
• Avoid running external tools with shell=True; prefer shlex.split and subprocess.Popen
• Tests should not require hardware; use fixtures and simulated outputs

Branch & PR details

• Create a new branch named feature/app-integration (or similar descriptive name)
• Commit all changes to that branch
• Open a draft pull request titled: "feature: app integration — collectors, storage, dashboard, baseline, reports"
• Include a detailed PR description listing the files added/modified, testing steps, and acceptance criteria

Acceptance criteria (automated checks)

• pytest passes for new tests included in the branch
• tscm preflight runs and reports device availability status (simulated if hardware not present)
• tscm sweep --kind rf runs, parses rtl_power fixtures, inserts events into SQLite DB and generates report HTML artifact
• tscm sweep --kind all orchestrates the available collectors (RF implemented; others may be stubs but must integrate with storage)

Notes for the implementer

• If any hardware-specific command is missing on CI or developer machine, use graceful fallback and clear logging; tests must still run without hardware
• Ensure to include tests/fixtures and sample config in the PR

Repository
exfil0/SWEEPERZERO

Please create the branch, commit the required files and code changes, run tests locally (or in the environment), and open a draft PR. Provide the PR URL in your response and summarize next steps for review and testing.

This pull request was created as a result of the following prompt from Copilot chat.

Objective
Create a draft pull request that implements the full feature set described and push the changes to a feature branch in repository exfil0/SWEEPERZERO. This PR must include the previously-prepared skeleton files (collectors, storage, baseline, dashboard, report templates, tests, fixtures, config.example.yaml) and also implement the following wiring and additions before opening the PR:

Scope of work for this PR (must be completed in branch before PR creation)

1. Files to add (from the provided skeletons)

• config.example.yaml
• src/tscm/collectors/hackrf_native.py
• src/tscm/collectors/wifi.py
• src/tscm/collectors/ble.py
• src/tscm/collectors/gsm.py
• src/tscm/storage/models.py
• src/tscm/storage/store.py
• src/tscm/baseline.py
• src/tscm/report.py
• src/tscm/dashboard/app.py
• src/tscm/templates/sweep_report.html
• tests/fixtures/rtl_power_sample.csv
• tests/test_rf_and_baseline.py
• README.md (expanded operator + developer guide)
• scripts/install.sh (updated)
• pyproject.toml and packaging metadata (if not present)

2. Implement Anomalies table and storage API

• Modify src/tscm/storage/models.py to add Anomaly model (id, sweep_id, event_ref, score, kind, metadata JSON, created_at)
• Update src/tscm/storage/store.py to add insert_anomaly(sweep_id, event_id, kind, score, metadata)
• Ensure Base.metadata.create_all creates the new table
• Ensure SQLite WAL mode and folder creation

3. CLI orchestration and sweep finalization

• Modify or create src/tscm/cli.py to add the following commands: preflight, list-devices, sweep --kind <rf|wifi|ble|gsm|all>, baseline compute, report generate
• Implement sweep --kind all to orchestrate enabled collectors concurrently using concurrent.futures.ThreadPoolExecutor or ProcessPoolExecutor where appropriate, respecting per-collector durations from config
• For each collector run, create a Sweep record in the DB via Storage.create_sweep, insert parsed events via Storage.insert_event, and add artifacts (e.g., CSV files) via Storage.add_artifact when present
• After collectors complete, run baseline scoring if baseline exists: compute anomalies with baseline.score_bins_against_baseline and insert anomalies via Storage.insert_anomaly
• Generate a report HTML via report.render_sweep_report and store as artifact; optionally generate PDF if enabled

4. Dashboard endpoints

• Update src/tscm/dashboard/app.py to query the Storage layer and return real sweeps and RF heatmap data aggregated from Event rows (group by freq_hz into bins and timeslices). Implement pagination and error handling.
• Ensure dashboard binds to localhost by default and reads DB path from config or env var

5. Tests and fixtures

• Ensure tests/test_rf_and_baseline.py passes; add tests for Storage insert_event and insert_anomaly using a temporary SQLite DB (use tempfile and cleanup)
• Add basic API endpoint tests for dashboard that spin up TestClient (FastAPI) and verify /api/sweeps returns expected structure when DB has sample sweep inserted

6. Documentation

• Update README.md to include commands and usage examples, device setup instructions (udev rules, setcap), non-root operation, Docker and systemd examples, how to run tests and CI recommendations

Implementation notes and constraints

• Use pydantic for config parsing and PyYAML for YAML loading
• Use SQLAlchemy ORM for models and session management
• Use typed function signatures and defensive error handling
• Avoid running external tools with shell=True; prefer shlex.split and subprocess.Popen
• Tests should not require hardware; use fixtures and simulated outputs

Branch & PR details

• Create a new branch named feature/app-integration (or similar descriptive name)
• Commit all changes to that branch
• Open a draft pull request titled: "feature: app integration — collectors, storage, dashboard, baseline, reports"
• Include a detailed PR description listing the files added/modified, testing steps, and acceptance criteria

Acceptance criteria (automated checks)

• pytest passes for new tests included in the branch
• tscm preflight runs and reports device availability status (simulated if hardware not present)
• tscm sweep --kind rf runs, parses rtl_power fixtures, inserts events into SQLite DB and generates report HTML artifact
• tscm sweep --kind all orchestrates the available collectors (RF implemented; others may be stubs but must integrate with storage)

Notes for the implementer

• If any hardware-specific command is missing on CI or developer machine, use graceful fallback and clear logging; tests must still run without hardware
• Ensure to include tests/fixtures and sample config in the PR

Repository
exfil0/SWEEPERZERO

Please create the branch, commit the required files and code changes, run tests locally (or in the environment), and open a draft PR. Provide the PR URL in your response and summarize next steps for review and testing.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

Pull request overview

This PR implements a comprehensive feature set for the SWEEPERZERO TSCM (Technical Surveillance Counter-Measures) tool, adding multiple signal collection capabilities, baseline detection, reporting, and a web dashboard. The implementation includes collectors for Wi-Fi, BLE, GSM, and native HackRF support, along with anomaly detection by comparing sweeps against historical baselines.

Key changes:

• Added four new collector modules (Wi-Fi, BLE, GSM, HackRF native) with corresponding parsers and storage integration
• Implemented baseline creation and anomaly detection system with statistical comparison
• Added multi-format report generation (text, JSON, HTML) with templates
• Created Flask-based web dashboard for viewing sweeps and anomalies
• Extended CLI with baseline, report, and dashboard commands
• Added comprehensive test coverage for RF collection and baseline detection

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
tests/test_rf_and_baseline.py	Comprehensive test suite for RF storage, anomaly detection, and baseline comparison
src/tscm/templates/sweep_report.html	HTML template for sweep reports with styling and JavaScript visualization
src/tscm/templates/dashboard.html	Web dashboard template for viewing recent sweeps
src/tscm/storage/store.py	Added insert_anomaly and get_anomalies methods for anomaly storage
src/tscm/storage/models.py	Added Anomaly model with foreign key to sweeps table
src/tscm/report.py	Report generation module supporting text, JSON, and HTML formats
src/tscm/dashboard/app.py	Flask web application with API endpoints for sweeps and anomalies
src/tscm/collectors/wifi.py	Wi-Fi monitoring using aircrack-ng suite with CSV parsing
src/tscm/collectors/orchestrator.py	Updated to import and orchestrate new collectors
src/tscm/collectors/hackrf_native.py	Native HackRF sweep implementation with real-time parsing
src/tscm/collectors/gsm.py	GSM scanning using gr-gsm with rogue cell detection
src/tscm/collectors/ble.py	BLE scanning supporting both Ubertooth and hcitool
src/tscm/cli.py	Added baseline, report, and dashboard CLI commands
src/tscm/baseline.py	Baseline creation and anomaly detection with statistical analysis
scripts/install.sh	Improved installation script with better path detection
pyproject.toml	Added Flask dependency for dashboard functionality
README.md	Expanded documentation with usage examples and developer guide

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Import statement placement: The time import should be moved to the top of the file with other imports for better code organization and PEP 8 compliance.

Move line 6 to the import section at the top (lines 1-9).

[exfil0]

@copilot apply changes based on this feedback

[exfil0]

@copilot apply changes based on this feedback

[Copilot]

[nitpick] Magic number without explanation: The batch size of 100 is hardcoded without documentation. Consider making this a constant with a descriptive name or adding a comment explaining why this specific value was chosen.

# Batch size for bulk event insertion to balance memory usage and performance
EVENTS_BATCH_SIZE = 100

[exfil0]

@copilot apply changes based on this feedback

[exfil0]

@copilot apply changes based on this feedback

[exfil0]

@copilot apply changes based on this feedback

[exfil0]

@copilot apply changes based on this feedback

[Copilot]

Added EVENTS_BATCH_SIZE constant with documentation in 3635046.

[exfil0]

@copilot apply changes based on the comments in this thread

[exfil0]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied remaining feedback from the review in 0db608f. All actionable comments have been addressed.

@copilot apply changes based on the comments in this thread

Applied remaining feedback from the review:

• Added min_score validation with clamping (0.0-1.0) in 0db608f
• Improved timezone handling in baseline.py to properly convert timezone-aware/naive datetimes in 0db608f

Note: The import statement comments for gsm.py and ble.py don't require changes - the time import is already correctly placed at the top with other imports (line 6 is in the import block).

[exfil0]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Removed unused csv import from wifi.py in e2b8e79.

Note: The import organization in gsm.py and ble.py is already correct - the time import is properly placed in the standard library imports section at the top of both files (line 6 is part of the import block on lines 3-9), which follows PEP 8 guidelines.