feat: cloud-native GPU health event management

.gitattributes

@@ -0,0 +1,35 @@
+# Copyright (c) 2026, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ==============================================================================
+# GIT ATTRIBUTES
+# ==============================================================================
+# Use 'linguist-generated=true' to hide generated code from GitHub PR diffs.
+# ==============================================================================
+
+# Hide Kubernetes generated helpers (DeepCopy, Defaults, Conversions, OpenAPI, etc)
+zz_generated.*.go linguist-generated=true
+
+# Hide generated Protobuf Go bindings
+*.pb.go linguist-generated=true
+
+# Hide generated client library
+client-go/client/** linguist-generated=true
+client-go/listers/** linguist-generated=true
+client-go/informers/** linguist-generated=true
+
+# Hide copied, unmodified upstream code
+code-generator/cmd/client-gen/types/** linguist-generated=true
+code-generator/cmd/client-gen/generators/scheme/** linguist-generated=true
+code-generator/cmd/client-gen/generators/util/** linguist-generated=true

.github/copilot-instructions.md

@@ -1,260 +1,115 @@
-# GitHub Copilot Instructions for NVSentinel
+# Copilot Instructions for NVSentinel
 
-## Project Overview
-
-NVSentinel is a GPU Node Resilience System for Kubernetes that automatically detects, classifies, and remediates hardware and software faults in GPU nodes. It's designed for high-performance computing environments running NVIDIA GPUs.
+This file provides repository-level instructions for GitHub Copilot to improve
+code reviews and suggestions.
 
-**Status**: Experimental/Preview Release - APIs and configurations may change.
+## Project Overview
 
-## Architecture & Technologies
+NVSentinel is the NVIDIA Device API — a Protocol Buffer and gRPC-based API for
+GPU device management. The codebase consists primarily of `.proto` definitions
+and their generated Go code.
 
-### Core Technologies
-- **Language**: Go 1.25+ (primary), Python 3.10+ (monitoring tools)
-- **Container Platform**: Kubernetes 1.25+
-- **Deployment**: Helm 3.0+, Tilt (development)
-- **Storage**: MongoDB (event store with change streams)
-- **Communication**: gRPC with Protocol Buffers
-- **GPU Monitoring**: NVIDIA DCGM (Data Center GPU Manager)
+## Repository Structure
 
-### Project Structure
 ```
-├── health-monitors/          # Pluggable health detection modules
-│   ├── gpu-health-monitor/   # DCGM-based GPU monitoring (Python)
-│   ├── csp-health-monitor/   # Cloud provider health checks (Go)
-│   └── syslog-health-monitor/ # System log analysis (Go)
-├── health-events-analyzer/   # Event classification and routing
-├── fault-quarantine/  # Node isolation (cordon)
-├── node-drainer/      # Workload eviction
-├── fault-remediation/ # Break-fix automation
-├── labeler/           # Node labeling (DCGM version, driver status, Kata detection)
-├── janitor/                  # State cleanup and maintenance
-├── platform-connectors/      # CSP integration (GCP, AWS, Azure)
-├── commons/                  # Shared utilities
-├── data-models/             # Protocol Buffer definitions
-├── store-client/        # MongoDB client library
-└── distros/kubernetes/      # Helm charts
+api/
+├── proto/                    # Protocol Buffer source definitions (edit these)
+│   └── device/v1alpha1/
+│       └── gpu.proto
+├── gen/go/                   # Generated Go code (DO NOT edit manually)
+│   └── device/v1alpha1/
+│       ├── gpu.pb.go
+│       └── gpu_grpc.pb.go
+├── go.mod
+└── Makefile
 ```
 
-## Coding Standards
-
-### Go Code Guidelines
-
-#### Module Organization
-- Each service is a separate Go module with its own `go.mod`
-- Use semantic import versioning
-- Keep dependencies minimal and up-to-date
-- Use `commons/` for shared utilities across modules
-
-#### Code Style
-- Follow standard Go conventions (gofmt, golint)
-- Use structured logging via `log/slog`
-- Error handling: wrap errors with context using `fmt.Errorf("context: %w", err)`
-- Within `retry.RetryOnConflict` blocks, return errors **without wrapping** to preserve retry behavior
-- Use meaningful variable names (`synced` over `ok` for cache sync checks)
-
-#### Kubernetes Integration
-- Use `client-go` for Kubernetes API interactions
-- Prefer informers over direct API calls for watching resources
-- Use `envtest` for testing Kubernetes controllers (not fake clients)
-- Implement proper shutdown handling with context cancellation
-
-#### Testing Requirements
-- Use `testify/assert` and `testify/require` for assertions
-- Write table-driven tests when testing multiple scenarios
-- Use `envtest` for integration tests with real Kubernetes API
-- Test coverage: aim for >80% on critical paths
-- Name tests descriptively: `TestFunctionName_Scenario_ExpectedBehavior`
-
-### Python Code Guidelines
-- Use Poetry for dependency management
-- Follow PEP 8 style guide
-- Use Black for formatting
-- Type hints required for all functions
-- Use dataclasses for structured data
-
-### Protobuf Guidelines
-- Define messages in `data-models/protobufs/`
-- Use semantic versioning for breaking changes
-- Include comprehensive comments for all fields
-- Generate code with: `make protos-generate`
-
-## Development Workflows
-
-### Building & Testing
-```bash
-# Lint and test all modules
-make lint-test-all
+## Code Review Guidelines
 
-# Lint specific module
-cd labeler && make lint
+### Protocol Buffers
 
-# Test specific module
-cd health-events-analyzer && make test
+When reviewing `.proto` files:
 
-# Build container images (uses ko)
-make images
+- **Field naming**: Use `snake_case` for field names
+- **Message/Service naming**: Use `CamelCase`
+- **Documentation**: Every message, field, and RPC must have documentation
+  comments explaining purpose and usage
+- **Field numbers**: Never reuse or change field numbers in existing messages
+- **Backwards compatibility**: New fields should be optional; avoid breaking
+  changes to existing APIs
+- **Style**: Follow [Google's Protocol Buffer Style Guide](https://protobuf.dev/programming-guides/style/)
 
-# View all make targets
-make help
-```
+### Generated Code
 
-### Version Management
-- All tool versions centralized in `.versions.yaml`
-- Use `yq` to read versions in scripts
-- Update versions in one place, propagates everywhere
-- Check versions: `make show-versions`
+- Files in `api/gen/` are **auto-generated** — do not suggest edits to these
+- If generated code appears outdated, suggest running `make protos-generate`
+
+### Go Code
+
+- Use `gofmt` formatting (enforced by golangci-lint)
+- Documentation comments should wrap at 80 characters
+- Follow standard Go idioms and error handling patterns
+- No manual edits to `*.pb.go` or `*_grpc.pb.go` files
+
+## Commit Standards
+
+### Conventional Commits
+
+All commits must follow conventional commit format:
 
-### Local Development with Tilt
-```bash
-cd tilt
-tilt up  # Start local development environment
 ```
+feat: add new GPU condition type
+fix: correct timestamp handling in conditions
+docs: update API documentation
+chore: update protoc-gen-go version
+```
+
+### DCO Sign-off
+
+All commits **must** include a DCO sign-off line:
 
-## Kata Containers Detection
-
-The labeler implements Kata Containers detection:
-
-### Detection Architecture
-- **Input labels** (on nodes): `katacontainers.io/kata-runtime` (default) + optional custom label
-- **Output label** (set by labeler): `nvsentinel.dgxc.nvidia.com/kata.enabled: "true"|"false"`
-- **Truthy values**: `"true"`, `"enabled"`, `"1"`, `"yes"` (case-insensitive)
-- **Lifecycle separation**: Pod events → DCGM/driver labels, Node events → kata labels
-
-### DaemonSet Variants
-- Separate DaemonSets for kata vs regular nodes
-- Selection via `nodeAffinity` based on kata.enabled label
-- Different volume mounts:
-  - Regular: `/var/log` (file-based logs)
-  - Kata: `/run/log/journal` and `/var/log/journal` (systemd journal)
-
-## Important Patterns
-
-### Error Handling in Retry Loops
-```go
-// ✅ CORRECT - Return error as-is for retry logic
-err := retry.RetryOnConflict(retry.DefaultBackoff, func() error {
-    _, err := client.Update(ctx, obj, metav1.UpdateOptions{})
-    return err  // Don't wrap!
-})
-
-// ❌ WRONG - Wrapping breaks retry detection
-err := retry.RetryOnConflict(retry.DefaultBackoff, func() error {
-    _, err := client.Update(ctx, obj, metav1.UpdateOptions{})
-    return fmt.Errorf("failed: %w", err)  // Breaks retry!
-})
 ```
+Signed-off-by: Name <email@example.com>
+```
+
+Use `git commit -s` to add this automatically.
+
+## License Headers
+
+All source files must include the Apache 2.0 license header:
 
-### Informer Usage
-```go
-// Use separate informers for different resource types
-podInformer := factory.Core().V1().Pods().Informer()
-nodeInformer := factory.Core().V1().Nodes().Informer()
-
-// Extract event handler setup into helper methods
-func (l *Labeler) registerPodEventHandlers() error {
-    _, err := l.podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
-        // handlers...
-    })
-    return err
-}
 ```
+Copyright (c) 2026, NVIDIA CORPORATION.  All rights reserved.
 
-### Testing with envtest
-```go
-// Use real Kubernetes API server for tests
-testEnv := &envtest.Environment{
-    CRDDirectoryPaths: []string{},
-}
-cfg, err := testEnv.Start()
-// Create clients from cfg
-// Run tests
-testEnv.Stop()
+Licensed under the Apache License, Version 2.0 (the "License");
+...
 ```
 
-## Documentation Standards
-
-### Code Comments
-- Package-level godoc for all packages
-- Function comments for exported functions
-- Inline comments for complex logic only
-- TODO comments should reference issues
-
-### Helm Chart Documentation
-- Document all values in `values.yaml` with inline comments
-- Include examples for non-obvious configurations
-- Note truthy value requirements where applicable
-- Explain DaemonSet variant selection logic
-
-## Security & Compliance
-
-### Licensing
-- All files must include Apache 2.0 license header
-- Use `make license-headers-add` to add headers
-- No GPL or viral licenses in dependencies
-
-### Security
-- Report security issues via SECURITY.md process
-- Never commit secrets or credentials
-- Use Kubernetes secrets for sensitive data
-- Scan dependencies: automated via CI
-
-## CI/CD Pipeline
-
-### GitHub Actions Workflows
-- **PR Checks**: lint, test, build for all modules
-- **Release**: automated image builds and Helm chart publishing
-- **Vulnerability Scanning**: daily Trivy scans uploaded to Security tab
-- **Provenance**: SLSA attestation for all artifacts
-
-### Container Images
-- Built with `ko` (Go) and `docker` (Python)
-- Multi-architecture support (amd64, arm64)
-- Published to `ghcr.io/nvidia/nvsentinel/*`
-- Signed with Sigstore cosign
-
-## Common Tasks
-
-### Adding a New Health Monitor
-1. Create directory in `health-monitors/`
-2. Implement gRPC service from `data-models/protobufs/`
-3. Add Makefile with standard targets
-4. Create Helm chart in `distros/kubernetes/nvsentinel/charts/`
-5. Register in platform-connectors for CSP integration
-6. Add to CI pipeline workflows
-
-### Updating Dependencies
-```bash
-# Update Go dependencies
-make gomod-tidy
+## What to Flag in Reviews
 
-# Update Python dependencies
-cd health-monitors/gpu-health-monitor
-poetry update
-```
+1. **Breaking API changes** without migration path
+2. **Missing documentation** on proto messages/fields/RPCs
+3. **Manual edits** to generated files
+4. **Missing license headers**
+5. **Unsigned commits** (missing DCO)
+6. **Non-conventional commit messages**
 
-### Adding New Node Labels
-1. Update labeler logic in `pkg/labeler/labeler.go`
-2. Add tests in `labeler_test.go`
-3. Document in Helm chart values
-4. Update KATA_TESTING.md if kata-related
+## What NOT to Flag
 
-## Debugging Tips
+1. Style issues in generated `*.pb.go` files
+2. Import ordering in generated code
+3. Line length in generated code
 
-### Local Development
-- Use Tilt for rapid iteration
-- Check logs: `kubectl logs -n nvsentinel <pod>`
-- Port-forward for direct access: `kubectl port-forward -n nvsentinel svc/janitor 8080:8080`
-- MongoDB Compass for event store inspection
+## Build Commands
 
-### Common Issues
-- **Informer not syncing**: Check RBAC permissions
-- **Labels not updating**: Verify labeler is running and has node permissions
-- **DaemonSet not scheduling**: Check node selectors and labels
-- **Kata detection failing**: Verify input labels on nodes
+```bash
+make protos-generate  # Regenerate Go code from .proto files
+make build            # Build the Go module
+make lint             # Run go vet
+make test             # Run tests
+```
 
-## References
+## Tool Versions
 
-- [DEVELOPMENT.md](../DEVELOPMENT.md) - Detailed development guide
-- [CONTRIBUTING.md](../CONTRIBUTING.md) - Contribution guidelines
-- [README.md](../README.md) - Project overview
-- [docs/](../docs/) - Architecture documentation
+Tool versions are centralized in `.versions.yaml`. When suggesting dependency
+updates, reference this file.