Breaking Change Detection Support

[connerohnesorge]

Breaking Change Detection Support

Summary

Implement native breaking change detection for Protocol Buffer schemas in Bufrnix to help teams maintain API compatibility and prevent unintentional breaking changes during schema evolution. This feature will integrate seamlessly with Bufrnix's local-first philosophy while leveraging proven tools from the protobuf ecosystem.

Problem Statement

Currently, Bufrnix focuses solely on code generation while schema compatibility validation is left to external tools like buf CLI. This creates several challenges:

1. Workflow Fragmentation: Teams must use multiple tools (buf for validation, Bufrnix for generation)
2. Missing Integration: No built-in way to validate schemas before code generation
3. Developer Experience: Additional setup complexity for comprehensive protobuf workflows
4. CI/CD Gaps: Breaking changes may not be caught until code generation or runtime
5. Context Loss: Current examples include buf in devShells but don't integrate it into the generation workflow

Goals

Primary Goals

• Native Integration: Built-in breaking change detection within Bufrnix workflows
• Local-First Approach: Maintain Bufrnix's philosophy of offline-first development
• Flexible Configuration: Support different compatibility modes (backward, forward, full)
• Developer-Friendly: Clear error messages and actionable feedback
• Seamless Integration: Work with existing Bufrnix architecture and language modules

Secondary Goals

• CI/CD Integration: Easy integration into automated pipelines
• Performance: Fast validation suitable for development workflows
• Extensibility: Plugin architecture for custom validation rules

Technical Implementation Strategy

Architecture Integration Points

Based on the Bufrnix architecture analysis, breaking change detection should integrate at these key points:

1. Configuration Schema (src/lib/bufrnix-options.nix): Add new breaking change options
2. Core Generation (src/lib/mkBufrnix.nix): Pre-generation validation hook
3. Language Modules (src/languages/*/): Optional per-language breaking change rules
4. Debug System (src/lib/utils/debug.nix): Enhanced logging for validation results

1. Configuration Schema Extension

Add breaking change detection options to bufrnix-options.nix:

# Add to the main options set
breaking = {
  enable = mkOption {
    type = types.bool;
    default = false;
    description = ''
      Enable breaking change detection using buf CLI.
      Validates schema compatibility before code generation.
    '';
  };

  mode = mkOption {
    type = types.enum ["backward" "forward" "full"];
    default = "backward";
    description = ''
      Compatibility mode for breaking change detection:
      - backward: New code can read old data (allows adding fields)
      - forward: Old code can read new data (allows removing optional fields)
      - full: Both backward and forward compatible (most restrictive)
    '';
  };

  baseReference = mkOption {
    type = types.str;
    default = "HEAD~1";
    description = ''
      Git reference for comparison base (commit, branch, tag).
      Examples: "HEAD~1", "origin/main", "v1.0.0"
    '';
  };

  basePath = mkOption {
    type = types.nullOr types.path;
    default = null;
    description = ''
      Alternative: path to base proto files directory for comparison.
      If set, takes precedence over baseReference.
    '';
  };

  configFile = mkOption {
    type = types.nullOr types.path;
    default = null;
    description = ''
      Path to buf.yaml configuration file.
      If null, uses default buf configuration.
    '';
  };

  failOnBreaking = mkOption {
    type = types.bool;
    default = true;
    description = ''
      Fail the build when breaking changes are detected.
      Set to false for warning-only mode.
    '';
  };

  outputFormat = mkOption {
    type = types.enum ["text" "json"];
    default = "text";
    description = ''
      Output format for breaking change reports.
      JSON format useful for CI/CD integration.
    '';
  };

  ignoreRules = mkOption {
    type = types.listOf types.str;
    default = [];
    description = ''
      List of breaking change rule IDs to ignore.
      Run 'buf breaking --help' to see available rules.
    '';
    example = literalExpression ''
      [
        "FIELD_REMOVED"
        "ENUM_VALUE_REMOVED"
        "SERVICE_REMOVED"
      ]
    '';
  };

  extraArgs = mkOption {
    type = types.listOf types.str;
    default = [];
    description = ''
      Additional arguments to pass to 'buf breaking' command.
      For advanced use cases and future buf CLI features.
    '';
  };

  timeout = mkOption {
    type = types.int;
    default = 30;
    description = ''
      Timeout in seconds for breaking change detection.
      Prevents hanging on complex schemas or Git operations.
    '';
  };
};

2. Core Integration in mkBufrnix.nix

Integrate breaking change detection into the main generation workflow:

# Add to package defaults section
packageDefaults = {
  # ... existing defaults ...
  breaking = {
    package = pkgs.buf; # buf CLI is already in nixpkgs
  };
};

# Add breaking change validation function
validateBreakingChanges = let
  breakingEnabled = cfg.breaking.enable;
  bufPackage = cfg.breaking.package or pkgs.buf;
in
  if !breakingEnabled
  then ""
  else let
    # Construct buf breaking command arguments
    modeArg = "--against-config-type ${cfg.breaking.mode}";
    
    baseArg =
      if cfg.breaking.basePath != null
      then "--against ${cfg.breaking.basePath}"
      else "--against .git#branch=${cfg.breaking.baseReference}";
    
    configArg =
      if cfg.breaking.configFile != null
      then "--config ${cfg.breaking.configFile}"
      else "";
    
    formatArg = "--format ${cfg.breaking.outputFormat}";
    
    ignoreArgs = concatMapStrings (rule: " --ignore ${rule}") cfg.breaking.ignoreRules;
    
    extraArgs = concatStringsSep " " cfg.breaking.extraArgs;
    
    timeoutCmd = "${pkgs.coreutils}/bin/timeout ${toString cfg.breaking.timeout}";
    
    allArgs = "${modeArg} ${baseArg} ${configArg} ${formatArg}${ignoreArgs} ${extraArgs}";
  in ''
    ${debug.log 1 "Running breaking change detection" cfg}
    
    echo "🔍 Checking for breaking changes..."
    
    # Ensure buf is available and working directory is set up
    cd "${cfg.root}"
    
    # Run breaking change detection with timeout
    if ${timeCmd} ${bufPackage}/bin/buf breaking ${allArgs} . 2>&1; then
      ${debug.log 2 "No breaking changes detected" cfg}
      echo "✅ No breaking changes detected"
    else
      exit_code=$?
      if [ $exit_code -eq 124 ]; then
        echo "❌ Breaking change detection timed out after ${toString cfg.breaking.timeout}s"
        ${if cfg.breaking.failOnBreaking then "exit 1" else "echo '⚠️  Continuing despite timeout (failOnBreaking=false)'"}
      elif [ $exit_code -ne 0 ]; then
        echo "❌ Breaking changes detected!"
        ${debug.log 1 "Breaking changes found, exit code: $exit_code" cfg}
        ${if cfg.breaking.failOnBreaking then "exit $exit_code" else "echo '⚠️  Continuing despite breaking changes (failOnBreaking=false)'"}
      fi
    fi
    
    echo ""
  '';

# Integrate into main text generation
text = ''
  ${debug.log 1 "Starting Bufrnix code generation" cfg}
  
  # Run breaking change detection if enabled (before any code generation)
  ${validateBreakingChanges}
  
  # Expand proto file globs if needed (existing logic)
  proto_files=""
  ${existingProtoFileLogic}
  
  # Continue with existing generation logic...
  ${existingGenerationScript}
'';

3. Runtime Dependencies Integration

Update runtime inputs to include buf when breaking change detection is enabled:

# Add conditional buf dependency
breakingRuntimeInputs =
  if cfg.breaking.enable
  then [pkgs.buf pkgs.git pkgs.coreutils] # git needed for baseReference, coreutils for timeout
  else [];

runtimeInputs = with pkgs;
  [
    bash
    protobuf
  ]
  ++ languageRuntimeInputs
  ++ breakingRuntimeInputs;

4. Example Integration Patterns

Basic Breaking Change Detection

config = {
  root = ./.;
  protoc = {
    sourceDirectories = ["./proto"];
    files = ["./proto/example/v1/user.proto"];
  };
  
  # Enable basic backward compatibility checking
  breaking = {
    enable = true;
    mode = "backward";
    baseReference = "origin/main";
  };
  
  languages = {
    go.enable = true;
    # ... other languages
  };
};

Advanced CI/CD Integration

breaking = {
  enable = true;
  mode = "full";                    # Strictest compatibility
  baseReference = "HEAD~1";         # Compare with previous commit
  outputFormat = "json";            # Machine-readable output for CI
  failOnBreaking = true;            # Fail build on breaking changes
  timeout = 60;                     # Extended timeout for large schemas
  
  ignoreRules = [
    # Allow specific breaking changes during major version bumps
    "FIELD_REMOVED"
    "SERVICE_REMOVED"
  ];
  
  extraArgs = [
    "--error-format json"           # Structured error output
  ];
};

Development Mode with Warnings

breaking = {
  enable = true;
  mode = "backward";
  baseReference = "HEAD~1";
  failOnBreaking = false;           # Warning mode for development
  outputFormat = "text";            # Human-readable output
};

5. Integration with Existing Examples

Update existing examples to show breaking change detection:

# In examples/multilang/flake.nix
devShells.default = pkgs.mkShell {
  packages = with pkgs; [
    # ... existing packages ...
    buf  # Already included, now integrated into Bufrnix workflow
  ];
  
  shellHook = ''
    echo "Breaking Change Detection: buf breaking commands available"
    echo "  buf breaking --against .git#branch=main"
    echo "  Or enable in flake.nix: breaking.enable = true;"
  '';
};

packages.default = bufrnix.lib.mkBufrnixPackage {
  inherit pkgs;
  config = {
    # ... existing config ...
    
    # Demonstrate breaking change detection
    breaking = {
      enable = true;
      mode = "backward";
      baseReference = "HEAD~1";
      failOnBreaking = false; # Warning mode for example
    };
  };
};

Workflow Integration Examples

Local Development Workflow

# Developer making schema changes
nix develop
# Edit proto files: add new field, remove old field, etc.
nix run  # Automatically checks for breaking changes before generation

# If breaking changes detected:
# ❌ Breaking changes detected!
# FIELD_REMOVED: Field "user.deprecated_field" was removed.
# To ignore this change, add "FIELD_REMOVED" to breaking.ignoreRules

CI/CD Pipeline Integration

# .github/workflows/protobuf.yml
- name: Validate Schema Compatibility
  run: |
    nix build .#bufrnix-check
    # This runs breaking change detection with JSON output
    # Fails CI if breaking changes found (unless failOnBreaking=false)

- name: Generate Code
  run: |
    nix build
    # Only runs if breaking change check passes

Git Hooks Integration

# pre-commit hook
#!/usr/bin/env bash
# Check breaking changes before commit
if nix build .#bufrnix --dry-run 2>&1 | grep "Breaking changes detected"; then
  echo "❌ Cannot commit: Breaking changes detected in proto files"
  echo "Run 'nix run' to see details or update breaking.ignoreRules"
  exit 1
fi

Implementation Benefits

1. Seamless Integration

• Breaking change detection runs automatically before code generation
• No separate commands or tools needed
• Consistent with Bufrnix's single-command philosophy

2. Local-First Philosophy

• Uses local Git history for comparison (no network required)
• buf CLI is packaged in nixpkgs (reproducible across environments)
• All validation happens offline

3. Flexible Configuration

• Multiple compatibility modes for different use cases
• Configurable ignore rules for controlled breaking changes
• Warning vs. error modes for different environments

4. Developer Experience

• Clear error messages with specific rule violations
• Actionable feedback (suggests ignore rules when appropriate)
• Integrated debug logging using existing debug system

5. CI/CD Ready

• JSON output format for machine processing
• Configurable timeouts prevent hanging builds
• Exit codes compatible with standard CI systems

Testing Strategy

1. Unit Tests for Configuration Schema

• Test option validation and defaults
• Verify type checking for all breaking change options
• Test configuration merging with existing options

2. Integration Tests

• Create test schemas with known breaking changes
• Verify detection of each breaking change rule type
• Test ignore functionality for each rule type

3. Example Testing Updates

Extend ./test-examples.sh to include breaking change scenarios:

# Add to test-examples.sh
test_breaking_changes() {
  echo "Testing breaking change detection..."
  
  cd examples/breaking-changes-example
  
  # Test 1: No breaking changes
  if nix build 2>&1; then
    echo "✅ No breaking changes test passed"
  else
    echo "❌ False positive breaking change detection"
    return 1
  fi
  
  # Test 2: Detected breaking changes (warning mode)
  git checkout breaking-schema
  if nix build 2>&1 | grep "Breaking changes detected"; then
    echo "✅ Breaking change detection working"
  else
    echo "❌ Breaking changes not detected"
    return 1
  fi
  
  git checkout main
}

4. Performance Testing

• Test with large schema repositories
• Verify timeout functionality
• Benchmark impact on generation time

Future Enhancements

Phase 1: Core Implementation

• Implement configuration schema in bufrnix-options.nix
• Add breaking change validation to mkBufrnix.nix
• Update examples to demonstrate functionality
• Add comprehensive documentation

Phase 2: Advanced Features

• Custom rule definitions beyond buf's built-in rules
• Integration with schema registries
• Breaking change impact analysis per language
• Automated migration guides for breaking changes

Phase 3: Ecosystem Integration

• VS Code extension integration
• GitHub Action for automated PR checking
• Integration with package versioning workflows
• Compatibility matrix generation for multi-version APIs

Success Metrics

Adoption Metrics

• Number of projects enabling breaking change detection
• Frequency of use in CI/CD pipelines
• Community feedback and feature requests

Quality Metrics

• Reduction in API compatibility issues reported
• Developer productivity improvements
• Time to detect breaking changes in development cycle

Related Work

• buf CLI: Proven breaking change detection with comprehensive rule set
• Protobuf Ecosystem: Standard practice for schema validation
• Bufrnix Examples: Already include buf in development shells, ready for integration

---

This implementation leverages Bufrnix's existing architecture while adding powerful breaking change detection capabilities. The integration is designed to be seamless, maintaining the local-first philosophy while providing enterprise-grade schema validation.