Add validations related to minReplicas and maxReplicas before updating HPA

[AVSBharadwaj]

Summary

Adds comprehensive validation logic to prevent Kubernetes HPA validation errors by ensuring minReplicas <= maxReplicas and respecting all replica bounds across all Tortoise phases.

What Changed

New Validation Logic

• Early validation: Check minReplicas/maxReplicas against user and cluster limits
• Relationship validation: Ensure minReplicas <= maxReplicas
• Phase-specific handling:
  • Emergency: Set minReplicas = maxReplicas
  • BackToNormal: Temporarily increase maxReplicas for safe gradual reduction
  • Working: Use recommended values directly
• Final validation: Double-check all constraints after phase adjustments

Enhanced Logging

• New WarningReplicaValidation event for better categorization
• Clear error messages explaining adjustments made

Test Coverage

Added 8 test cases covering all validation scenarios including edge cases and phase transitions.

Why These Changes

Problem

• Kubernetes rejects HPA configurations where minReplicas > maxReplicas
• Missing validation for replica bounds and user/cluster limits
• Poor error handling during phase transitions (especially BackToNormal)

Solution

• Comprehensive validation prevents HPA errors while maintaining safety
• Smart handling during BackToNormal phase prioritizes safety over strict validation
• Enhanced logging improves debugging and observability

Benefits

• ✅ Reliability: Prevents HPA validation errors
• ✅ Safety: Maintains safety during critical phases
• ✅ Observability: Clear logging for adjustments
• ✅ Backward Compatible: No breaking changes

Testing

• 8 comprehensive unit tests covering all validation scenarios
• Tests edge cases, phase transitions, and boundary conditions

---

Type: Enhancement
Breaking Change: No
Testing: ✅ Unit tests added

[djahandarie]

I feel like there's a bug with the recommender logic if it ever returns recommendMin > recommendMax, should there also be a fix there?

[AVSBharadwaj]

The recommender will not result in such a recommendations ( where recommendMin > recommendMax) . This is added just for completion sake to cover all possible cases.

[djahandarie]

The changes look safe to me and fix a bug, so I think we should more forward with them.

Though, as mentioned on Slack, I think this routine needs some refactoring to make it clearer what invariants we are trying to preserve (probably by having a pure subroutine which purely is responsible for enforcing those invariants, and nothing else), and that subroutine can be called as many times as needed. Should be fine to do in a separate PR.