Revision spuriously marked "Initial scale was never achieved" seconds after creation (since 1.21) #16694

/area autoscale
/area API

What version of Knative?

1.22.1 (Serving), net-contour 1.22.2, on EKS.

By code inspection the relevant path is unchanged in release-1.23 and main.

Expected Behavior

A revision that is progressing normally should not be marked
ResourcesAvailable=False with reason ProgressDeadlineExceeded and message
Initial scale was never achieved while it is still well inside its progress
deadline. The message asserts that the deadline elapsed without the initial
scale being reached, so it should only appear once that is actually true.

Actual Behavior

A revision is marked failed 2 seconds after being created, with the default
progress-deadline of 600s, and becomes Ready=True ~31s later.

Observed timeline for a single revision (-00003 of a Service updated in place
by an operator):

The revision is healthy today (Ready=True, actualReplicas: 0, it is the
Service's latestReadyRevisionName). The Knative side self-heals.

The problem is that the failure is externally observable while it lasts. Any
controller that watches the Service or the Revision status and treats
ProgressDeadlineExceeded as terminal — which the message invites, since it
claims a deadline expired — records a permanent failure for a deployment that
actually succeeded. In our case that state is written to an external system and
is not reversible there.

Relevant configuration: nothing unusual. progress-deadline is the upstream
default (600s); the only autoscaler override in the cluster is
scale-to-zero-pod-retention-period: 300s. The revisions involved have
autoscaling.knative.dev/min-scale: "0", i.e. they scale to zero.

Steps to Reproduce

We do not have a deterministic reproducer — it is a race, and it fired on 1 of
19 Services updated in the same batch (~11 seconds apart). The shape that
triggered it:

1. A Service with min-scale: 0 that has been scaled to zero for a while.
2. Update the Service in place so a new revision is created.
3. Watch the new revision's conditions from creation.

Occasionally the new revision is marked
ResourcesAvailable=False / ProgressDeadlineExceeded / "Initial scale was never achieved" within a couple of seconds, and recovers shortly after.

Analysis

The message comes from RevisionStatus.PropagateAutoscalerStatus
(pkg/apis/serving/v1/revision_lifecycle.go), in the PodAutoscalerConditionReady == False
branch, guarded by:

if !ps.IsScaleTargetInitialized() && !resUnavailable && ps.ServiceName != "" {
    rs.MarkResourcesAvailableFalse(ReasonProgressDeadlineExceeded,
        "Initial scale was never achieved")
}

That function itself has been stable for a long time — it is byte-identical
from release-1.15 through main.

What did change is where it is called from. In reconcilePA
(pkg/reconciler/revision/reconcile_resources.go):

• up to and including release-1.20, rev.Status.PropagateAutoscalerStatus(&pa.Status)
  was the last statement of the function, operating on the PA returned by
  the Update call (i.e. a server-side object, after the spec reconcile);
• from release-1.21 onward it is the first statement after the lister
  Get, operating on the informer-cached PA, before the spec/annotation
  reconcile.

That move came in with commit 7c317848a (PR #16186, "Allow manual scaling of
the Pod Autoscaler", 2025-11-04). It is present in release-1.21, release-1.22,
release-1.23 and main, and absent in release-1.18, 1.19 and 1.20.

Our hypothesis is that the revision status is now derived from a PA snapshot
that can be stale or mid-transition — a PA that already has a ServiceName but
whose ScaleTargetInitialized has not been observed yet, while Ready is still
False — which satisfies the guard above and produces the spurious failure. We
have not captured the controller's own logs for the event, so the exact
interleaving is inferred from the object timestamps rather than proven; we are
happy to collect debug logs if that would help.

Questions

1. Is propagating the PA status before the PA is reconciled intentional in
   Allow manual scaling of the Pod Autoscaler #16186, or an incidental side effect of that refactor?
2. Should the guard require a stronger signal that the deadline actually
   elapsed (for example, the backing Deployment's ProgressDeadlineExceeded)
   before asserting Initial scale was never achieved? As written, the message
   can be emitted seconds into a revision's life.

Possibly related

• Potential race condition in PodAutoscaler creation #16036 / add default conditions to PA to avoid potential race conditions (2nd attempt) #16078 — "Potential race condition in PodAutoscaler creation",
  fixed by initializing the PA's conditions (merged 2025-10-02, before Allow manual scaling of the Pod Autoscaler #16186).
  Same family, different symptom (revision stuck in Deploying).
• Knative reconcile fails in-middle, but be ready eventually for service creation #10511 — "Knative reconcile fails in-middle, but be ready eventually for
  service creation" (open), which describes the same externally-visible shape.