Bound Flink sink retries by time, not attempt count

[karthikk-01]

Problem

Production Flink jobs restart on transient UNAVAILABLE during query-server rollouts:

io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset
  before headers ... delayed connect error: Connection refused
  at core.ChalkFeatureUploader.uploadWithRetry(ChalkFeatureUploader.java:105)
Wrapped by: ChalkUploadException: upload_features failed after 4 attempt(s)

The retries worked — after 4 attempt(s) is maxRetries=3 + 1, and UNAVAILABLE is retryable. The budget was just too short. With the default 500ms base backoff:

attempt 0 fails -> sleep 0.5s
attempt 1 fails -> sleep 1.0s
attempt 2 fails -> sleep 2.0s
attempt 3 fails -> give up      total coverage ~3.5s

Connection refused fails fast rather than hanging to the 30s upload timeout, so the real coverage is ~3.5s — shorter than a routine deploy. Channel-level gRPC retry doesn't help: maxAttempts=3 with backoff capped at 0.1s adds ~0.16s.

Change

An attempt count can't express "survive a rollout" — the real budget silently shifts whenever the backoff is tuned. Replaced it with an explicit wall-clock budget:

• retryTimeout (new, default 120s) — total budget for retrying one batch's transient failures. Sized to outlast a normal query-server rollout.
• maxRetries — now an opt-in hard cap, unlimited by default, so retryTimeout is the effective bound. Setting it explicitly preserves the old exact-attempt semantics.
• Timing uses nanoTime so an NTP step can't move the deadline, and the loop breaks when the next sleep would overrun the budget instead of sleeping past it.

Retries still run inline on the task thread, so the budget is also the worst-case subtask stall — documented next to the checkpoint-timeout interaction.

Also included

1cd1803 — the sources/javadoc fix for the chalk-java-shaded publication, currently only on release/1.3.3 (#205). Without it on main, cutting 1.3.4 fails Central validation exactly as 1.3.3 did: the incomplete shaded component sinks the whole deployment bundle, taking chalk-java with it, while CI still reports green. Harmless duplicate if #205 merges first.

Tests

New: stopsRetryingWhenTimeBudgetSpent, timeBudgetAllowsMoreAttemptsThanTheOldFixedCap, explicitMaxRetriesStillCapsBeforeBudget, retryBudgetDefaultsToTimeNotAttempts. All pre-existing retry tests pass unchanged.

The 4 initializationError failures in TestChalkClient* / TestGrpcClient / TestAllClients are pre-existing credential-dependent integration tests — they fail identically on a clean main locally.

Behaviour note

An outage longer than the budget still surfaces as a task failure and a checkpoint restart. That is at-least-once working as intended, not data loss — this change makes the common case (a rollout) stop causing restarts at all.