Skip to content

[codex] Fix Kubernetes config watch reloads#633

Merged
winhowes merged 2 commits into
mainfrom
codex/k8s-config-watch
Apr 24, 2026
Merged

[codex] Fix Kubernetes config watch reloads#633
winhowes merged 2 commits into
mainfrom
codex/k8s-config-watch

Conversation

@winhowes
Copy link
Copy Markdown
Owner

@winhowes winhowes commented Apr 23, 2026

Summary

  • Watch containing directories instead of individual config files so Kubernetes projected-volume ..data symlink swaps trigger hot reloads.
  • Add trailing debounce for watcher events so atomic replacement bursts produce one reload after the volume settles.
  • Update the Helm chart to mount the ConfigMap at /conf instead of using subPath, pass /conf/*.yaml explicitly, and enable -watch by default.
  • Add targeted watcher tests for debounce behavior, ignored events, closed watcher channels, path-resolution failures, full notification channels, and Kubernetes-style symlink swaps.

Root Cause

Kubernetes updates projected ConfigMap and Secret volumes by swapping symlinks in the parent directory. A watch on the resolved file can miss that update. The chart also used subPath mounts, and Kubernetes does not update subPath-mounted ConfigMap files after pod startup.

Validation

  • go test ./... passed.
  • go test ./app -run 'TestWatchFiles|TestWatchEventInDirs|TestWatchDebouncer|TestMainWatchReload' -coverprofile=/tmp/authtranslator-watch.cover -count=1 passed.
  • go tool cover -func=/tmp/authtranslator-watch.cover reports watchFiles, watchEventInDirs, and newWatchDebouncer at 100.0%.
  • git diff --check passed.
  • helm template authtranslator charts/authtranslator could not be run because helm is not installed in this environment.

@winhowes winhowes marked this pull request as ready for review April 23, 2026 23:29
@winhowes winhowes marked this pull request as draft April 23, 2026 23:36
@winhowes winhowes marked this pull request as ready for review April 24, 2026 00:38
@winhowes winhowes merged commit 021b3ab into main Apr 24, 2026
3 checks passed
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a605b5fcf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread app/main.go
Comment on lines +1246 to +1247
if watchEventInDirs(ev, watchedDirs) && ev.Op&(fsnotify.Write|fsnotify.Create|fsnotify.Rename|fsnotify.Remove|fsnotify.Chmod) != 0 {
timerC = debouncer.trigger()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict reload triggers to configured file paths

The new filter treats any event in a watched directory as relevant, so changing unrelated sibling files now triggers config reloads even when config.yaml/allowlist.yaml/denylist.yaml are untouched. This is a behavioral regression from file-level watching and can cause noisy reload loops in directories like /etc or /tmp; with frequent unrelated writes, the debounce timer can keep resetting and delay real config reloads.

Useful? React with 👍 / 👎.

Comment thread app/main.go
Comment on lines +1218 to +1220
for dir := range watchedDirs {
if err := w.Add(dir); err != nil {
logger.Error("watch add failed", "dir", dir, "error", err)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep watching symlinked config files themselves

Switching from w.Add(file) to w.Add(dir) drops reload notifications when a configured file path is a symlink whose target lives outside the symlink’s parent directory. In that setup, writes update the target inode but do not emit directory-entry events in the symlink directory, so -watch no longer reloads unless the symlink entry itself changes.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant