fix: prevent redelivered run_bot tasks from re-running finished bots (#587)

[yangtheman]

Summary

Follow-up to #897, addressing the trigger behind #587: why a run_bot task runs a second time for an already-ENDED bot.

Root cause

The production settings profiles set CELERY_TASK_ACKS_LATE = True with CELERY_TASK_REJECT_ON_WORKER_LOST = True. Since run_bot runs for the entire meeting, its message stays unacked the whole time and gets redelivered in two scenarios:

1. Worker lost/restarted mid-task — the broker requeues the unacked message; a worker then picks up run_bot for a bot that has meanwhile reached ENDED. This matches the reporter's "restart → recording wiped" sequence.
2. Redis visibility_timeout expiry — the Redis broker's default is 3600s and was not overridden, so any bot running longer than ~1 hour gets a duplicate, concurrent run_bot delivery even without a restart.

Nothing made run_bot idempotent, so the redundant run created a BotController that never recorded and, on shutdown, uploaded an empty file over the completed recording (guarded separately in #897).

Changes

• run_bot aborts immediately if the bot is already in a post-meeting state (BotStates.post_meeting_states()), as suggested in the issue discussion.
• Default the Redis broker visibility_timeout to 21600s, env-overridable via CELERY_BROKER_VISIBILITY_TIMEOUT_SECONDS, merged with the existing CELERY_BROKER_TRANSPORT_OPTIONS env support (env keys win).

The state guard handles redelivery for finished bots; the visibility_timeout bump prevents duplicate concurrent deliveries for still-running bots.

Trade-off: crash-recovery latency for short tasks

visibility_timeout is broker-wide, so raising it to 6h also delays the redelivery safety net for short tasks (deliver_webhook, process_utterance, etc.): if a worker dies hard (OOM-kill/node loss, no graceful shutdown), its unacked messages now wait up to 6h instead of 1h before another worker restores them. In practice the blast radius is small: reject_on_worker_lost=True requeues immediately in the common child-process-death case (visibility_timeout not involved), and prefetch_multiplier=1 means a dead worker holds at most ~1 extra message. All ETA/countdown usage in the codebase is ≤64s, far below either value — raising the timeout actually moves further from the classic Redis+Celery duplicate-ETA-execution failure mode. Operators who prefer the old behavior can set CELERY_BROKER_VISIBILITY_TIMEOUT_SECONDS=3600.

Testing

• New bots/tests/test_run_bot_task.py — skips all post-meeting states, runs otherwise; passing.
• Verified broker_transport_options resolves correctly with and without CELERY_BROKER_TRANSPORT_OPTIONS set.
• Existing test_launch_scheduled_bot_task + test_throttling still passing.
• ruff check and ruff format --check clean.

Docs

No API change — no docs/openapi.yml updates needed.