Support auto scaling - #171
Merged
Merged
Conversation
… into feat/auto-scaling
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This reverts commit a16dfc1.
…tedSocketRepository from package entry point - Add exports for JobRepository, RecordSocketAssociationRepository, and ConnectedSocketRepository to the main package entry point - Add type export for JobRow to support job-queue-persistence and record-concurrency dependent tasks Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Postgres-backed primitives to make @openforis/arena-server safe to run on multiple Heroku dynos (horizontal auto-scaling), by moving previously in-memory, per-process coordination (WebSocket delivery/presence, record↔socket associations, job state/progress) into shared DB state and a lightweight LISTEN/NOTIFY “cluster bus”.
Changes:
- Introduces a Postgres LISTEN/NOTIFY-based
ClusterBuswith spillover storage (ws_relay_message) and advisory-lock helper for cluster-wide deduped maintenance. - Makes WebSocket notification delivery cluster-aware via
ClusterBusand a sharedconnected_socketpresence table (+ heartbeats + TTL pruning). - Adds DB-backed repositories and migrations for job tracking and record↔socket associations to support cross-dyno coordination.
Reviewed changes
Copilot reviewed 62 out of 63 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/webSocket/server.ts | Cluster-aware socket delivery (broadcast on miss), presence persistence, heartbeats + TTL sweep, graceful shutdown hook. |
| src/server/arenaServer/stop.ts | Calls WebSocketServer.shutdown() during graceful shutdown. |
| src/repository/wsRelayMessage/tests/wsRelayMessage.test.ts | Tests for spillover message table repository. |
| src/repository/wsRelayMessage/insert.ts | Inserts relay payload row and returns id. |
| src/repository/wsRelayMessage/index.ts | Exposes WsRelayMessageRepository API. |
| src/repository/wsRelayMessage/getById.ts | Fetches relay payload by id. |
| src/repository/wsRelayMessage/deleteExpired.ts | TTL-based pruning for relay rows. |
| src/repository/recordSocketAssociation/tests/testUtils.ts | Test helpers for users and connected_socket rows. |
| src/repository/recordSocketAssociation/tests/recordSocketAssociation.test.ts | Tests for record↔socket association persistence and cascade behavior. |
| src/repository/recordSocketAssociation/index.ts | Exposes RecordSocketAssociationRepository API. |
| src/repository/recordSocketAssociation/getSocketIdsByRecordUuid.ts | Reads socket ids for a record (cluster-wide). |
| src/repository/recordSocketAssociation/dissocSocketsByRecordUuid.ts | Deletes all associations for a record. |
| src/repository/recordSocketAssociation/dissocSocketBySocketId.ts | Deletes all associations for a socket. |
| src/repository/recordSocketAssociation/dissocSocket.ts | Deletes a single record/socket association. |
| src/repository/recordSocketAssociation/assocSocket.ts | Inserts record/socket association idempotently. |
| src/repository/job/utils.ts | Row type + DB→domain transform helpers for jobs. |
| src/repository/job/updateStatus.ts | Updates job status and optionally merges props. |
| src/repository/job/updateProgress.ts | Updates processed/total counters. |
| src/repository/job/tests/testUtils.ts | Job test helpers for user/survey setup. |
| src/repository/job/tests/job.test.ts | Job repository behavior tests (insert/get/active/progress/status). |
| src/repository/job/insert.ts | Inserts a pending job row. |
| src/repository/job/index.ts | Exposes JobRepository and JobRow type. |
| src/repository/job/getByUuid.ts | Fetches a job by uuid (cluster-wide polling). |
| src/repository/job/getActiveByUserUuid.ts | Fetches active job for a user. |
| src/repository/job/getActiveBySurveyId.ts | Fetches active job for a survey. |
| src/repository/index.ts | Exports new repositories/types (connectedSocket/job/recordSocketAssociation/wsRelayMessage). |
| src/repository/connectedSocket/upsert.ts | Inserts/refreshes a connected socket presence row. |
| src/repository/connectedSocket/touchMany.ts | Heartbeat refresh for a set of socket ids. |
| src/repository/connectedSocket/tests/testUtils.ts | Test helpers for connected socket tests. |
| src/repository/connectedSocket/tests/connectedSocket.test.ts | Presence table repository tests (upsert/exists/remove/touch/deleteStale). |
| src/repository/connectedSocket/removeMany.ts | Bulk delete of presence rows (shutdown cleanup). |
| src/repository/connectedSocket/remove.ts | Deletes a presence row by socket id. |
| src/repository/connectedSocket/index.ts | Exposes ConnectedSocketRepository API. |
| src/repository/connectedSocket/exists.ts | Cluster-wide “is socket connected” check via DB. |
| src/repository/connectedSocket/deleteStale.ts | TTL prune for stale presence rows. |
| src/processEnv/index.ts | Adds stable instanceId derived from DYNO/HOSTNAME/hostname(). |
| src/log/logFileS3Upload.ts | Uploads logs under per-instance prefix to avoid collisions. |
| src/job/index.ts | Exports JobConstructor type. |
| src/index.ts | Public exports for ClusterBus, runWithClusterLock, and new repositories/types. |
| src/db/table/schemaPublic/wsRelayMessage.ts | Table definition for ws relay messages. |
| src/db/table/schemaPublic/recordSocketAssociation.ts | Table definition for record↔socket associations. |
| src/db/table/schemaPublic/job.ts | Table definition for jobs. |
| src/db/table/schemaPublic/index.ts | Exports new schemaPublic table definitions. |
| src/db/table/schemaPublic/connectedSocket.ts | Table definition for connected socket presence. |
| src/db/table/index.ts | Re-exports new public tables from db/table. |
| src/db/index.ts | Re-exports new tables from db/index. |
| src/db/dbMigrator/migration/public/migrations/sqls/20260819100100-create-table-record-socket-association-up.sql | Creates record_socket_association table + indexes + FK cascade. |
| src/db/dbMigrator/migration/public/migrations/sqls/20260819100100-create-table-record-socket-association-down.sql | Drops record_socket_association table. |
| src/db/dbMigrator/migration/public/migrations/sqls/20260819100000-create-table-job-up.sql | Creates job table + indexes + FKs. |
| src/db/dbMigrator/migration/public/migrations/sqls/20260819100000-create-table-job-down.sql | Drops job table. |
| src/db/dbMigrator/migration/public/migrations/sqls/20260806120000-add-tables-cluster-bus-up.sql | Creates connected_socket + ws_relay_message tables + indexes. |
| src/db/dbMigrator/migration/public/migrations/sqls/20260806120000-add-tables-cluster-bus-down.sql | Drops cluster-bus related indexes/tables. |
| src/db/dbMigrator/migration/public/migrations/20260819100100-create-table-record-socket-association.js | Migration runner for record_socket_association SQL. |
| src/db/dbMigrator/migration/public/migrations/20260819100000-create-table-job.js | Migration runner for job SQL. |
| src/db/dbMigrator/migration/public/migrations/20260806120000-add-tables-cluster-bus.js | Migration runner for connected_socket/ws_relay_message SQL. |
| src/clusterBus/types.ts | Defines generic ClusterEvent envelope types. |
| src/clusterBus/tests/clusterBus.test.ts | Tests for ClusterBus publish/spillover and advisory-lock helper. |
| src/clusterBus/index.ts | Exports cluster-bus APIs and types. |
| src/clusterBus/clusterLock.ts | Advisory-lock helper (runWithClusterLock). |
| src/clusterBus/clusterBus.ts | Postgres-backed cluster bus implementation with spillover and TTL sweep. |
| docs/superpowers/specs/2026-08-06-heroku-horizontal-autoscaling-design.md | Design spec documenting overall multi-dyno strategy and constraints. |
| .env.template | Documents per-instance S3 log prefix behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.