Skip to content

chore: hibernate_after in many places - #3344

Open
thecristen wants to merge 7 commits into
mainfrom
cbj/genserver-hibernation-wip
Open

chore: hibernate_after in many places#3344
thecristen wants to merge 7 commits into
mainfrom
cbj/genserver-hibernation-wip

Conversation

@thecristen

@thecristen thecristen commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

From debugging our application locally over the past week, I found that

  • most of our memory usage is tied up in processes (this is almost true for prod metrics too, but there we have even more memory attributed to our usage of ETS)
  • many processes maintain a pretty large heap size.

From the GenServer documentation:

Hibernating a GenServer causes garbage collection and leaves a continuous heap that minimises the memory used by the process.

Hibernating should not be used aggressively as too much time could be spent garbage collecting, which would delay the processing of incoming messages. Normally it should only be used when you are not expecting new messages to immediately arrive and minimising the memory of the process is shown to be beneficial.

So in this branch I explored using this strategically.

  • the alerts-fetching processes were already using :hibernate (except after initializing, which I've added here), so I figured it's ok to have the SystemStatus caches, which rely on alerts, to use these too
  • ServiceDateRollover is a ~once a day thing, so I added :hibernate after that message,
  • for a few other places: :hibernate_after will hibernate after a given amount of milliseconds without messages. This felt appropriate in most cases, because most of our GenServers have sporadic activity

Some slightly more involved tweaks here:

  • fixup(SystemStatus.SubwayCache): remove state - since we already have the status saved in the ETS table, it didn't make sense to also save it in the GenServer state
  • fixup(Vehicles.Repo): remove table from state - similar story here, but kept a reference to the table name

Results

Alas, what do the :observer load charts say. The lower left charts depict memory usage pretty soon after application startup.

Before After
image image

There's a noticeable reduction in the baseline memory usage here.

@thecristen thecristen added the dev-green Deploy to dev-green label Jul 17, 2026
@thecristen
thecristen force-pushed the cbj/genserver-hibernation-wip branch from 8cdcf61 to dd09f5f Compare July 27, 2026 19:44
@thecristen thecristen changed the title feat(SystemStatus): use :hibernate, force gc chore: hibernate_after in many places Jul 27, 2026
@thecristen
thecristen marked this pull request as ready for review July 27, 2026 20:28
@thecristen
thecristen requested a review from a team as a code owner July 27, 2026 20:28
@thecristen
thecristen requested a review from lvachon1 July 27, 2026 20:28

@lvachon1 lvachon1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good and works well on my end.

@thecristen thecristen removed the dev-green Deploy to dev-green label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants