docs(compute): RFC for compute rolling restart with prewarm

[ololobus]

Problem

Neon currently implements several features that guarantee high uptime of compute nodes:

1. Storage high-availability (HA), i.e. each tenant shard has a secondary pageserver location, so we can quickly switch over compute to it in case of primary pageserver failure.
2. Fast compute provisioning, i.e. we have a fleet of pre-created empty computes, that are ready to serve workload, so restarting unresponsive compute is very fast.
3. Preemptive NeonVM compute provisioning in case of k8s node unavailability.

This helps us to be well-within the uptime SLO of 99.95% most of the time. Problems begin when we go up to multi-TB workloads and 32-64 CU computes. During restart, compute looses all caches: LFC, shared buffers, file system cache. Depending on the workload, it can take a lot of time to warm up the caches, so that performance could be degraded and might be even unacceptable for certain workloads. The latter means that although current approach works well for small to
medium workloads, we still have to do some additional work to avoid performance degradation after restart of large instances.

Rendered version

Part of https://github.com/neondatabase/cloud/issues/19011

[github-actions]

7964 tests run: 7580 passed, 0 failed, 384 skipped (full report)

---

Code coverage* (full report)

• functions: 32.4% (8732 of 26955 functions)
• lines: 48.4% (74847 of 154546 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
cfaaa34 at 2025-03-19T20:19:46.609Z :recycle:}

[MMeent]

I don't like this. S3 writes are expensive to scale, and if we're about to write every N seconds it's going to be expensive to host read replicas.

I'd expected CPlane-triggered writes; either during shutdown (for warming up after start) or before shutdown (for hot restart); not systematic writes.

[knizhnik]

+1 for CPlane-triggered writes.
Alternatively it can done as part of checkpoint.

[MMeent]

I don't think checkpoint is a good place for this; it's much too frequent on highly loaded systems.

[ololobus]

Well, seconds is just a unit, it can be set to 5, 15 minutes. For cplane-orchestrated there is a separate API, I imagined periodic dumping to be useful for
i) auto-prewarm, i.e. compute periodically dumps LFC content, so later it can be used at restart. In theory, we can only dump at graceful shutdown, but then it won't help with accidental compute crash/restart, as there might be no LFC state to warm up from
ii) later for having a hot standby, i.e. it can periodically fetch fresh content from S3 and do prewarming

I don't want to wire too complex logic via cplane, so TBH, I don't see other options to have a robust auto-prewarm without periodic dumping of the LFC state, pg_prewarm does the same via pg_prewarm.autoprewarm_interval

@MMeent @knizhnik do you have any specific suggestions of how we can implement it without periodic dumping?

[MMeent]

We can make it part of endpoint shutdown procedures?

Note that "i.e. it can periodically fetch fresh content from S3 and do prewarming" won't work as you seem to expect it to, as prewarming is explicitly designed to never evict existing pages from LFC, and thus won't do much if the size of LFC doesn't change for the hot standby. It's prewarming by replacing unused pages with potentially useful pages, and explicitly not LFC state synchronization.

[MMeent]

As for hot standby, it's probably good enough to "just" disable the replay filter and require replay of all WAL (rather than only those records which we already have in our caches)

[ololobus]

Thanks for the comment about hot standby

We can make it part of endpoint shutdown procedures?

Yes, this is what I meant by 'In theory, we can only dump at graceful shutdown'. That'd work in most of the cases, but what I don't like is that it doesn't cover any abnormal termination like OOM, VM restarts, etc.

With your cost estimation, dumping every 5 minutes is completely reasonable

[mtyazici]

While this is happening compute is not able to receive queries right? If so, we shouldn't use this in normal compute starts which are not part of restart logic.

[MMeent]

No, autoprewarm can happen in the background while there are user queries active.

[ololobus]

Yeah, as Matthias mentioned it should be completely async. Furthermore, even any failures during auto-prewarm shouldn't be critical, it's mentioned in the failure modes

[mtyazici]

I see thanks. I think I asked my question incorrectly. While pre-warming the LFC from where it's stored is the compute able to accept queries?

[ololobus]

Yes, it should be fully functional, so auto-prewarm doesn't block anything

[mtyazici]

In case of 500 I assume we are going to suspend the compute and start a new one without pre-warm right?

[ololobus]

Yeah, 500 means a permanent failure, so it's likely unsafe to retry without full restart

[mtyazici]

I see, linked to my question above in that case if there is prewarmed LFC and if we are able to accept queries while prewarming (I doubt it's the case though) we could still achieve computes with prewarmed LFC on start.

[ololobus]

There two separate flows:

1. Auto-prewarm, i.e. just start compute normally, then prewarm it in the background, but clients can connect to it right after start already even if prewarm wasn't finished, so we trade performance for HA

2. Restart via promotion of the already warm replica, this is a preferred way, as clients get guaranteed performance

Yet, if 2. fails, we can always fall back to just 1., so that's what this section is about. Does my comment make it any clearer?

[mtyazici]

What do you mean by proceed with auto-prewarm? Does it mean primary will start without pre-warming but will continue doing the automatic pre-warm. In this case I don't see how we use auto-prewarm in restart code path. Do we have any usage for it outside of this one?

[ololobus]

I mean that we just start primary from scratch with empty caches, the only option to improve the situation is to do async auto-prewarm, while already accepting new connections. So from cplane perspective, it's just a normal start from with auto-prewarm enabled

[mtyazici]

the only option to improve the situation is to do async auto-prewarm

I see, how does this improve the situation if the compute is already starting with empty caches? When would we use the auto-prewarmed LFC if we always try to trigger prewarm on restarts triggered from the cplane?

[ololobus]

Well, that's debatable whether we need auto-prewarm or not at all. It exists in vanilla Postgres. The idea is that we can prewarm caches faster when we do it intentionally vs. when user tries to prewarm by just doing their normal workload

Imagine cplane, it eventually accesses all non-deleted projects/endpoint/branches. If we just restart it at Neon, it will take some time to visit all objects. Yet, if we actively prewarm caches in the brackground, the chance that next project read will hit the cache will be higher, as it will be already there, even though cplane hasn't read it explicitly

In practice, it may not suite all workloads, but we cannot answer for sure until we implement it and battle-test, but imo it exists in vanilla Postgres for a reason, so there are use-cases

[MMeent]

This concurrency control, does that cover both/all lfc endpoints, or just this one?

[ololobus]

I think it makes sense only for blocking calls /promote and /store_lfc_state. For other methods, it should be pretty safe to do parallel requests, although I don't expect that cplane will do that intentionally

[MMeent]

Shouldn't this be more along the lines of the following?

Suggested change

	pub enum PrewarmStatus {
	/// Prewarming was never requested on this compute
	Off,
	/// Prewarming was requested, but not started yet
	Pending,
	/// Prewarming is in progress. The caller should follow
	/// `PrewarmState::progress`.
	InProgress,
	/// Prewarming has been successfully completed
	Completed,
	/// Prewarming failed. The caller should look at
	/// `PrewarmState::error` for the reason.
	Failed,
	/// It is intended to be used by auto-prewarm if none of
	/// the previous LFC states is available in S3.
	/// This is a distinct state from the `Failed` because
	/// technically it's not a failure and could happen if
	/// compute was restart before it dumped anything into S3,
	/// or just after the initial rollout of the feature.
	Skipped,
	}
	pub enum PrewarmStatus {
	/// Prewarming was never requested on this compute
	Off,
	/// Prewarming was requested, but not started yet
	Pending { segments: N },
	/// Prewarming is in progress. The caller should follow
	/// `PrewarmState::progress`.
	InProgress { segments: N, done: N },
	/// Prewarming has been successfully completed
	Completed { segments: N },
	/// Prewarming failed. The caller should look at
	/// `PrewarmState::error` for the reason.
	Failed { error: Option<String> },
	/// It is intended to be used by auto-prewarm if none of
	/// the previous LFC states is available in S3.
	/// This is a distinct state from the `Failed` because
	/// technically it's not a failure and could happen if
	/// compute was restart before it dumped anything into S3,
	/// or just after the initial rollout of the feature.
	Skipped,
	}

[ololobus]

It could be, yeah, it's just such non-uniformly-typed enums have very-very bad representation in OpenAPI spec and json generally, so I'd prefer to split the status from any aux info. We can have some custom serializer, but that's more work for no good reason

I'm afraid that it will be harder to consume such response from cplane, wdyt @mtyazici?

[mtyazici]

If we are able to present it in the spec I think generators would handle them alright. But I think instead of using union types we could have a status object that contains all mentioned fields in the suggestion.

type PrewarmStatus struct {
Status string
Segments int64
DoneSegments int64
Error string
}

[MMeent]

This is missing the important element where the Primary that has to be shut down first, but the (old) primary does not show up in this diagram.

[ololobus]

Primary in the diagram is the old primary, and it's shut down first, it's also mentioned in text. Or what do you mean?

[MMeent]

Oh, it didn't have the same actor labeling as those of the secondary, so I'd failed to notice this.

Either way, that needs additional details, as there are some compute_ctl-postgres interactions which we need to detail on the primary as well for this to work correctly and consistently.

[ololobus]

My intent was to keep it reasonably high-level. Do you see some important interactions missing here?

[MMeent]

Not "missing" per se, but I do find it confusing that the RFC does go into detail for one side (the promoting replica) but not the other (the primary), while both sides are critial for correct functioning of the system. I'd expected both sides to have the same amount of detail.

[MMeent]

We can wait for PostgreSQL to become primary before starting streaming, I don't think it needs to be all that difficult.

[VladLazar]

We terminate the old primary before promoting the secondary. If I'm reading this right, there's a an availability gap between the termination and promotion. Is this reading correct? If so, can we avoid it?

[MMeent]

If I'm reading this right, there's a an availability gap between the termination and promotion

The secondary could be made available for read-only queries before the primary shuts down. However, we can not allow write queries to the secondary before the primary has shut down, so there will always be a gap where there is no writer available.

The system should be fast enough, however, that this gap is no longer than a second at most; and probably much less.

[VladLazar]

It would be great to aim for something where the happy path has no hiccups. If that's not possible, let's be explicit about it and say why.

---

Proxy learns about about the new primary after the promotion of the new compute, so there should be no queries on it at that point (according to the diagram). It seems like we can swap promotion and termination around while maintaining this property.

What would happen to the client in this case? Let's say we have an in-progress query when the old primary terminates. Client would probably retry and eventually get routed to the new primary.

[MMeent]

It seems like we can swap promotion and termination around while maintaining this property.

No, Vlad, we can not do that.

Two computes can Never, NEVER, never both be Primary on the same timeline at the same time. CPlane is supposed to make sure of that, and Compute will fail if it doesn't. Promotion of the replica before the original Primary shut down will cause errors, panics, data loss, shutdowns, and/or nasal deamons on the Primary, this promoting Secondary, or both.

We don't want to test that theory.

What would happen to the client in this case? Let's say we have an in-progress query when the old primary terminates.

Their session would and should get disconnected. How that client handles disconnections is not something we care about here.

[VladLazar]

Two computes can Never, NEVER, never both be Primary on the same timeline at the same time. CPlane is supposed to make sure of that, and Compute will fail if it doesn't. Promotion of the replica before the original Primary shut down will cause errors, panics, data loss, shutdowns, and/or nasal deamons on the Primary, this promoting Secondary, or both.

At the risk of being annoying: why? I understand it doesn't work in the case when two primaries are being written to. But what about if we could guarantee the new primary is idle? More specifically, by idle I mean it would have to not write any WAL and be in some sort of consensus observer role.

I'm sure you're right about this working with the current state of afairs, but I'm curious about what the technical roadblock is.

[knizhnik]

Postgres can write WAL not only as result of some query execution.
There many background activities which cause writing WAL: checkpoint, vacuum,...

[MMeent]

More specifically, by idle I mean it would have to not write any WAL and be in some sort of consensus observer role.

That's what a hot standby is for PostgreSQL - the hot standby is not allowed to write WAL, but could stop reading and applying WAL from the primary and start writing its own WAL at a moment's notice.

The issue is that you can't promote until you've replayed all acknowledged WAL from safekeepers, because otherwise you'd fail to replay commits that the primary may have already sent acknowledgements for to its clients, essentially losing commits.

[VladLazar]

The compute is available during pre-warm, correct? Wondering if there's any interactions between the user workload and prewarm that are worth considering.

[mtyazici]

Yes, I was wondering this as well. Will we try to pre-warm computes when not restarting a compute; for example compute start due to a proxy connection.

If compute is not available during pre-warm the answer is definitely no but if it's available then the LFC cache might have become stale over time.

[ololobus]

Replied here #11294 (comment) and here #11294 (comment) as well

Wondering if there's any interactions between the user workload and prewarm that are worth considering.
if it's available then the LFC cache might have become stale over time.

This PR #10442 introduced additional locking when accessing LFC, so it's now considered safe to write there concurrently, so that's the base for all this work.

During prefetch, we always request the latest pages from the pageserver. If, after loading page gets modified, then it will be either updated in LFC (in case of primary) or evicted from LFC after receiving the corresponding WAL record (in case of replica). In other words, if pages is not present in the LFC, we will fetch it from the pageserver; if someone (backend, normal client workload) tries to write it concurrently, then the access will be synchronized, and we should still get a freshness guarantee. @knizhnik or @MMeent can correct me, as I'm not fluent in the underlying mechanism, I consider it as given here

Thus, it should be safe to prewarm LFC concurrently with user load. The only problem is performance, I wrote about it in other comments, but anyway. Yes, if it's highly intensive workload, then prewarm can compete for storage resources with user workload, so we can consider auto-prewarm to be user-togglable feature, I wrote about it in the section about auto-prewarm concerns

@VladLazar @mtyazici let me know if it makes it clearer

[MMeent]

or evicted from LFC after receiving the corresponding WAL record (in case of replica).

This is incorrect: WAL will be replayed for every page that's currently in the LFC or shared buffers.

Wondering if there's any interactions between the user workload and prewarm that are worth considering.
if it's available then the LFC cache might have become stale over time.

The LFC can become stale, but only if the main page is still in shared buffers. The (modified) page from buffers will at some point be written, which will make the LFC lose its stale-ness.

In any case, the stale-ness of a page in LFC doesn't (shouldn't) matter for this RFC.

[conradludgate]

Question: Is it possible to make the old primary close the connections instead?

It's easy enough to prevent proxy from starting a new connection (assuming at-least-once-delivery) but closing existing connections is not so easy at the moment, and with the upcoming pglb work this likely would not be practical. It's not a major blocker, but I want to explore the options.

[ololobus]

This step is right after we terminate the primary, so yes, during normal termination, we can expect that at this moment primary will be already terminated and all connections to it will be closed.

I added this item after talking to Stas, as he had a fair point that the old primary could be unresponsive during this promotion flow, so we will send termination and k8s resources deletion requests, but we cannot generally guarantee that it will be dead by this time. So this step is more to protect from the situation, when old connections will still be connected to the old primary

See also item 7 in failure modes. I'm not quite sure how big the problem is. Safekeepers will guarantee that there is only one running writer at a time, so it's more like a nice-to-have, than must-have feature, just to prevent unnecessary interference and side effects (I worried about some stale reads from the old primary and failing writes because safekeepers should reject them)

[myrrc]

Suggested change

	## Authentication and authorization
	1. Control plane should generate a JWT token which would be used by compute for authorizing requests
	to S3 proxy. This token should be passed in `/compute/api/v2/computes/{id}/spec` route,
	parsed by `compute_ctl` as an optional field (to preserve backward compatibility) and passed as-is
	to proxy requests during prewarm or prewarm offload. If no token is passed, requests to proxy
	should return 505 Not Supported (other services have `--dev` flags which disable token check or
	config fields which bypass authentication when absent but that's error-prone) and log the error.
	2. S3 proxy should have `auth_public_key.pem` similar to pageserver
	for spawning an https server for compute requests (this would also further help in getting
	PCI-DSS certification) and verifying compute requests. The pemfile should be supplied by
	infra/. As with pageserver, S3 proxy should have `/reload_auth_validation_keys` endpoint
	to reload the pemfile from config should it change.
	## Metrics
	In addition to changing `compute_ctl`'s /status, proxy should provide request duration
	metrics along with result server codes as labels

[ololobus]

@myrrc looks mostly good to me, thanks, I only have minor comments. I suggest we put it into other PR -- #9661 as it belongs to the unlogged storage/S3 proxy RFC, not just to prewarm flow specifically

cc @MMeent

[myrrc]

I believe Matthias's RFC focuses more on the high-level overview, but ok, I'll copy the comments. Btw this will probably be split into tasks for https://github.com/neondatabase/cloud/issues/24225 (see sub-issues)

[Bodobolero]

Do we have a calculaton on the network bandwidth needed on the compute k8s cluster for a set of (large) tenants with LFCs in the order of hundreds of GB either storing or retrieving their LFC state from S3?
Do we have enough spare network capacity?
If the pages loaded are actually used it wouldn‘t be additional network traffic because we save the getpage request to pageserver. If they are restored and then not used (evicted) it would be additional network traffic.
S3 supports parallelism when retrieving from object storage. What are the plans for parallelism when restoring LFC. Will this depend on the compute size and or pricing plan?

[myrrc]

S3 supports parallelism when retrieving from object storage. What are the plans for parallelism when restoring LFC. Will this depend on the compute size and or pricing plan?

As for the compute_ctl->proxy side, it would be regular async streaming file download/upload. Regarding network capacity, most of clients will use general proxies (one per region), however, with some big ones with large CUs we'll probably have dedicated proxy instances running on the compute nodes (not confirmed as I need to test performance implications).

[Bodobolero]

Regarding network capacity, most of clients will use general proxie

The question was regarding capacity of 25 Gib/s NIC at physical hardware of k8s node running computes.