catalog: Pass timestamps for transaction commit

[jkosh44]

This commit adds plumbing for the catalog to accept a commit timestamp
for transactions.

In general, it is up to the adapter to provide that timestamp.
Currently, the timestamp that the adapter uses depends on the context.
During startup, when the timestamp oracle isn't always readily
available, the adapter will pass some timestamp that it knows will
be greater than or equal to the current catalog upper. Sometimes this
is a write timestamp from the timestamp oracle, other times it's the
current catalog upper. During all other times, the adapter
passes a write timestamp from the timestamp oracle.

After startup, the adapter applies the catalog upper with the timestamp
oracle to linearize all writes to the catalog that happened during
startup.

When getting a timestamp to allocate IDs in the catalog, the
adapter does not apply the received timestamps. If the adapter did
apply these timestamps, then strict serializable reads on tables might
block if the table upper is not past the applied timestamp.

Catalog transactions are able to re-use the oracle write timestamp
that is already available and used for audit log writes. The adapter
always performs a write to builtin tables immediately after catalog
transactions, which will allocate and apply a new write timestamp. So
effectively catalog transaction write timestamps are applied.

The goal of this PR is to move the catalog closer to the EpochMillis
timeline and set up the necessary infrastructure to pass timestamps
from the adapter to the catalog.

This work doesn't fully move the catalog onto the EpochMillis
timeline, the following work still remains:

• Always apply the commit timestamp with the timestamp oracle after
  committing.
• Add the catalog shard to the txn shard with all other tables.
  This would prevent blocking table reads when applying the commit
  timestamp. Additionally, it would ensure that the catalog is
  readable at the timestamp oracle read timestamp after a table
  write.

An alternative framing of this commit is that it actually does move the
catalog shard onto the EpochMillis timeline, but the catalog shard
exhibits eventual consistency.

Works towards resolving #MaterializeInc/database-issues/8384

Motivation

This PR adds a known-desirable feature.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[jkosh44]

This block, and the ones below, are the exceptions to the "don't apply catalog timestamps" rule. These happens at the end of startup and don't have a negative impact on read latency because we're going to do a bunch of table writes after this which bumps and applies the oracle timestamp.

[shepherdlybot]

Mitigations

Completing required mitigations increases Resilience Coverage.

• (Required) Code Review 🔍 Detected
• (Required) Feature Flag
• (Required) Integration Test
• (Required) Observability
• (Required) QA Review 🔍 Detected
• (Required) Run Nightly Tests
• Unit Test

Risk Summary:

The pull request has a high-risk score of 83, primarily driven by predictors such as the average line count in files and the number of executable lines within files. Historically, PRs with these predictors are 156% more likely to cause a bug than the repo baseline. Additionally, the repository has a steady observed bug trend, and 7 modified files are identified as hotspots with elevated levels of recent bug fixes.

Note: The risk score is not based on semantic analysis but on historical predictors of bug occurrence in the repository. The attributes above were deemed the strongest predictors based on that history. Predictors and the score may change as the PR evolves in code, time, and review activity.

Bug Hotspots:
What's This?

File	Percentile
../durable/upgrade.rs	95
../durable/persist.rs	95
../inner/create_continual_task.rs	96
../src/coord.rs	98
../src/catalog.rs	90
../sequencer/inner.rs	95
../catalog/open.rs	92

[jkosh44]

@def- the RQG tests seem to be consistently failing on Nightly (https://buildkite.com/materialize/nightly/builds/10500#0193553d-f924-4716-b493-c3a809bd6705), but I can't imagine how this PR would impact SELECT statements. Is that a known flake?

[def-]

I think you just got unlucky and hit a bad seed. The difference is a slight rounding difference compared to Postgres, which we are tracking as https://github.com/MaterializeInc/database-issues/issues/8366

[ParkMyCar]

LGTM