Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

MadLittleMods · 2025-08-29T00:34:40Z

Implements MSC3871: Gappy timelines

Add the gaps field to the /messages endpoint

Complement tests: matrix-org/complement#801

Motivating use case

For the hack day, I'll be tackling the backfill problem with this strategy:

Default to fast responses with gaps: As a default, we can always respond quickly and indicate gaps (MSC3871) for clients to paginate at their leisure.

Fast back-pagination: Clients back-paginate with /messages?dir=b&backfill=false, and Synapse skips backfilling entirely, returning only local history with gaps as necessary.

Explicit gap filling: To fill in gaps, clients use /messages?dir=b&backfill=true which works just like today to do a best effort backfill.

This allows the client to back-paginate the history we already have without delay. And can fill in the gaps as they see fit.

Gaps can be represented in the UI with a message like "We failed to get some messages in this gap, try again 🗘.", giving users clear feedback. Regardless of clients trying to fill in gaps automatically, I would still suggest to display gaps so people can tell what's happening.

This is basically a simplified version of MSC4282 leveraging MSC3871: Gappy timelines to get proper client feedback to indicate where the gaps are so we can skip backfill without worrying. For reference, skipping backfill without letting clients know where the gaps are just means they won't ever know that they are missing messages.

-- @MadLittleMods, https://github.com/element-hq/how-hard-can-it-be-2025/issues/47#issuecomment-3234339497

Testing strategy

Clone synapse
Checkout the correct branch: git checkout madlittlemods/msc3871-gappy-timeline (this PR)
Install Synapse's dependencies: poetry install
Clone complement next to synapse as a sibling
Checkout the correct branch: git checkout madlittlemods/msc3871-gappy-timeline (Tests for MSC3871: Gappy timelines matrix-org/complement#801)

In order to manually test with a real Matrix client:

In complement, uncomment the sleep call in TestRoomMessagesGaps
Start the Complement tests: COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestRoomMessagesGaps
Find which port hs3 is running on: docker ps -f name=complement_ (ex. 0.0.0.0:33413->8008/tcp)
Configure your Matrix client with that homeserver URL: http://localhost:33413
Register a new user
Join the room from the test (in the log output)
Back paginate in the room: /messages?dir=b&backfill=false and notice the gaps in the room
To fill in gaps: /messages?dir=b&backfill=true&from=<prev_pagination_token>

Reference: Complement docs on how to hook up a Matrix client

Dev notes

Document how to hook up Element to the resultant homeservers from Complement: matrix-org/complement#164

matrix-org/complement -> ONBOARDING.md -> How do I hook up a Matrix client like Element to the homeservers spun up by Complement after a test runs?

Todo

Put the behavior behind a experimental feature flag
Add test_gaps_going_forwards
Add some Complement tests
Separate out the /messages?backfill=true/false changes (simplified version of MSC4282)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

See [MSC: Gappy timelines](matrix-org/matrix-spec-proposals#3871)

To try out the flow: - **Default to fast responses with gaps**: As a default, we can always respond quickly and indicate gaps ([MSC3871] (matrix-org/matrix-spec-proposals#3871)) for clients to paginate at their leisure. - **Fast back-pagination**: Clients back-paginate with `/messages?dir=b&backfill=false`, and Synapse skips backfilling entirely, returning only local history with gaps as necessary. - **Explicit gap filling**: To fill in gaps, clients use `/messages?dir=b&backfill=true` which works just like today to do a best effort backfill. This allows the client to back-paginate the history we already have without delay. And can fill in the gaps as they see fit. This is basically a simplified version of [MSC4282] (matrix-org/matrix-spec-proposals#4282).

MadLittleMods · 2025-08-29T01:11:14Z

synapse/types/__init__.py

-    def get_stream_pos_for_instance(self, instance_name: str) -> int:
-        """Get the stream position that the given writer was at at this token.
+    def is_before_or_eq(self, other_token: Self) -> bool:
+        is_before_or_eq_stream_ordering = super().is_before_or_eq(other_token)
+        if not is_before_or_eq_stream_ordering:
+            return False

-        This only makes sense for "live" tokens that may have a vector clock
-        component, and so asserts that this is a "live" token.
-        """
-        assert self.topological is None


Removed get_stream_pos_for_instance because there is no need to have this specialized version that only allows stream-based tokens. We will fallback to the super() version which is the same but without the topological restriction.

While only "live" tokens may have a vector clock component, historical topological tokens still include a stream position (t426-2633508) and it makes sense for this to still work.

synapse/synapse/types/__init__.py

Lines 697 to 703 in 68068de

Historic tokens start with a "t" followed by the `depth`

(`topological_ordering` in the event graph) of the event that comes before

the position of the token, followed by "-", followed by the

`stream_ordering` of the event that comes before the position of the token.

An example token is:

t426-2633508

…mate comparisons

MadLittleMods · 2025-09-05T15:25:43Z

synapse/storage/databases/main/event_federation.py

+        It's best to pad the `current_depth` by the number of messages you plan to
+        backfill from these points.


This is a good useful change that we could ship outside of this PR.

Noting in case this PR goes stale

MadLittleMods · 2025-09-05T15:26:08Z

synapse/storage/databases/main/events_worker.py

+# from synapse.storage.databases.main.stream import (
+#     generate_next_token,
+# )


Suggested change

# from synapse.storage.databases.main.stream import (

# generate_next_token,

# )

MadLittleMods · 2025-09-05T15:27:19Z

synapse/storage/databases/main/events_worker.py

+            ignore_gap_after_latest: Whether the gap after the latest events (forward
+                extremeties) in the room should be considered as an actual gap.


Per matrix-org/matrix-spec-proposals#3871 (comment), we should revert the ignore_gap_after_latest change.

The default of "omit the gap after the latest messages in the room" is the correct choice

MadLittleMods added 2 commits August 28, 2025 16:58

First stab at gaps in /messages

a14808e

See [MSC: Gappy timelines](matrix-org/matrix-spec-proposals#3871)

Add test_gaps_going_backwards

50d8337

MadLittleMods added the A-Messages-Endpoint label Aug 29, 2025

MadLittleMods added 2 commits August 28, 2025 19:38

Add changelog

d353cfc

MadLittleMods commented Aug 29, 2025

View reviewed changes

Picking backfill points: current_depth should be padded for approxi…

e835134

…mate comparisons

MadLittleMods mentioned this pull request Sep 2, 2025

Tests for MSC3871: Gappy timelines matrix-org/complement#801

Draft

2 tasks

MadLittleMods commented Sep 5, 2025

View reviewed changes

This was referenced Sep 24, 2025

MSC4282: Interactive /rooms/ROOM_ID/messages (Client-Server API) matrix-org/matrix-spec-proposals#4282

Open

MSC3871: Gappy timelines matrix-org/matrix-spec-proposals#3871

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

Uh oh!

MadLittleMods commented Aug 29, 2025 •

edited

Loading

Uh oh!

MadLittleMods Aug 29, 2025 •

edited

Loading

Uh oh!

MadLittleMods Sep 5, 2025

Uh oh!

MadLittleMods Sep 5, 2025

Uh oh!

MadLittleMods Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	Historic tokens start with a "t" followed by the `depth`
	(`topological_ordering` in the event graph) of the event that comes before
	the position of the token, followed by "-", followed by the
	`stream_ordering` of the event that comes before the position of the token.
	An example token is:

	t426-2633508

		It's best to pad the `current_depth` by the number of messages you plan to
		backfill from these points.

	# from synapse.storage.databases.main.stream import (
	# generate_next_token,
	# )

		ignore_gap_after_latest: Whether the gap after the latest events (forward
		extremeties) in the room should be considered as an actual gap.

Implement MSC3871: Gappy timelines - indicate gaps in /messages #18873

Are you sure you want to change the base?

Implement MSC3871: Gappy timelines - indicate gaps in /messages #18873

Uh oh!

Conversation

MadLittleMods commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivating use case

Testing strategy

Dev notes

Todo

Pull Request Checklist

Uh oh!

MadLittleMods Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

MadLittleMods commented Aug 29, 2025 •

edited

Loading

MadLittleMods Aug 29, 2025 •

edited

Loading