Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

TommyBrosman · 2025-05-12T22:02:29Z

Adds a new Persisted Schema Format with persisted metadata support. This change does not enable it by default.

Changes

Add schema FormatV2
Register FormatV2 with schema codec
Update codec.spec to cover FormatV2 as well as FormatV1
Add schema v2 snapshots
Persisted metadata object in schema
FormatV2-specific tests
Documentation

Notes for reviewers

Todo

Changeset description
simple-tree changes to expose persisted metadata
Tests for simple-tree layer
Clone and freeze the metadata trait passed to simple-tree

Related PRs

Previous PR updating the schema and schema change codecs: Shared Tree: Dispatching schema codec #24479
Refactor: Rename CodecFormatVersion to CodecVersion: Shared Tree: Rename CodecFormatVersion to CodecVersion #24666
Feature flag to turn on persisted metadata: (future PR)

- Added in-memory schema format v2. - Implemented the schema format v2 codec. - Updated test utils that previously assumed there was only one format. - Added snapshots for many of the v2 tests.

packages/dds/tree/src/codec/codec.ts

packages/dds/tree/src/feature-libraries/schema-index/codec.ts

…ion as a parameter. - Fixed naming of schema index formats. - Changed schemaChangeFormat to be schema format-agnostic.

…formats. - Moved SchemaCodecVersion.

packages/dds/tree/src/core/schema-stored/formatV2.ts

github-actions · 2025-05-13T19:57:08Z

🔗 No broken links found! ✅

Your attention to detail is admirable.

linkcheck output


> [email protected] ci:check-links /home/runner/work/FluidFramework/FluidFramework/docs
> start-server-and-test "npm run serve -- --no-open" 3000 check-links

1: starting server using command "npm run serve -- --no-open"
and when url "[ 'http://127.0.0.1:3000' ]" is responding with HTTP status code 200
running tests using command "npm run check-links"


> [email protected] serve
> docusaurus serve --no-open

[SUCCESS] Serving "build" directory at: http://localhost:3000/

> [email protected] check-links
> linkcheck http://localhost:3000 --skip-file skipped-urls.txt

Crawling...

Stats:
  210260 links
    1663 destination URLs
    1895 URLs ignored
       0 warnings
       0 errors

packages/dds/tree/src/core/schema-stored/formatV2.ts

Co-authored-by: Alex Villarreal <[email protected]>

- Made importCompressed always use a v1 schema when encoding.

packages/dds/tree/src/feature-libraries/schema-index/codec.ts

packages/dds/tree/src/test/feature-libraries/schema-index/codecUtil.ts

- Added better test data to one of the codec tests for the v2 schema format.

…added in another PR. - Updated a comment.

packages/dds/tree/src/feature-libraries/schema-index/formatV2.ts

CraigMacomber · 2025-05-20T18:10:12Z

In the context of SharedTree, "Schema" without any specific qualifiers (meaning not "Stored Schema" or "Persisted Schema") refers to the "View Schema" the user facing API and objects for defining schema (TreeNodeSchema, SchemaFactory etc).

As this PR makes not changes at that layer, I don't think "Schema v2 with persisted metadata support" is a great name for this.

I think "Persisted Schema Format v2 with persisted metadata support" would be better.

Similar changes should be made to the PR description as well.

CraigMacomber · 2025-05-20T18:13:51Z

This PR does not meet the criteria in https://github.com/microsoft/FluidFramework/wiki/PR-Guidelines#guidelines (also linked from the default PR description via the template) to be marked as non-draft.

I think the status of this PR and the kind of feedback desired could be better communicated by marking it as draft until those criteria are fulfilled.

- Added tests calling schemaFactoryAlpha.

alexvy86 · 2025-05-22T19:20:13Z

packages/dds/tree/src/feature-libraries/schema-edits/schemaChangeCodecs.ts

@@ -45,21 +48,23 @@ export function makeSchemaChangeCodec(
 /**
 * Compose the v1 schema change codec.
 * @param options - The codec options.
+ * @param schemaWriteVersion - The schema write version.


With support for this now, should the function name should change? Or what am I not getting about what "V1" means?

alexvy86 · 2025-05-22T19:27:12Z

packages/dds/tree/src/simple-tree/toStoredSchema.ts

@@ -89,7 +89,7 @@ export function convertField(schema: SimpleFieldSchema): TreeFieldStoredSchema {
 	const kind: FieldKindIdentifier =
 		convertFieldKind.get(schema.kind)?.identifier ?? fail(0xae3 /* Invalid field kind */);
 	const types: TreeTypeSet = schema.allowedTypesIdentifiers as TreeTypeSet;
-	return { kind, types };
+	return { kind, types, persistedMetadata: undefined };


Should this be schema.metadata, like we already do below for Map and Array nodes?

Initial change.

bd8406a

- Added in-memory schema format v2. - Implemented the schema format v2 codec. - Updated test utils that previously assumed there was only one format. - Added snapshots for many of the v2 tests.

TommyBrosman mentioned this pull request May 12, 2025

Shared Tree: Schema v2 with persisted metadata support #24541

Closed

persistedMetadata is now backed by an object instead of a string.

3165fe3

Updated codec.spec.ts to cover FormatV2.

9709ede

TommyBrosman commented May 13, 2025

View reviewed changes

packages/dds/tree/src/codec/codec.ts Outdated Show resolved Hide resolved

TommyBrosman commented May 13, 2025

View reviewed changes

packages/dds/tree/src/feature-libraries/schema-index/codec.ts Outdated Show resolved Hide resolved

Removed extraneous whitespace change.

ba058e1

github-actions bot removed the area: examples Changes that focus on our examples label May 13, 2025

TommyBrosman added 2 commits May 13, 2025 12:06

- Changed TreeNodeStoredSchema implementations to take the write vers…

b819742

…ion as a parameter. - Fixed naming of schema index formats. - Changed schemaChangeFormat to be schema format-agnostic.

- Changed the stored schema implementation to handle multiple schema …

0b5c842

…formats. - Moved SchemaCodecVersion.

TommyBrosman commented May 13, 2025

View reviewed changes

packages/dds/tree/src/core/schema-stored/formatV2.ts Show resolved Hide resolved

Merge branch 'main' into metadata-schema-3

ecdca7a

TommyBrosman marked this pull request as ready for review May 13, 2025 19:49

TommyBrosman requested a review from a team as a code owner May 13, 2025 19:49

CraigMacomber reviewed May 13, 2025

View reviewed changes

packages/dds/tree/src/core/schema-stored/formatV2.ts Outdated Show resolved Hide resolved

TommyBrosman and others added 5 commits May 15, 2025 15:44

Switched back to using min client version for importCompressed.

7c48b9d

Minor: reverted accidental change.

a2f4195

Apply suggestions from code review

4786dcc

Co-authored-by: Alex Villarreal <[email protected]>

Made persisted metadata schema field naming consistent.

6fa8836

- Created a separate storedSchemaDecodeDispatcher for v2 schemas.

e4064c7

- Made importCompressed always use a v1 schema when encoding.

CraigMacomber reviewed May 19, 2025

View reviewed changes

packages/dds/tree/src/feature-libraries/schema-index/codec.ts Outdated Show resolved Hide resolved

CraigMacomber reviewed May 19, 2025

View reviewed changes

packages/dds/tree/src/test/feature-libraries/schema-index/codecUtil.ts Outdated Show resolved Hide resolved

TommyBrosman added 5 commits May 19, 2025 12:02

- Refactored the node kind dispatch.

ab1b882

- Added better test data to one of the codec tests for the v2 schema format.

Updated snapshots.

1027daa

- Removed minimum client version changes. Equivalent changes will be …

e527eba

…added in another PR. - Updated a comment.

Removed more minimumClientVersion glue.

3eae1f6

Updated API files.

98e1b76

github-actions bot removed area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct public api change Changes to a public API labels May 19, 2025

CraigMacomber reviewed May 20, 2025

View reviewed changes

packages/dds/tree/src/feature-libraries/schema-index/formatV2.ts Outdated Show resolved Hide resolved

Rename: SchemaCodecVersion -> SchemaVersion.

ecc77c9

TommyBrosman changed the title ~~Shared Tree: Schema v2 with persisted metadata support~~ Shared Tree: Persisted Schema Format v2 with persisted metadata support May 20, 2025

TommyBrosman marked this pull request as draft May 20, 2025 20:30

TommyBrosman added 4 commits May 20, 2025 14:02

Merge branch 'main' into metadata-schema-3

37e900f

Removed top-level persisted metadata.

737633b

Fixed comments.

4bb136c

Exposed persistedMetadata as a JsonCompatibleReadOnlyObject.

0bcb90f

github-actions bot added the public api change Changes to a public API label May 22, 2025

Changeset.

a675338

github-actions bot added the changeset-present label May 22, 2025

TommyBrosman added 2 commits May 22, 2025 10:38

Wired up toStoredSchema. Still needs tests.

1d1ef5f

- Updated an old snapshot that didn't include the metadata field.

b9898e8

- Added tests calling schemaFactoryAlpha.

alexvy86 reviewed May 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

Uh oh!

TommyBrosman commented May 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CraigMacomber commented May 20, 2025

Uh oh!

CraigMacomber commented May 20, 2025

Uh oh!

alexvy86 May 22, 2025

Uh oh!

alexvy86 May 22, 2025

Uh oh!

Uh oh!

Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

Are you sure you want to change the base?

Shared Tree: Persisted Schema Format v2 with persisted metadata support #24590

Uh oh!

Conversation

TommyBrosman commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Notes for reviewers

Todo

Related PRs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 13, 2025

linkcheck output

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CraigMacomber commented May 20, 2025

Uh oh!

CraigMacomber commented May 20, 2025

Uh oh!

alexvy86 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

alexvy86 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TommyBrosman commented May 12, 2025 •

edited

Loading