Skip to content

Commit 59baf03

Browse files
feat(tree): staged allowed types (#25116)
## Description Adds `staged` API to `SchemaFactoryAlpha`. This creates an allowed type in the view schema that may or may not be included int he stored schema. Continuation of #24631, with a new PR from my fork so I can have push permissions. Major changes: - Adds SchemaFactoryAlpha.staged - There is no longer a single choice for how to derive a stored schema from a view schema: - Different use cases have been split into different APIs for example (toInitalSchema, toUpgradeSchema) - The underlying toStoredSchema now takes options to control how each staged schema is handled. - Unhydrated content always permits staged content (with the exception of clone): this ensures that export/import round tripping of staged content works. - testDocuments test suite has been expanded so existing round trip tests cover the above. Known issues/limitations: - Recursive types are not supported. Tracked by https://dev.azure.com/fluidframework/internal/_workitems/edit/45711 - Clone should produce nodes with a union of source context and staged types so that both unknown optional fields work, and new staged types can be inserted. Tracked by https://dev.azure.com/fluidframework/internal/_workitems/edit/45725 but also partially hidden by https://dev.azure.com/fluidframework/internal/_workitems/edit/45723 which covers how inserting the out of schema types does not error in unhydrated context currently. This is ok, as we catch them when inserting into hydrated documents and thus the two pugs combine to make the desired behavior, but itn't great, and could cause other issue due to violating internal invariants. Some of the new clone tests cover this. - Some places, mainly those not using alpha typer, can't accept annotated allowed types and thus don't accept staged types. Examples include the root of the tree view config, recursiveObject fields, and likely more. Many of these can be worked around using dedicated alpha APIs and/or wrapping the implicit field schema in an explicit one using SchemaFactoryAlpha.required. Some cases, like recursiveArray do not have a viable workaround, and can be addressed in future work, possibly after stabilizing annotated allowed types.
1 parent 580e627 commit 59baf03

File tree

90 files changed

+2917
-425
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

90 files changed

+2917
-425
lines changed

.changeset/curly-bikes-train.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
"fluid-framework": minor
3+
"@fluidframework/tree": minor
4+
"__section": feature
5+
---
6+
Adds staged allowed types to SchemaFactoryAlpha
7+
8+
This adds the `staged` API to [`SchemaFactoryAlpha`](https://fluidframework.com/docs/api/fluid-framework/schemafactoryalpha-class).
9+
Staged allowed types can be used for schema evolution to add members to an [`AllowedTypes`](https://fluidframework.com/docs/api/fluid-framework/allowedtypes-typealias) while supporting cross version collaboration.
10+
11+
Staged allowed types are [allowed types](https://fluidframework.com/docs/api/fluid-framework/allowedtypes-typealias) that can be upgraded by [schema upgrades](https://fluidframework.com/docs/api/fluid-framework/treeview-interface#upgradeschema-methodsignature).
12+
Before being upgraded, any attempt to insert or move a node to a location which requires its type to be upgraded to be valid will throw an error.
13+
14+
To add a new member to an `AllowedTypes`, add the type wrapped by `staged`.
15+
For example, migrating an array which previously supported only numbers to support both numbers and strings would start by deploying a version of the app using `staged`:
16+
```typescript
17+
class TestArray extends schemaFactoryAlpha.arrayAlpha("TestArray", [SchemaFactoryAlpha.number, SchemaFactoryAlpha.staged(SchemaFactoryAlpha.string)]) {}
18+
```
19+
20+
Once enough clients have this code update, it is safe to allow writing strings to the array.
21+
To allow writing strings to the array, a code change must be made to remove the staged annotation:
22+
```typescript
23+
class TestArray extends schemaFactoryAlpha.arrayAlpha("TestArray", [schemaFactoryAlpha.number, schemaFactoryAlpha.string]) {}
24+
```
25+
26+
Then when opening old documents [upgradeSchema](https://fluidframework.com/docs/api/fluid-framework/treeview-interface#upgradeschema-methodsignature) is used to upgrade the stored schema:
27+
```typescript
28+
view.upgradeSchema()
29+
```
30+
31+
The `@alpha` API [extractPersistedSchema](https://fluidframework.com/docs/api/fluid-framework#extractpersistedschema-function) now takes the schema as an `ImplicitAnnotatedFieldSchema` and an additional parameter to filter which staged upgrades it includes.
32+
33+
Below is a full example of how the schema migration process works.
34+
This can also be found in the [tests](https://github.com/CraigMacomber/FluidFramework/blob/readonly-allowedtypes/packages/dds/tree/src/test/simple-tree/api/stagedSchemaUpgrade.spec.ts).
35+
36+
```typescript
37+
// Schema A: only number allowed
38+
const schemaA = SchemaFactoryAlpha.optional([SchemaFactoryAlpha.number]);
39+
40+
// Schema B: number or string (string is staged)
41+
const schemaB = SchemaFactoryAlpha.optional([
42+
SchemaFactoryAlpha.number,
43+
SchemaFactoryAlpha.staged(SchemaFactoryAlpha.string),
44+
]);
45+
46+
// Schema C: number or string, both fully allowed
47+
const schemaC = SchemaFactoryAlpha.optional([
48+
SchemaFactoryAlpha.number,
49+
SchemaFactoryAlpha.string,
50+
]);
51+
52+
// Initialize with schema A.
53+
const configA = new TreeViewConfiguration({
54+
schema: schemaA,
55+
});
56+
const viewA = treeA.viewWith(configA);
57+
viewA.initialize(5);
58+
59+
// Since we are running all the different versions of the app in the same process making changes synchronously,
60+
// an explicit flush is needed to make them available to each other.
61+
synchronizeTrees();
62+
63+
assert.deepEqual(viewA.root, 5);
64+
65+
// View the same document with a second tree using schema B.
66+
const configB = new TreeViewConfiguration({
67+
schema: schemaB,
68+
});
69+
const viewB = treeB.viewWith(configB);
70+
// B cannot write strings to the root.
71+
assert.throws(() => (viewB.root = "test"));
72+
73+
// View the same document with a third tree using schema C.
74+
const configC = new TreeViewConfiguration({
75+
schema: schemaC,
76+
});
77+
const viewC = treeC.viewWith(configC);
78+
// Upgrade to schema C
79+
viewC.upgradeSchema();
80+
// Use the newly enabled schema.
81+
viewC.root = "test";
82+
83+
synchronizeTrees();
84+
85+
// View A is now incompatible with the stored schema:
86+
assert.equal(viewA.compatibility.canView, false);
87+
88+
// View B can still read the document, and now sees the string root which relies on the staged schema.
89+
assert.deepEqual(viewB.root, "test");
90+
```

examples/apps/tree-cli-app/src/test/schema.spec.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ const historicalSchema: {
112112

113113
describe("schema", () => {
114114
it("current schema matches latest historical schema", () => {
115-
const current = extractPersistedSchema(config, FluidClientVersion.v2_0);
115+
const current = extractPersistedSchema(config.schema, FluidClientVersion.v2_0, () => true);
116116

117117
// For compatibility with deep equality and simple objects, round trip via JSON to erase prototypes.
118118
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment

examples/apps/tree-cli-app/src/utils.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,7 +163,7 @@ export function exportContent(destination: string, tree: List): JsonCompatible {
163163
idCompressor,
164164
}),
165165

166-
schema: extractPersistedSchema(config, compatVersion),
166+
schema: extractPersistedSchema(config.schema, compatVersion, () => true),
167167
idCompressor: idCompressor.serialize(true),
168168
};
169169
return file as JsonCompatible;

packages/dds/tree/api-report/tree.alpha.api.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ export function adaptEnum<TScope extends string, const TEnum extends Record<stri
1818
// @alpha
1919
export interface AllowedTypeMetadata {
2020
readonly custom?: unknown;
21+
readonly stagedSchemaUpgrade?: SchemaUpgrade;
2122
}
2223

2324
// @public @system
@@ -157,7 +158,7 @@ export function evaluateLazySchema<T extends TreeNodeSchema>(value: LazyItem<T>)
157158
type ExtractItemType<Item extends LazyItem> = Item extends () => infer Result ? Result : Item;
158159

159160
// @alpha
160-
export function extractPersistedSchema(schema: SimpleTreeSchema, oldestCompatibleClient: FluidClientVersion): JsonCompatible;
161+
export function extractPersistedSchema(schema: ImplicitAnnotatedFieldSchema, oldestCompatibleClient: FluidClientVersion, includeStaged: (upgrade: SchemaUpgrade) => boolean): JsonCompatible;
161162

162163
// @alpha @system
163164
export type FactoryContent = IFluidHandle | string | number | boolean | null | Iterable<readonly [string, InsertableContent]> | readonly InsertableContent[] | FactoryContentObject;
@@ -850,6 +851,8 @@ export class SchemaFactoryAlpha<out TScope extends string | undefined = string |
850851
static readonly requiredRecursive: <const T extends System_Unsafe.ImplicitAllowedTypesUnsafe, const TCustomMetadata = unknown>(t: T, props?: Omit<FieldPropsAlpha_2<TCustomMetadata>, "defaultProvider"> | undefined) => FieldSchemaAlphaUnsafe_2<FieldKind_2.Required, T, TCustomMetadata>;
851852
readonly requiredRecursive: <const T extends System_Unsafe.ImplicitAllowedTypesUnsafe, const TCustomMetadata = unknown>(t: T, props?: Omit<FieldPropsAlpha_2<TCustomMetadata>, "defaultProvider"> | undefined) => FieldSchemaAlphaUnsafe_2<FieldKind_2.Required, T, TCustomMetadata>;
852853
scopedFactory<const T extends TName, TNameInner extends number | string = string>(name: T): SchemaFactoryAlpha<ScopedSchemaName<TScope, T>, TNameInner>;
854+
static staged: <const T extends LazyItem<TreeNodeSchema>>(t: T | AnnotatedAllowedType<T>) => AnnotatedAllowedType<T>;
855+
staged: <const T extends LazyItem<TreeNodeSchema>>(t: T | AnnotatedAllowedType<T>) => AnnotatedAllowedType<T>;
853856
}
854857

855858
// @alpha
@@ -877,6 +880,17 @@ export interface SchemaStatics {
877880
readonly string: LeafSchema<"string", string>;
878881
}
879882

883+
// @alpha @sealed @system
884+
export interface SchemaStaticsAlpha {
885+
staged: <const T extends LazyItem<TreeNodeSchema>>(t: T | AnnotatedAllowedType<T>) => AnnotatedAllowedType<T>;
886+
}
887+
888+
// @alpha @sealed
889+
export class SchemaUpgrade {
890+
// (undocumented)
891+
protected _typeCheck: MakeNominal;
892+
}
893+
880894
// @alpha @input
881895
export interface SchemaValidationFunction<Schema extends TSchema> {
882896
check(data: unknown): data is Static<Schema>;

packages/dds/tree/src/index.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@ export {
118118
type InternalTreeNode,
119119
type WithType,
120120
type NodeChangedData,
121+
type SchemaUpgrade,
121122
// Types not really intended for public use, but used in links.
122123
// Can not be moved to internalTypes since doing so causes app code to throw errors like:
123124
// Error: src/simple-tree/objectNode.ts:72:1 - (ae-unresolved-link) The @link reference could not be resolved: The package "@fluidframework/tree" does not have an export "TreeNodeApi"
@@ -171,6 +172,7 @@ export {
171172
type UnannotateAllowedTypeOrLazyItem,
172173
type UnannotateImplicitFieldSchema,
173174
type UnannotateSchemaRecord,
175+
type SchemaStaticsAlpha,
174176
// Beta APIs
175177
TreeBeta,
176178
type TreeChangeEventsBeta,

packages/dds/tree/src/shared-tree/schematizeTree.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ import {
1919
defaultSchemaPolicy,
2020
mapTreeFromCursor,
2121
} from "../feature-libraries/index.js";
22-
import { toStoredSchema, type SchemaCompatibilityTester } from "../simple-tree/index.js";
22+
import { toUpgradeSchema, type SchemaCompatibilityTester } from "../simple-tree/index.js";
2323
import { isReadonlyArray } from "../util/index.js";
2424

2525
import type { ITreeCheckout } from "./treeCheckout.js";
@@ -219,7 +219,7 @@ export function ensureSchema(
219219
return false;
220220
}
221221
case UpdateType.SchemaCompatible: {
222-
checkout.updateSchema(toStoredSchema(viewSchema.viewSchema.root));
222+
checkout.updateSchema(toUpgradeSchema(viewSchema.viewSchema.root));
223223
return true;
224224
}
225225
default: {

packages/dds/tree/src/shared-tree/schematizingTreeView.ts

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,10 +51,10 @@ import {
5151
areImplicitFieldSchemaEqual,
5252
prepareForInsertionContextless,
5353
type FieldSchema,
54-
toStoredSchema,
5554
tryDisposeTreeNode,
5655
FieldSchemaAlpha,
5756
TreeViewConfigurationAlpha,
57+
toInitialSchema,
5858
} from "../simple-tree/index.js";
5959
import {
6060
type Breakable,
@@ -172,7 +172,7 @@ export class SchematizingSimpleTreeView<
172172
}
173173

174174
this.runSchemaEdit(() => {
175-
const schema = toStoredSchema(this.config.schema);
175+
const schema = toInitialSchema(this.config.schema);
176176
const mapTree = prepareForInsertionContextless(
177177
content as InsertableContent | undefined,
178178
this.rootFieldSchema,
@@ -181,6 +181,7 @@ export class SchematizingSimpleTreeView<
181181
policy: defaultSchemaPolicy,
182182
},
183183
this,
184+
schema.rootFieldSchema,
184185
);
185186

186187
initialize(this.checkout, {
@@ -436,7 +437,12 @@ export class SchematizingSimpleTreeView<
436437
);
437438
}
438439
const view = this.getFlexTreeContext();
439-
setField(view.root, this.rootFieldSchema, newRoot as InsertableContent | undefined);
440+
setField(
441+
view.root,
442+
this.rootFieldSchema,
443+
newRoot as InsertableContent | undefined,
444+
this.checkout.storedSchema.rootFieldSchema,
445+
);
440446
}
441447

442448
// #region Branching

packages/dds/tree/src/shared-tree/treeAlpha.ts

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,6 @@ import {
3636
verboseFromCursor,
3737
type TreeEncodingOptions,
3838
type VerboseTree,
39-
toStoredSchema,
4039
extractPersistedSchema,
4140
type TreeBranch,
4241
TreeViewConfigurationAlpha,
@@ -52,6 +51,9 @@ import {
5251
tryGetTreeNodeForField,
5352
isObjectNodeSchema,
5453
isTreeNode,
54+
toInitialSchema,
55+
convertField,
56+
toUnhydratedSchema,
5557
} from "../simple-tree/index.js";
5658
import { brand, extractFromOpaque, type JsonCompatible } from "../util/index.js";
5759
import {
@@ -493,7 +495,11 @@ export const TreeAlpha: TreeAlpha = {
493495
return undefined as Unhydrated<TreeFieldFromImplicitField<TSchema>>;
494496
}
495497
const cursor = cursorFromVerbose(data, schemalessConfig);
496-
return createFromCursor(schema, cursor);
498+
return createFromCursor(
499+
schema,
500+
cursor,
501+
convertField(normalizeFieldSchema(schema), toUnhydratedSchema),
502+
);
497503
},
498504

499505
exportConcise,
@@ -523,11 +529,18 @@ export const TreeAlpha: TreeAlpha = {
523529
const batch: FieldBatch = [cursor];
524530
// If none provided, create a compressor which will not compress anything.
525531
const idCompressor = options.idCompressor ?? createIdCompressor();
532+
533+
// Grabbing an existing stored schema from the node is important to ensure that unknown optional fields can be preserved.
534+
// Note that if the node is unhydrated, this can result in all staged allowed types being included in the schema, which might be undesired.
535+
const storedSchema = isTreeNode(node)
536+
? getKernel(node).context.flexContext.schema
537+
: toInitialSchema(schema);
538+
526539
const context: FieldBatchEncodingContext = {
527540
encodeType: TreeCompressionStrategy.Compressed,
528541
idCompressor,
529542
originatorId: idCompressor.localSessionId, // TODO: Why is this needed?
530-
schema: { schema: toStoredSchema(schema), policy: defaultSchemaPolicy },
543+
schema: { schema: storedSchema, policy: defaultSchemaPolicy },
531544
};
532545
const result = codec.encode(batch, context);
533546
return result;
@@ -543,7 +556,8 @@ export const TreeAlpha: TreeAlpha = {
543556
const config = new TreeViewConfigurationAlpha({ schema });
544557
const content: ViewContent = {
545558
// Always use a v1 schema codec for consistency.
546-
schema: extractPersistedSchema(config, FluidClientVersion.v2_0),
559+
// TODO: reevaluate how staged schema should behave in schema import/export APIs before stabilizing this.
560+
schema: extractPersistedSchema(config.schema, FluidClientVersion.v2_0, () => true),
547561
tree: compressedData,
548562
idCompressor: options.idCompressor ?? createIdCompressor(),
549563
};

packages/dds/tree/src/simple-tree/api/configuration.ts

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,11 @@ import {
2525
evaluateLazySchema,
2626
markSchemaMostDerived,
2727
} from "../core/index.js";
28-
import { toStoredSchema } from "../toStoredSchema.js";
28+
import {
29+
permissiveStoredSchemaGenerationOptions,
30+
restrictiveStoredSchemaGenerationOptions,
31+
toStoredSchema,
32+
} from "../toStoredSchema.js";
2933
import {
3034
isArrayNodeSchema,
3135
isMapNodeSchema,
@@ -161,6 +165,9 @@ export interface ITreeViewConfiguration<
161165

162166
/**
163167
* Configuration for {@link ViewableTree.viewWith}.
168+
* @privateRemarks
169+
* When `ImplicitAnnotatedFieldSchema` is stabilized, TSchema should be updated to use it.
170+
* When doing this, the example for `staged` will need to be updated/simplified.
164171
* @sealed @public
165172
*/
166173
export class TreeViewConfiguration<
@@ -213,7 +220,8 @@ export class TreeViewConfiguration<
213220

214221
// Eagerly perform this conversion to surface errors sooner.
215222
// Includes detection of duplicate schema identifiers.
216-
toStoredSchema(config.schema);
223+
toStoredSchema(config.schema, restrictiveStoredSchemaGenerationOptions);
224+
toStoredSchema(config.schema, permissiveStoredSchemaGenerationOptions);
217225

218226
const definitions = new Map<string, SimpleNodeSchema & TreeNodeSchema>();
219227

packages/dds/tree/src/simple-tree/api/create.ts

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,9 @@ import {
1313
mapCursorFields,
1414
type ITreeCursorSynchronous,
1515
type SchemaAndPolicy,
16+
type TreeFieldStoredSchema,
1617
} from "../../core/index.js";
17-
import {
18-
normalizeFieldSchema,
19-
type ImplicitFieldSchema,
20-
type TreeFieldFromImplicitField,
21-
} from "../fieldSchema.js";
18+
import type { ImplicitFieldSchema, TreeFieldFromImplicitField } from "../fieldSchema.js";
2219
import {
2320
type Context,
2421
getOrCreateNodeFromInnerNode,
@@ -32,33 +29,31 @@ import {
3229
throwOutOfSchema,
3330
} from "../../feature-libraries/index.js";
3431
import { getUnhydratedContext } from "../createContext.js";
35-
import { convertField } from "../toStoredSchema.js";
3632
import { unknownTypeError } from "./customTree.js";
3733

3834
/**
3935
* Creates an unhydrated simple-tree field from a cursor in nodes mode.
4036
* @remarks
4137
* Does not support providing missing defaults values.
42-
* Validates the field is in schema using the provided `contextForNewNodes` of the default unhydrated context if not provided.
38+
* Validates the field is in schema using `destinationSchema` the provided `contextForNewNodes` or the default unhydrated context if not provided.
4339
*/
4440
export function createFromCursor<const TSchema extends ImplicitFieldSchema>(
4541
schema: TSchema,
4642
cursor: ITreeCursorSynchronous | undefined,
43+
destinationSchema: TreeFieldStoredSchema,
4744
contextForNewNodes?: Context,
4845
): Unhydrated<TreeFieldFromImplicitField<TSchema>> {
4946
const context = contextForNewNodes ?? getUnhydratedContext(schema);
5047
assert(context.flexContext.isHydrated() === false, 0xbfe /* Expected unhydrated context */);
5148
const mapTrees = cursor === undefined ? [] : [unhydratedFlexTreeFromCursor(context, cursor)];
5249

53-
const rootFieldSchema = convertField(normalizeFieldSchema(schema));
54-
5550
const schemaAndPolicy: SchemaAndPolicy = {
5651
policy: defaultSchemaPolicy,
5752
schema: context.flexContext.schema,
5853
};
5954

6055
// Assuming the caller provides the correct `contextForNewNodes`, this should handle unknown optional fields.
61-
isFieldInSchema(mapTrees, rootFieldSchema, schemaAndPolicy, throwOutOfSchema);
56+
isFieldInSchema(mapTrees, destinationSchema, schemaAndPolicy, throwOutOfSchema);
6257

6358
if (mapTrees.length === 0) {
6459
return undefined as Unhydrated<TreeFieldFromImplicitField<TSchema>>;

0 commit comments

Comments
 (0)