schema top down explainer

[snarfed]

courtesy of @pfrazee in https://matrix.to/#/#bluesky-dev:matrix.org just now. copying this here to preserve it somewhere a bit easier to read and link to. lines without a username are from him. thanks @pfrazee et al!!!

let me give a kind of top down explainer here
apologies for covering stuff you likely already know!
right so schemas, first things first, you create them under domain-names that you own. Very important you control the domain; never create new schemas under com.atproto.* or something like that
we call those reverse-dns identifiers "NSIDs" (NameSpaced IDentifiers) and basically the last segment of it is free-form while every leading segment is reverse dns
so if I'm putting my schemas under pfrazee.com then I would do com.pfrazee.whateverIWant
in the com.atproto and app.bsky we break it down into a lot of subdomains for organizational purposes
next up: conformity
once a schema has been put into use, you can never publish a breaking change. You can't add new required fields, you can't make a formerly required field optional, you can't remove or add a constraint like maxLength. I mean, you can, but you will absolutely break software that's in the field and applying old rules
it's subtle what you can and cant do but the easy rule is that there's only one safe thing to do: adding new optional fields
yeah not intuitive and totally sucks
to give an example: if we ever decided "you know what, character limits suck, let's bump it to 5000" (we wont decide this) then we'd have 2 options
the first option would be to add a new longText field to the post record with the 5000 character limit. It would have to be optional and, importantly, any records we create would need to populate the old text field with a truncated version
the second option would be to create an entirely new post record type like post-v2
with that understood, let's touch on stuffing unspecced fields into records
again, you can put data in unspecced fields on a record. It doesn't fail validation and it's not considered invalid. However we strongly recommend against it
there are two reasons we recommend against it: 1) it's entirely possible that the person controlling the schema will eventually use that field as a new optional field. Now you're hosed. 2) there's very little semantic information attached to these unspecced fields so you're going to end up with siloes of compatibility
George Antoniadis: again, you can put data in unspecced fields on a record. It doesn't fail validation and it's not considered invalid. However we strongly recommend against it
The doesn’t fail validation part, is that defined on the spec or up to pds/client implementaters?
that's according to spec
there may be usecases where it's actually the right call to stuff unspecced fields into records and that's why we don't stop it
for a simple example, you're just trying to track something on your bot's posts. There's no reason for us to get in your way on that; the point of all this is to help people coordinate and get work done; not to be dicks
okay so that's unspecced fields, so how do we extend schemas
Lexicon has two "complex" types for that, the Union reference and the Unknown reference
a Union reference type is equivalent to this in typescript: foo: Foo | Bar | unknown
and the Unknown reference is foo: unknown
you can actually specify a "Closed Union" which is the equivalent of foo: Foo | Bar but we recommend against it strongly because you can never add values to the field if you do that
when a Union or Unknown is used, the value of the field is now flexible; you have to look at the object it has (Unions and Unknowns can only ever have typed objects as their values)
this is where the $type field comes into play. You look at the $type of a Union/Unknown value to determine what its schema is. (The value will be an NSID)
generally speaking an open Union only differs from an Unknown by suggesting some well-known possible values. Technically you can put an object of any schema in there
the consumer of a Union/Unknown looks at the $type and then first establishes if it supports that schema at all. If not, it decides how to handle that (ignore, panic, etc). If it does support that schema, it then runs additional validation on that known schema
mikuhl: If you don't want data on existing schemas, you can have a collection with just one item right?
correct; if you make a new record schema you can do whatevs, and there's a pattern similar to SQL extension records which we're looking at that could use backlinks to essentially accomplish JOINs
snarfed: in the panic case, that means that PDS would handle forward compatibility by rejecting records until it gets upgraded with the new version of the lexicon, and then it would "catch up" with other PDSes's records that include that new field/type?
this isn't true yet, but the goal is to make the PDS indifferent to application schemas and provide generic behaviors only; that way adding some new schema doesnt run the risk of losing forward progress
snarfed: oh true good point. it's the app layer that would panic until it's upgraded, but the records would get accepted and stored before that
correct
but this segues well into the last item!
and that's the "Schema Negotiation" idea, which we're still thinking about how to do well. That idea is essentially, "How do we deal with unsupported schemas that are embedded in supported schemas?" like when somebody creates a new poll schema and embeds it in a post. This is actually pretty important
the thing is, you need to be able to predict what other software is going to do when they hit your new schemas
if you can't predict it then you'll never feel safe adding new features and why even bother
the broad idea is to find a way to include information within the records that explain to a consumer what to do if a piece of information is not supported
that could be instructions like "please abort if you don't understand this extension; you will fuck up and cause issues for my users"
they could also be "please display this fallback message / link"
they could also be "try this other object, maybe you'll know that schema"
mikuhl: That seems to be what the extensions array shows rn
and they could also be "just ignore it's fine"
well the extensions array -- if you're referring to $ext -- isnt actually implemented. That was an early iteration of Lexicon that I dropped for v1
the interesting challenge is finding a way to encode this information in a way that works for a lot of very different scenarios
instructing a client how to react to an unknown post embed is very different than reacting to an unknown like record, for instance
so we decided to defer the answer to schema negotiation until we had more real world information
mikuhl: The only other question I would have is in terms of performance, let's say you want to get custom reactions for each post, how would you tell the server to do that without calling an endpoint for EVERY post.
this gets into creating sufficiently generic APIs and indexes, which is an ongoing piece of work by the protocol team
snarfed: hopefully the indexing part can be handled mostly out of scope, locally by PDSes as an implementation detail, globally with the big world indexing part?
indeed! And there are some pretty solid generic indexes we can make, like backlinks, that should be able to do a lot of useful things
shreyanjain: This is probably a really silly idea, but I can imagine a schema for just letting people add random unspecced fields and play. Early Twitter I think used to let profiles have a feed of what was essentially arbitrary json that wouldn’t appear on your main profile but other apps could interact with it.
this basically exists already; you can create any arbitrary collection you want under your own schema and then enumerate records from it

[1inguini]

that's the "Schema Negotiation" idea, which we're still thinking about how to do well.

Maybe looking at Apache Avro's specification on handshake might give people some hint.
I wonder why the atproto didn't make use of existing RPC/serialization protocols like Apache Avro, gRPC, Apache Thrift, Cap’n Proto, FlatBuffers Simple Binary Encoding, etc and reinvented XRPC.