if/then/else validation sequence tests

[gregsdennis]

I had an issue raised in my new implementation that highlighted a bug around if/then/else appearing in the "wrong" order in a schema. Would be a good few tests to have.

I'm on mobile, so I'm just making a note, really. I'll try to work up a PR later.

[Relequestual]

This is confusing. A JSON object, and therefore a JSON Schema, has no order. Arrays have order, but objects do not.

A JSON file may have a textual order in which keys appear in an object, but I've never seen a JSON parser include that information once JSON is parsed into something native.

Could you clarify what you mean here?

[ssilverman]

If a parser has a bug then it’s always a good idea to have a test for it, even if we think “that shouldn’t happen because assumption A, B, or C.” Just because your implementation assumes something, doesn’t mean other implementations do as well.

To wit: JSON parsers aren’t required to have non-ordered objects, nor are they required to have non-duplicated keys in objects. It just so happens that it’s very common to implement them as an unordered “map” with unique keys.

I’m with Greg in this one. If an implementation breaks on some schema, then a test for why it broke is a good idea.

[ssilverman]

Ok, I know y’all are clamoring for a link: https://tools.ietf.org/html/rfc8259#section-4 (and kinda section 9, but that’s a distant relationship). :)

[gregsdennis]

Regardless of how keys exist in a file, I still have to iterate over them, meaning I get them in some order.

I must process the if before the then, or it doesn't work. If I don't take special care and just process them as I iterate over them, the then may be given to me first, but at that point, I don't have any annotations from an if, so it gets skipped.

This is wrong, and it deserves a test to ensure that implementations aren't making that mistake.

[Relequestual]

So when you serialise from JSON and then the the keys of the object, you get them in the order they were in the file?

In all the languages I've worked in, getting the keys from an object are never ordered. Therefore I'd never considered anyone would simply iterate over them to process a schema.

Given that, unless the serialised JSON keys are in the same order as they were in the file, you can't put such a test in JSON.

[gregsdennis]

They're not "ordered," but there is some sequence in which the implementation processes them. No implementation runs them in perfect parallel.

[gregsdennis]

Manatee.Json, Newtonsoft.Json, and System.Text.Json are all capable of outputting the exact text they read in, which means they process them in the order of the file.

If any implementation reads a file at all, it must read them in the order of the file simply by virtue of reading the bytes.

[Relequestual]

What I'm trying to say here is, I understand the problem, but because of the nature of the problem, it cannot be reliably tested in this test suite, because the order won't be the same as it was in JSON, when parsed.

[Relequestual]

If you did, you'd sometimes get false positives, especially if the order of keys is deterministic, which it is in some languages.

[Relequestual]

@ssilverman While we test that order in arrays doesn't effect applicators that have to happen in some order, and that's fine, we can't test the JSON parser in the test suite.

There are MANY tests.
This is my go to link on this issue:
http://seriot.ch/parsing_json.php

[notEthan]

a fair number of json parsers preserve key order, including the one built into ruby. a test that key order does not affect if/then/else validation sounds like a good addition to me. I don't really understand the opposition to it - of course it won't make any difference in parsers where key order is not preserved, but in the not-particularly-rare case that it is preserved, order independence is worth affirming.

it's probably worthwhile to do this for other order-dependent keys as well.

[Julian]

There are other tests like this in the suite. To me too it's a good idea.

You can test this by having two different tests with different serialized orders. It's true that as JSON they're supposed to be equal but as you say in languages with ordered maps you still can get bugs.

I'd just say in my typical style though that these kinds of tests are exactly more reasons I am a stickler about running automated scripts over the suite :). Since a test like this will likely get totally screwed up when the parser used e.g. sorts the keys upon reserializing.

But yeah +1 from me on adding these.

[Relequestual]

I'm not yet convinced. I'm going to ask some more knowledgeable colleagues.

[handrews]

[EDIT: I read this issue first thing in the morning, which you'd think I'd know not to do by now, and got it turned around in my head- the following is left up for amusement value only 😅 ]

OK absolutely flat-out no. JSON keys are NOT ordered. The fact that many parsers do preserve order (and some preserve order at least most of the time- I've encountered some that only do so up to some size, although don't ask me for a reference now it was years ago) is completely irrelevant. The spec is 100% clear on this (from the section @ssilverman referenced):

JSON parsing libraries have been observed to differ as to whether or
not they make the ordering of object members visible to calling
software. Implementations whose behavior does not depend on member
ordering will be interoperable in the sense that they will not be
affected by these differences.

That means that the ONLY interoperable behavior is not depending on member ordering. From the perspective of JSON Schema, "interoperability" is important across all languages/libraries/environments, so just having one library/language/environment that is consistently ordered within itself is irrelevant. There exist valid RFC 8259-compliant JSON parsers to produce unpredictable ordering, therefore that is the behavior that MUST be supported. JSON Schema authors can and have always been able to write keywords in any order, and as the person who added if/then/else there is not and never has been any requirement that they appear in order, nor was it ever even briefly considered.

The JSON Schema data model is also intentionally divorced from the JSON text representation (and JSON Schemas, which can be JSON schema instances, are therefore handled according to the JSON Schema data model), for numerous reasons including the unpredictability of object parsing order.

---

json-schema-org/json-schema-spec#996 provides a machine-readable solution for noting these and other dependencies. Given then information in that file format, it is trivial to note that when you see then or else, you MUST first check for if and, if it is present, evaluate it. And then evaluate either then or else (but if if is not present, neither then nor else are evaluated). Likewise for dependencies such as additionalProperties on properties and patternProperties, and for unevaluated* on all in-place applicators regardless of the specific keyword.

Ordering. Is. Not. Significant. Ever.

[Julian]

I'm not sure what we're arguing about here. Is someone actually saying these tests (to ensure in fact ordering is not significant) shouldn't go in?

If so, please elaborate on why.