utility: Deserialize (an array) to a range

[Inkrementator]

Deserializing a JSON-array to a range is risky due to having to be careful with ownership of the underlying text-iopipe, but useful to save on allocations.

So until the range is fully consumed, client code wouldn't be allowed to advance the JsonTokenizer. Maybe an implementation could be clever and detect this and throw in that case, maybe only in debug mode.

Another source of problems will be that ranges are mostly value-types in D. foreach will implicitly create a copy of the range it iterates over, so care has to be taken to not store the full JsonTokenizer object but only a handle to it.

cc @benjones

[Inkrementator]

Re ownership: Appenders in phobos solve this by allocating the real state on the heap and only having the user visible struct having a handle to it

[schveiguy]

I'm assuming we are talking about deserialization here? Because I already handle serialization of ranges (as a JSON array).

There are 2 things here, first, there may be a desire to deserialize to an output range, which means it's just like the array case, but uses put for each element. The second is that you may want to use the deserialization as an element in an input range. So for instance, let's say you have a json file with a huge array of megabytes of elements. Instead of deserializing that to an array, you parse to the start of the array, and then wrap the deserialization process into a range that you then use from the outside. Essentially, deserializing each element as you go.

In both cases, you are correct, the lifetime management of the parsed data is critical. Even when you keep all the json tokens in memory, the underlying buffer can move!

Handling this gracefully requires some thought...

[Inkrementator]

I'm assuming we are talking about deserialization here?

edited the post for clarity

---

In both cases, you are correct, the lifetime management of the parsed data is critical.

I think disabling copying for JSONTokenizer would be a good start. They should be either moved or borrowed via reference.

Same is true for most iopipes, I think.

[schveiguy]

OK, so in #61 I have started a new idea of storing the absolute position of tokens instead of the buffer position. This makes it immune to relocating the buffer (As long as you don't save the underlying slice directly). Slicing the data after it's released from the buffer will result in a bounds check error.

I also have changed the element type of JSONTokenizer and the new JSONPipe to an element which knows how to fetch the slice.

I kind of want to make the element type no-copy, since it depends on the pipe not changing addresses, and this would discourage using them longer than the immediate expression.

I'm very hesitant to enforce a copy ban on iopipes or the tokenizer. It will disable a lot of the cool syntax that makes iopipes pleasant. Storing them in a ref-counted struct also adds indirections for cases where you never make a copy anyway.

I think this will have to be addressed in the near future, because phobos v3 is seriously looking at iopipe as the basis for streams, and we need a good answer for this.

[Inkrementator]

I kind of want to make the element type no-copy, since it depends on the pipe not changing addresses, and this would discourage using them longer than the immediate expression.

The new API is already much harder to accidentally misuse, since the sequence points are more obvious.

But if you wanted to make it actually safe, I suggest to instead add a sequence number to JSONTokenizer that gets incremented on every flushCache and also put it into Element and check that on every access. That's also invisible to the user instead of the 'nudge' that is forbidding copying.

This could even be selectively activated with a build flag if anybody cares about the runtime cost since it's such a small change

[schveiguy]

effectively, that's what I'm doing in #61. Instead of using the "offset" which is a buffer offset, it's using "position", which is an absolute position in the stream. If the position math means the slice goes outside the buffer, then you should get an out of bounds error.

The tricky part is that you can always just save the slice, and then it won't be "correct" if the buffer is adjusted. However, it should always be safe as in memory safe (the buffer doesn't get deallocated). It just might look like it's garbage.

[Inkrementator]

Using a policy now allows you to process the array elements piece by piece with a callback.

While this is not as convenient as an input range in some situations, it still solves the issue of having to allocate each array.