Userdata support

[nwinn-student]

What would users need when working with userdata?

Would users like a reference to the existing table, should the userdata be within a table?

• Solely useful for when/if table.serialize can take the existing table as an argument.

[nwinn-student]

The functions would be deoptimized since get/setfenv could be used on them.

I feel like the ability to get/pass the existing table would benefit w/o sacrificing too much performance for the general case. So passing the internal functions will not be accepted (due to the performance implications applied to the general case).

Actually, the serialTable function and serialCache function can already be obtained using the debug library. It isn't consistent, since pairless may be added. So users cannot expect a single approach to always work.

But, it is possible to already get these values and use them.

Is there a way to get the existing table?

[nwinn-student]

Is there a way to get the existing table?

In Lua there is, the debug.getupvalue function. But in Luau, debug cannot get the upvalue and I do not think that getfenv can get the upvalues either, just the globals.

[nwinn-student]

Actually, the serialTable function and serialCache function can already be obtained using the debug library. It isn't consistent, since pairless may be added. So users cannot expect a single approach to always work.

But, it is possible to already get these values and use them.

The correct way is to go up the function call chain until table.serialize is reached. The user should then go down one, to find serialTable, or two, to find serialCache. Most cases would just require calling serialCache.

[nwinn-student]

I can benefit from the serialCache idea, specifically for Roblox Instance serialization performance.

Internal functions/structures should remain internal. Users should not be able to use serialCache, or have a need to get it.

In the examples directory, I point towards using Rose serializer, which converts Instances into tables. Currently we are serializing each table individually, not carrying gained information from table to table.

I believe that the new idea solves this particular issue, as the table can be directly serialized.

[nwinn-student]

Internal functions/structures should remain internal. Users should not be able to use serialCache, or have a need to get it.

There is no way, I think, to protect the internal functions from being accessed. I presume that coroutines will still have references to the internal functions.

[nwinn-student]

If we pass the existing table, do we also need to pass the existing_index and roundRobin stuff?

We need both. The existing_index is needed to know the current position. The roundRobin is needed to ensure that the added unique values and read existing values do not have conflicting ids.

[nwinn-student]

Should we create a table that holds all the values and pass that each userdata? Like below:

type Existing = {
  data: {[any]: any},
  index: number,
  round_robin: {any},
  round_index: number,
}

The entire type taken should be the same type that can be passed to table.serialize for a better UX.

[nwinn-student]

Should we create a table that holds all the values and pass that each userdata? Like below:

Before I jump the gun and assume that it has vast performance implications I will explain the implications.

Existing and round_robin becomes slower to add elements to, since a table needs to be newindex'd. We could mitigate this regression by only updating whenever a userdata is going to be serialized and updating the localized index values after. The initialization of table.serialize becomes slightly slower and takes more memory.

[nwinn-student]

type Existing = {
  data: {[any]: any},
  index: number,
  round_robin: {any},
  round_index: number,
}

What if the tables are modified in transit? What if index or round_index is not a number?

[nwinn-student]

What if the tables are modified in transit? What if index or round_index is not a number?

The risk of illegal modification and overhead of validation cause this idea to be put on the backburner. Another idea is currently being honed that would make this PR useless from the userdata perspective.

[nwinn-student]

A prior commit for the implementation passed a table that contained the buffer, position, and size. Going from that version to now, there was a performance improvement from passing each individually. I could revert in order to improve the user experience.

On the note of safety, the userdata functions can already cache the buffer or mess with the data. Modifying the table to have unnecessary values, or a metatable, is not a concern relative to the buffer.

On the note of likelihood, it is unlikely that this concept will be reconsidered. The experience is not all around better by passing a table. There are other ways, like adding the size and position condition to inflate.

[nwinn-student]

What if you want to remove supported userdata?

We could do something like link.values. Have a table internally that can be access to remove elements. I moreso think that we should only provide methods to remove elements since we want the table to be protected. For example: userdata.clearSupported(keepNLatest: number?). Removing by knowing the table is not practical.

[nwinn-student]

Can we formalize the idea into one coherent message so we can continue to iterate without needing to bounce back and forth?

[nwinn-student]

We propose the addition of two functions to the public API.

type SupportUserdata = (record: {
  -- tag number must be between 0 and 255 (or it will be morphed into a corresponding 0-255 value)
  tag: (userdata)->number?,
  serialize: (userdata, tag: number) -> any?,
  deserialize: (data: any, tag: number) -> userdata?
}) -> ()

-- bs.supportUserdata

type IsSupported = (userdata) -> boolean

-- bs.isSupported

For bs.supportUserdata, if any of the provided functions return nil, or effectively nil, then the next supported userdata function will be called, until there are none left. Upon a userdata not being supported by the tag functions or the userdata not being supported by the serialize functions, the NIL userdata byte will be stored. The order called goes from the latest addition via supportUserdata to the earliest. If the userdata is not supported by the deserialize functions, returning nil, an error will be thrown.

As a safety precaution, the provided tables are immediately cloned, frozen and stored, thus any modification to the provided table will not register.

We also propose the removal of readers/writers and their corresponding methods in the internal userdata API. Instead of the user needing to get the reader/writer function and call it when their case does not support a userdata, we will internally take on that responsibility. As such, the getter functions will not be exposed. However, much like pairs, removal of support must be present. As such a new internal function will be added that provides the capability to clear the supported userdata.

type ClearSupport = (keepNLastest: number?) -> ()

-- userdata.clearSupport

Unexpected Gains

I will not mention the ability to modify userdata or data currently being serialized, however it is now possible to use userdata as blockers or senders of data. It is even possible to have access to the root table being deserialized near immediately upon the start of deserialization.

In order to achieve such an act, the userdata should be able to access the root table being currently serialized. This root table can then be returned, which allows for cheap references to be held that trigger code to be run upon deserialization. The code could intentially error, which would be useful in cases where the desired data has already been deserialized and continuation would be a waste of time and memory, a "fast-path".

[nwinn-student]

We propose the addition of two functions to the public API.

type SupportUserdata = (record: {
  -- tag number must be between 0 and 255 (or it will be morphed into a corresponding 0-255 value)
  tag: (userdata)->number?,
  serialize: (userdata, tag: number) -> any?,
  deserialize: (data: any, tag: number) -> any?
}) -> ()

-- bs.supportUserdata

type IsSupported = (userdata) -> boolean

-- bs.isSupported

For bs.supportUserdata:

• Returning nil in tag, serialize, or deserialize causes the next series of corresponding supported functions to be called. From henceforth, supporting means returning nonnil.

• Upon a userdata not being supported by all of the tag or serialize functions, the NIL userdata byte will be stored.

• If none of the deserialize functions support the userdata, an error will occur.

• The latest supported functions will be called first, with the earliest supported functions being called last.

• The provided tables are immediately cloned, frozen and stored.

We also propose the removal of readers/writers and their corresponding methods in the internal userdata API. Instead of the user needing to get the reader/writer function and call it when their case does not support a userdata, we will internally take on that responsibility. As such, the getter functions will not be exposed. As such a new internal function will be added that provides the capability to clear the supported userdata.

type ClearSupport = (keepNLastest: number?) -> ()

-- userdata.clearSupport

[nwinn-student]

We return to the circle I guess. We declined existing earlier, but I believe it was partially since it could be mishandled. If we instead assume that users won't ever use internals, we won't need to think about that.

There is still a maintenance concern. Table serialize needs access to existing, but it also needs to give userdata existing. I presume the same holds for deserialize. Again, we can implement and see how painful it is.

We could also make assumptions that involve no maintenance, or at least minimal, but understanding what is going on is important from the user-side. We shall see. I guess this puts the option back on the table, so we need to yield externalizing userdata until we have come to a clear decision.

[nwinn-student]

Fun to get to the issues with userdata then start doubting whether it is worth keeping, just like pairs. And similar to pairs, it has migration issues.

I can enforce that tags cannot be overwritten during that execution, however later versions may remove tags and add other tags. If a tag that was removed happens to be replaced, it causes old data serialized under that tag to no longer be supported.

Note: The solution would be to tell users they can only add support for userdata and never remove. They can shift supported functions to be called first, but clearSupport must be removed and renamed swapSupport. Or we swap when the tag, and corresponding functions are all the same on addSupport.

We could even extend userdata tags using the pairs next approach to hold upwards of a few thousand tags, if that is needed.

Simply, removing userdata is not an option.

[nwinn-student]

I know I mentioned output size earlier, but there are ways to minimize the additional cost, in most cases.

With Color3 we currently store in 5 bytes, the naive approach would be to store as { r, g, b } which results in 10 bytes. The most optimal approach size-wise would be to store the number as a tryte, so 6 bytes. For alot of other values, using an array would be necessary. Dictionaries should be avoided, even if existing works for userdata serialized stuff, which it may not.

[nwinn-student]

bs.supportUserdata({
  tag = 0,
  supported = function(ud) return ud == foo end,
  serialize(ud) return ud:deconstruct() end,
  deserialize(d) return construct(d) end,
})

This would have likely been better since users wouldn't need to handle tag logic, and it would be clear which function to run when.

For supported we could have had util functions that return new ones. Like IsSupportedType("Foo") would return a function that returns typeof(ud) == "Foo". Or IsEqual(foo) would return ud == foo.

I have discovered another issue. Since the developer has less feedback about the output size, they are inclined to make it maintainable. However, if they try to minimize output size, it will become unmaintainable. As in a prior reply, Color3 would either be 6 or 10 bytes. But imagine the developer goes for maintainability, it becomes { r: number, b: number, g: number } which is 16 bytes. I would rather give the developer control so they can easily make it 5 bytes, instead of making them think about some skewwed version of maintainability.

[nwinn-student]

Key question: Is this worth implementing?

It could be more performant.

Userdata could take the table approach if we implement this, instead of passing (de)serializers.

We could internally generate a table of (de)serializers for each buffer, which would allow users to call number(5) and get the buffer back. I wonder how it would work though.

Does it need to be sequential? So I cannot re-serialize a section or re-deserialize a section? I presume so.

The internally provided functions can return the buffer, but if we pass them to userdata fns, the userdata fns should not gain access to the buffer.

[nwinn-student]

Too difficult to maintain. Although, if we externalize them as functions that take in some structure containing the buffer,pos,size and maybe more, they c-. No. The buffer when it gets inflated would not update for other functions. If we use a table, it would be too slow and difficult to maintain too. Tried it.

So, no to this idea.

[nwinn-student]

Fresh view, not looking at anything else in this thread until this reply is complete.

How do we know a userdata is supported and how do we treat unsupported userdata?

We currently cannot know when a userdata is or isn't supported until it has passed the function and even then we cannot be certain. We also don't know what to do in situations where the userdata isn't supported.

We are using different userdata to represent NIL, which should be renamed to UNSUPPORTED. Is there a reason for it producing different userdata? I may have prior used it to differentiate unsupported userdata, however I believe that unsupported userdata will not be used. In my case, I merely store unsupported userdata since I don't want to reconstruct the entire structure to remove them. In fact, if they share a userdata, one could say if ud == unsupported then ... end and create logic to default the value, in the current case it is not as easy to be certain whether a userdata is unsupported.

When userdata writers are provided and a userdata is unsupported, what should occur?

The user should return the respective values, all being equal, then we rewrite the CUSTOM byte into UNSUPPORTED. However, currently we write CUSTOM UNSUPPORTED. It takes away a bytecode that users may need and is unintuitive.

[nwinn-student]

We currently cannot know when a userdata is or isn't supported until it has passed the function and even then we cannot be certain.

Other replies have mentioned that returning nil from the serialize userdata function is more intuitive for the user. I can make that viable, where if the position hasn't changed it is deemed unsupported. So returning nil is fine.

We could still add bs.isSupported(ud: userdata) and have it serialize then deserialize and check if the userdata is userdata.unsupported. Quite expensive, but prior replies have hinted towards its importance and the function is intended to be used on startup and seldomly past that. Although, if the userdata passed is userdata.unsupported it will be quite fast. The intent of the function isn't to provide an external way to quickly check whether a userdata is userdata.unsupported, although it could be interpretted that way. Thus documentation would need to clarify that misunderstanding.

[nwinn-student]

Inflate is necessary for userdata, should it be externalized?

The same goes for the tag or byte right after the CUSTOM byte. Most userdata use that byte to distinguish between userdata. Note that there must be a byte after CUSTOM when deserializing since serialization does not allow for userdata to be solely defined as custom.

[nwinn-student]

Inflate is necessary for userdata, should it be externalized?

It could be provided in the serialize fn, but it is quite unintuitive for users that way.

bs.inflate doesn't make sense either.

So we just dismiss it for later.

[nwinn-student]

How would the function be morphed to include tag or id? (id: number, buf: buffer, pos: number) -> (any, number) may work, but it breaks current userdata code. I am willing to break the code, since it is minor to update.

If we don't do anything, we are forced to use big functions. I would like to be able to make many small functions so I can maintain them easily.

If we move id to the last, does it still break code? We send the position of the byte of the id, which doesn't make sense as the id is already obtained, but if it isn't supported it is better to just say to continue at the next position, which is pos + 1.

[nwinn-student]

We send the position of the byte of the id, which doesn't make sense as the id is already obtained, but if it isn't supported it is better to just say to continue at the next position, which is pos + 1.

I looked at the examples, we usually return pos after it has been processed by some other deserializer. Saying that we are at the byte after the first byte (id) makes sense, we need to ensure that the documentation makes that clear.

[nwinn-student]

I looked at the examples, we usually return pos after it has been processed by some other deserializer. Saying that we are at the byte after the first byte (id) makes sense, we need to ensure that the documentation makes that clear.

After further consideration, I will not be changing the position value. Some usage cases do not use tags and read the data immediately, thus the first position is necessary.

[nwinn-student]

The id problem is also present in serialize, but with typeof. And with both, there is no guarantee they will be used.

If a user really needs performance, they can use a single support function instead. Or minimize the amount of support functions.