JSONValue should represent integers with a string

[schveiguy]

Instead of storing the json number value as an integer or float, store it as a string, and then let the user decide how to read it.

This could potentially cause problems if the json stream is transient, meaning we would have to make a copy of the string.

Potentially, if we know the string is an integer and fits within an int, then we could eagerly parse it.

Otherwise, we can store numbers using a SSO string. Currently the union contains a string, which is 16 bytes. So a 16 byte character array is possible without allocating.

[Aditya-132]

hello sir i want to work on this issue could you assign me this issue

[schveiguy]

Hi @Aditya-132! Sorry this slipped passed my attention for a while. If you want to work on this issue please give me a layout of a plan of how you will implement this. I had something in mind, but am interested to hear your thoughts on it! I don't want you to spend too much time on it if it's not what I'm looking for.

If you'd rather, I can lay out what I think should happen in more detail.

[Aditya-132]

i dont know is my plan 100% correct
this is brief of what i thought
Define JSONValue:
Create a JSONValue struct with fields: JSONToken token, string value (for the string representation), and JSONParseHint hint.
Store number data as a string (e.g., "123", "3.14") instead of parsing it into int or float.
Integrate with JSONTokenizer:
Update JSONTokenizer.next() to return JSONValue instead of JSONItem.
Convert JSONItem to JSONValue by copying the number’s string data from the chain (e.g., item.data(chain).idup).
Handle Transient Streams:
Use .idup to copy the string data, ensuring it’s safe if the buffer changes.
Optionally, implement Small String Optimization (SSO) with a fixed-size char[16] buffer to store small numbers without allocation.
Optional Eager Parsing:
For numbers with hint == JSONParseHint.Int, optionally parse and store an int value (e.g., eagerInt) if it fits, alongside the string.
User Parsing:
Add a parse!T() method to JSONValue (e.g., parse!int, parse!float) to let users convert the string to their desired type.

[schveiguy]

OK so let me expand on my notes from above, as this was a TBD note to myself originally.

There already is a JSONValue, it's a basic tagged union, which stores the converted type from the parsed data. See here:

jsoniopipe/source/iopipe/json/dom.d

Line 22 in 959ddd2

struct JSONValue(SType)

There is no need to adjust JSONItem, this is a combination of the token enumeration, and the portion of the iopipe which contains that token (along with hints of what is detected in this item) This serves its purpose perfectly.

But a JSONValue stores the value as its native D type. This is problematic for a number, since a valid JSON number might not be representable as a native D type.

For example, the number 10000000000000000000 is not representable as a `long'. This means when it is being parsed, it will throw an exception during parsing.

However, this is a perfectly valid number in json grammar.

What I was thinking is that we can represent any number as a string, and then parse it on demand with the type needed. Perhaps the user is aware that the number might be larger than a long, and parses it using BigInt. Perhaps he knows it's going to fit into a ubyte.

Representing as a string gives us maximum flexibility, and also doesn't have edge case errors.

What I think should be done:

1. Use small string optimization to store any string data in string.sizeof bytes (more on that later)
2. If the string data doesn't fit, allocate a duplicated string.
3. Change numeric type to be also stored via string.
4. add a simple get(T)() => this.stringForm().to!T accessor.

I'm not sure at the beginning whether we need to limit the T there, as custom number types make perfect sense for some use cases. It can also work with strings that are in the stream as well.

For the SSO, this is what I propose:

union {
   char[16] sso;
   string allocatedString;
}

For the SSO, you can have 2 new types in the enum for each flavor that might be stored as a string, one for SSO where the full 16 chars are used. And one for SSO where the first byte indicates how many bytes are used in the string.

I also am OK getting rid of the template parameter on JSONValue, and just always using string.

Let me know if you have questions.

[Aditya-132]

making PR till tomorrow