PerfResults Tool

[nwinn-student]

With the introduction of the Benchmark tool discussed in #53, there needs to be discussion about the results and what should be shown.

We should have two separate visuals for the results. One for the performance (time and space) and the other for the output size.

The performance isn't as important, since it is implementation specific and device specific. We still should still show the performance implications.

For output size:

• We should show how compressible the serialized form is using various compressors.
• We should show how the paired form compares to the compressed form, and how compressible it is.

Am I missing anything vital?

[nwinn-student]

What about comparing performance against prior versions or other implementations of formats?

[nwinn-student]

We should have two separate visuals for the results.

What should the json file used to generate the results contain?

[nwinn-student]

Size can be shelved for later or an entirely different tool.

[nwinn-student]

We could do:

--serialize
|---base
|---pair
|---schema-pair

I think we should instead just have pair be the same as schema-pair. So --base, --pair.

I wonder though, some versions tested may not have pairs supported, so the metadata approach will be better. We can also just adjust the generating cli based on the version. I think that the --pair option already exists internally, just isn't used.

I've chosen to just bake them together.

[nwinn-student]

I've chosen to just bake them together.

By that, I mean that I chose the easiest path that involved downloading fewer files. So instead of downloading the schemas and pairing each word, I have chosen to look at the word frequency (5+ usages to be paired). I also chose storing pair benchmark data in metadata, since generating additional files will make luau-analyze take even longer.

I am not generating scripts on a per-version basis. That requires too much work, so I generalized version differences and put them in the scripts.

[nwinn-student]

Removing pairs, so the format can be simplified.

[nwinn-student]

For comparing we only need time and space differences, but output size may differ between versions but it is optional.

name	time	space	size
...	+N	+M	+0

For size we can do something similar to the json category size reference.

Name

format	size
bs	N
bs-pair	M
compression..	Z

[nwinn-student]

For pairs, we can do the same.

Optimized time (+N%)
Optimized space (+N%)
Optimized size (+N%)
Optimized pair time (+M%)
...
Regressed time (-N%)
...

Per Category (base)

time	space	size
...	...	...

Per Category (pair)

time	space	size
...	...	...

I have split the optimize and regress M into two sections, one for serialize and one for deserialize. I have also ensured they are consistently ordered. Should any of the values be 0%, they will not be displayed.

For Per Category, each category is noted and has a corresponding optimize and regress M. Again, Per Category is sorted and split into serialize and deserialize.

Per Category

Category	Results
tier 1..	Optimized time (N%) Regressed space (N%) ...

[nwinn-student]

For size we can do something similar to the json category size reference.

Name

format	size
bs	N
bs-pair	M
compression..	Z

We could want to make it easier to understand visually and for analysis. I am used to a csv approach where we have datasets, format, and size as headers. For each dataset there are 3 formats, and for each format there is 1 size.

Datasets	Format	Size
circle	bs	N
circle	bs-pair	M
circle	compression..	Z

Note that there may be more, since compression has 2 values. So it could be 4 formats.

[nwinn-student]

For Per Category, each category is noted and has a corresponding optimize and regress M. Again, Per Category is sorted and split into serialize and deserialize.

On mobile or smaller screens, the readability is horrible. Do we take mobile/smaller screen experiences into account?

[nwinn-student]

Do we take mobile/smaller screen experiences into account?

Tables will always look or feel horrible on mobile. If we do a columnar approach, users have to scroll to the right. If we use text, it will look less optimal for larger screens.

• category..
  • optimize time (+N%)
  • ...

Bullet points also don't work that well for smaller screens. But maybe they are decent enough.

It would be ideal to separate smaller screens and larger screens and what they see. If there is a way to do that in markdown that would be nice.

[nwinn-student]

We should have categorical size comparisons alongside specific size comparisons, like vectors and referenced tables (cyclic or not).

For time/space we should have databases based on the components, categorizing them as such. We can then say that numbers are N% faster to serialize.