Asynchronous usage

[SamMousa]

I'm aware of the discussion in the open PR #6, this ticket is here to have the discussion without referring to a specific implementation.

One thing you mention is that using coroutines is not a preferred solution, could you elaborate on why not?
Coroutines have low overhead (similar to function calls), and are easy to implement.

I have created a proof of concept PR for LibSerialize that shows how it can be done transparently for the calling code. (rossnichols/LibSerialize#7)

The idea is that the library yields at given byte intervals when it detects it is running inside a coroutine.

[SafeteeWoW]

1. It could be solved by WA, by splitting communications into chunks.
2. You could ask WA to patch LibDeflate with coroutine, shouldn't be very hard.
3. I do not like coroutine. It should be implemented by streaming interface, APIs similar to zlib
4. I am not actively maintaining this library. Mostly because I do not play this game any more.
5. Maybe it's time to setup my IDE for this tiny project and start to work on it again?

[SamMousa]

I know you don't like coroutines, can you elaborate on why? I'm curious!

[hehaoqian]

Ok. I may allow the user to pass in a hook function.
LibDeflate will call it in the middle of compression or decompression.
LibDeflate does not care what the hook function does.
The hook function can call coroutine.yield(), for example

[SamMousa]

Can someone explain the issues people are having with coroutines to me?

[Meorawr]

Can someone explain the issues people are having with coroutines to me?

Coroutines add a lot of complexity and risk breaking invariants in-place throughout a lot of code.

The first - and most critical - issue with respect to coroutines is that they're stackful. A thread can be suspended at any point within any nested stack frame. Within the WoW environment we're on Lua 5.1 which isn't a fully resumable VM and as such you can't actually yield across a few boundaries.

• You can't yield across any stack frame above yours that belongs to a C function.
• You can't yield from within the iterator y of a for x in y loop.
• You can't yield across any stack frame that represents an invoked metamethod on a table.

The problem is you can't actually reasonably guarantee that none of these invariants aren't actually broken at runtime, either by another loaded addon being present or you making seemingly-innocuous changes elsewhere in your own code. Something as simple as using pcall to capture errors or a secure hook being present on any function in the stack frame above where you're yielding from would break the first invariant for example, as the implementation of those involves a trip through C and back into Lua.

Further, most Lua code is assumed to only ever be called from a single-thread which leads to decisions that are made - either knowingly or accidentally - that prevent re-entrancy.

A simple example would be a function that, to conserve memory, reuses a table stored in an upvalue for processing and polls a user-supplied function to provide some data (such as an iterator). If this iterator recursively executed the function, you'd typically blow up spectacularly then and there with a stack overflow error which makes it extremely obvious and easily fixed. If the iterator yields and another separate thread takes a trip into this function, you've got a subtle bug that's much harder to detect.

This isn't to say that coroutines don't solve problems, but it's generally more reliable to use them in a stackless - or at least near-stackless - manner and to try not go all-in with them, or in the case of libraries, "force" them onto the user.

With respect to LibDeflate (and LibSerialize), an iterative streaming API as mentioned would be the best solution from a least-astonishment perspective. Such an API wouldn't force the use of coroutines with all their complexities and risks on the caller, but also doesn't prevent the caller from processing the stream within a coroutine if they so choose.

[SamMousa]

Thanks! That's a very interesting read!

[SafeteeWoW]

I will provide a feature for hook function.
User can use coroutine.yield inside the hook function.
coroutine API will not be used directly in libdeflate