Dynamically resolved c functions

[roastpiece]

In response to @rexim's video on umka-lang, i am trying to implement dynamically resolving c functions, without requiring explicit bindings from c.

I managed the following POC working. An example can be found at (https://github.com/roastpiece/umka-ffi-example).
It takes the mentioned approach, by searching the running executable for matching symbols dlopen(NULL, ...).
It now loads the symbols from a mod.umi file (eg: raylib_linux.umi for a raylib.um).

I added a new keyword extern ffi to mark prototype fn declarations to be dynamically resolved.
So one could do:

ffi fn foo(num: int);

fn main() {
    foo(69);
}

• Libffi integration
• Structures supported for args & return value
• Now should run on all supported platforms, just need to fix the build scripts, will test on windows sometime
• Constructing fn prototype declarations from C Headers
• Get rid of keyword somehow?

There are some problems atm:
No type checking is done on the C side at the moment. (I'm actually not sure if this would be possible).

I'm opening this as a Draft atm to get some feedback and to see, if I'm on the right track and I should continue spending time on this.

[vtereshkov]

@roastpiece This is very interesting as an experiment. However, I don't consider it to be part of the Umka interpreter, for the following reasons:

• Portability: Umka should build with any C99 compiler (GCC, Clang, MSVC, MinGW) and run on any 32-bit or 64-bit platform having enough RAM: Windows (x86, x86-64), Linux (x86-64), macOS (x86-64, ARM), WASM
• Security: Calling functions only by name, without checking their signatures, is insecure. Even when you're adding a new C function via umkaAddFunc(), you must provide the function's prototype in a module contained in a string buffer rather than a physical file, to reduce the risks of the prototype being modified
• Consistency: Umka already has external functions whose prototypes are declared without any specific keywords. You're introducing some other sort of external functions not compatible with the existing ones and marked as extern. How are they related to all the other functions? Can I store an extern function in a function-type variable? Can I use it as a closure? How are its parameters reference-counted?
• Complexity: To implement FFI properly, you'll need an amount of code comparable to the whole Umka interpreter

I would be happy to see Umka FFI as a separate project (UMI?), but not Umka itself.

[roastpiece]

Thank you for your quick response.

@roastpiece This is very interesting as an experiment. However, I don't consider it to be part of the Umka interpreter, for the following reasons:

• Portability: Umka should build with any C99 compiler (GCC, Clang, MSVC, MinGW) and run on any 32-bit or 64-bit platform having enough RAM: Windows (x86, x86-64), Linux (x86-64), macOS (x86-64, ARM), WASM

Yes, my implementation was more of an exercise for me, to implement the SYSTEM V calling convention. I switched to using libffi, which adds a dependency, but implements almost every ABI imaginable.

• Security: Calling functions only by name, without checking their signatures, is insecure. Even when you're adding a new C function via umkaAddFunc(), you must provide the function's prototype in a module contained in a string buffer rather than a physical file, to reduce the risks of the prototype being modified

I thought about this. Is there a way to check the signature at runtime, without the header files? If not, i think the only way would be to generate the umka prototypes during build time. But that would goes against the dynamic nature of ffi.

• Consistency: Umka already has external functions whose prototypes are declared without any specific keywords. You're introducing some other sort of external functions not compatible with the existing ones and marked as extern. How are they related to all the other functions? Can I store an extern function in a function-type variable?
  Can I use it as a closure? How are its parameters reference-counted?

Yes i saw that in the code while implementing this. It can be changed of course. Wasn't sure what to call it. I was thinking of extern "C" when i chose that name.
Regarding the other questions: I am still in the process of undertanding how those parts of umka work. I (think I), declare the functions the same way any other function is declared. I basically copied the calling code, of doCallExtern and added a new instruction which calls doCallExternDynamic, which calls the function (resolved during doResolveExtern, where i hooked into the generation).

[vtereshkov]

@roastpiece FYI, there is a related project being discussed on Discord:

https://discord.com/channels/846663478873030666/1441947520468127916

[ske2004]

@roastpiece I think to get rid of the keyword you can look it up in the symbol table as last resort if function's prototype wasn't resolved. This happens with UMI functions already as far as I'm aware.

[vtereshkov]

@ske2004 These new external functions are not compatible with the conventional fn () type.

[ske2004]

@ske2004 These new external functions are not compatible with the conventional fn () type.

Oh right, I didn't consider this. Would it be possible to simply forbid taking reference of them?

[roastpiece]

At the moment the keyword is needed to differnetiate how to call the function.
If it is marked with the keyword, it gets called directly through libffi, with the arguments defined in the fn prototype.
other wise it will be called as an UmkaExternFunc, as it is happening now.

One way i think i can get rid of the keyword, is by loading the dll/so directly as a seperate module. then if the fn is found in a umi, it gets called the same way as it is now, otherwise it looks in the dll, and calls it directly.