Some way to import c-header-files directly would be a huge advantage

[porky11]

This may be done with c2ffi, (but I'm not experienced with it, nor json)
It could be implemented as a cto-function and macro
I need to know the functions anyway, but when I need foreign types and variables, where I don't have to know the exact values, nor all struct fields, I'd still have to rewrite them in dale for all libraries.
maybe even cl-autowrap can be copied without too much effort

[porky11]

I think, it may be easier to implement it as output driver for c2ffi, see this (untested, will not compile, currently only have older llvm installed):
https://github.com/porky11/c2ffi

[tomhrr]

Given that the existing output drivers for c2ffi are language-agnostic, it looks like it would be difficult to have it merged upstream. If a user's OS packages c2ffi, too, then it's more hassle for them to have to compile a custom version rather than using the existing one and passing its output to a Perl script.

[porky11]

Nice, most things should work now. Unions may be implemented as variants maybe, but I don't think, they are important.
Being able to use different files (on for types, one for constants, one for functions etc.) or as they were in the original files (additional using includes in the main file) may also be useful.
And also something like auto-renaming and using specific namespaces for libs would be nice (not sure, if this would be implemented in this perl program) (Something like this: glClearColor->gl.clear-color, glUniform3f->gl.uniform, glUniform1i->gl.uniform, thats what I have written defc for (#165))

[porky11]

One simple error I found:
typedef GLvoid void => (def GLvoid (struct extern ((a void))))
Solutions:
fix manually (not a clever solution, maybe better, if the type just doesn't get defined)
use macros instead of structs (easy, maybe even a better solution than structs, because casts won't be necessary that often anymore, but types don*t have to be used anymore)
only use macro for typedef void (difficult to know, in the funciotns using this type. Symbol-macros could fix this)

[porky11]

Some other problem:
In c, arrays as function parameters are valid (even if it may be bad style), in dale not.
In c, arrays as function parametere are implemented as pointers (remember #104), so in dale this could be converted to function pointers in this case.

[porky11]

I currently try to make a more complete opengl wrapper using your perl program: https://github.com/porky11/dale-opengl
some things still seem difficult (especially defines, but thats a problem with opengl, i think)

[tomhrr]

Thanks for raising this issue, the script should cover the main use cases well enough now. Some comments on specifics:

Unions may be implemented as variants maybe, but I don't think, they are important.

This has been implemented.

Being able to use different files (on for types, one for constants, one for functions etc.) or as they were in the original files (additional using includes in the main file) may also be useful.

Do you mean using the per-binding file information from the c2ffi data to produce separate output files?

And also something like auto-renaming and using specific namespaces for libs would be nice (not sure, if this would be implemented in this perl program) (Something like this: glClearColor->gl.clear-color, glUniform3f->gl.uniform, glUniform1i->gl.uniform, thats what I have written defc for (#165))

This has been implemented, via the --casing and --namespace options, except that it doesn't automatically derive overloaded functions.

typedef GLvoid void => (def GLvoid (struct extern ((a void))))

This is a tricky problem, as per your comments. Using macros just for void typedefs won't work, and using them for all typedefs will make the common case more annoying, as well as requiring parentheses for struct typedefs due to not being able to distinguish them from others. Given that this should be relatively uncommon, it skips defining them and leaves it to the user to fix, per your pull request.

[porky11]

I meant you can specify multiple files for different objects (one for functions, one for constants etc.), but I don't think, this is that useful (pcl-autowrap can do it, I think). Specifying output files seems useful enough which can also be done by multiple calls.
About void type: OpenGL has GLvoid. In my automatic opengl wrapper I replaced the types inside the perl program. Not because of the void problem but also becuase it's easier to use the original types since in c typedefs also work similar to macros, so operations on the original types can still be used and casting doesn't lead to problems.
(having to write (gl.uniform3f (GLuint ((value loc))) (GLfloat ((value 0.0))) …) instead of just (gl.uniform3f 0.0 0.0 0.0) is too complicated)

[porky11]

I found some problems:
With casing or namespace options enabled, even extern-c functions are only defined with the new name, which should mean, the original functions in the c lib won't be accessed by calling them.
You removed the way for only printing things from specific files i added. This means, if something like <stdlib.h> is included in the header file, all functions from stdlib.h are also included into the generated header file. Another option may be only including the new objects from namespace. (so if namespace is gl, only functions that start with gl will be included into the namespace gl). But this will often not work (for example if something starts with glut and should be in namespace glut, but not gl (not a realistic problem, since gl.h doesn't include glut.h))

[tomhrr]

Not sure how I missed the need to retain the original bindings :S. It should work properly now. The option to include only the bindings from a specified file has been restored, too, via the --file option.

[porky11]

some things don't work correctly

typedef struct S {int x;} S;
typedef union U {int x;} U;
typedef enum E {x} E;

I get this:

(import enum)
(import variant)

(def S (struct extern ((x int))))
(def S (struct extern ((a S))))
(def-variant U ((U-int ((value int)))))
(def U (struct extern ((a U))))
(def-enum E extern int ((x 0)))
(def E (struct extern ((a enum))))

Errors:

test.dt:5:8: error: struct 'S' has already been defined in this scope
test.dt:7:8: error: struct 'U' has already been defined in this scope
test.dt:9:27: error: type not in scope: 'enum'
test.dt:9:24: error: invalid type

Namespaces don't get expanded correctly:
often every form gets it's own namespace, but if there is a form, not starting with the prefix, it gets included into the namespace without removing something.
Example: library lib defines Functions prefixed with lib, types prefixed with Lib and constants prefixed with LIB_`

(namespace Lib
  (def X (struct …))
)
…
(namespace Lib
  (def Z (struct …)) 

  (def LIB_C1 …)
)

Other problem: when namespaces and casing is enabled, not everything gets cased the same way. For type-definitions, they prefix only is dropped at define position, sometimes casing is disabled. For variant sub-struct names casing and namespaces are disabled.

[tomhrr]

Thanks, these problems have been fixed (assuming I understand all the problems correctly).

[porky11]

Most important was the double struct problem, which lasts pretty long to fix manually.
Namespaces don't work correctly now. But removing namespace-prefixes is mostly also pretty easy in texteditor.
Changing casing is not that important. For most types using the original names seems the better option anyway (especially for Types).
So changing everything to the same casing doesn't seem that useful.

[BitPuffin]

Hey, can't macros in dale execute basically any code? If you can. You could write a C pre-processor in dale. And a C parser. And then you could use those to implement an include macro which processes and parses and generates declarations in dale? Seems like it would be neat to be able to just include C files in the language instead of running an external tool.

Afaik Terra does something like this.

[porky11]

Parsing directly in dale was also my first idea. Maybe that's better than using c2ffi, since some information isn't shown in its output.
But writing parsers in such a low level language probably is more difficult. Maybe there already are other good paresers for C written in C.

[BitPuffin]

I think since there is shared pointers etc it should be easy to use that to build some higher level DSL stuff which makes it feel fairly high level. You could probably basically implement scheme except without the eval.

But to be honest when I write parsers in higher level languages I miss pointer arithmetic.

[BitPuffin]

For optimal usability you'd really kind of need to parse the header and transpile it. Since they could sometimes contain static function definitions or C macros. So you'd also need to be able to translate that into dale lol. But You could probably also just skip them since that's rare. And in that case a Dale programmer can translate by hand.

[porky11]

I also thought, it would be possible to implement some real lisp (i.e. CL) in dale. Most things should be implementable in dale itself, reader macros would need parsing, which could also be done in dale, but not (that easily) be used in the same file.
The only case, where I use pointer arithmetic is iterating over an array, but this is also possible in heigh level languages using iterators or similar. Where I would use it for parsing, I don't know.
In most cases it is enough to parse #defines that define variables.

[porky11]

And writing eval should also be possible. There is a cto function called eval-expression, so it's possible to add a repl with something like this

(def eval (macro intern (void)
  (let ((a \ (parse-into-sexpression)))
    (eval-expression mc (q int) …(qq fn int (void) (uq a) 0)))
  (q (eval)))

[BitPuffin]

Yeah compile time eval is possible. I just meant that without implementing an actual interpreter that you embed in your program, you can't have runtime eval that modifies the entire dale program. It would sort of live a little bit outside your dale program, but you could still use compile-time introspection to expose basically all of your program's API to an interactive environment!

Well, in C, when you parse a string you can just use pointer arithmetic while iterating over, and if you have a function like startsWith you can just pass the pointer there and it will work just like a string, you don't have to treat it separately. Recently in .NET I had to implement my own ParserString which is a string with an offset. And implement helper functions like that. I can't use the StartsWith method that is built in to the actual string class. What I had done earlier as a bad hack was to create substrings, but of course it's not nice to generate a bunch of garbage by copying substrings so much.

But yeah I would like to have reader macros in dale itself. Lisps quasi quotation syntax is more pleasant than qq. So it would be neat to be able to make a quasi quote reader macro that expands to qq.

[porky11]

When I wrote a parser in C (or when writing read macros in CL) I just used read-char, peek-char and unread-char, but for more complicated parsers this may probably become too complicated.

Wouldn't it be preferable to define some Substring class, which contains two indices and a string reference, instead of copying?

I like that dale itself doesn't have reader macros, so it's easier to use it as IR, that's easy to generate.
In CL most reader macros do more complicated things than expanding into an sexpression. Quasiquote gets expanded to something with list or list* in most cases, and vectors will be converted to vectors at read time. But I think, this should be discussed in some different issue

[BitPuffin]

Well in some parsers you wanna be able to backtrack when you failed parsing. And a nice way to do this is to just save the pointer for the current state, try to parse, and then if you fail set the value back to where it starter.

For the thing you mentioned that's basically what I did in .NET but with only an offset from the beginning since I don't need to offset from the end of the string while parsing.

I don't see how having reader macros in Dale would make it harder to use it as IR. And nor is the purpose of Dale to be IR to the best of my knowledge.

[porky11]

I just thought, it would also be good as IR for other languages, since generating sexpressions is easier than generating llvm or similar directly.

[BitPuffin]

I mean that's fine. But having reader macros in dale would be a none issue for that use case. Since you would still have the ability to output code that is without reader macros.