Skip to content

Conversation

joelpaulkoch
Copy link

This PR moves the sqlite_vec package into this repository.

Here the remaining open points:

License

I'm guessing you would want to copy the Apache and MIT license from the root repository? In any case, after updating the license, we must make sure it's included in the published package on hex.

Transfer ownership of the package on hex

I can transfer the ownership of the package on hex.pm. If you give me the go, I'll do that with this user.

CI

I've included my github workflows file, not sure what you want to do with that.

@MiraiTunga
Copy link

Can this be merged?

@joelpaulkoch
Copy link
Author

You can use the package as git dependency in mix: https://github.com/joelpaulkoch/sqlite_vec.

I want to refactor the PR so that it doesn't depend on octo_fetch. Instead we can use elixir_make, this way we don't have a runtime dependency and it will integrate better with the ecosystem (cross compilling with burrito, probably also nerves).
I expect this to work pretty good given that we can just reuse the existing Makefile, sqlite-vec.c and sqlite-vec.h.

@asg017
Copy link
Owner

asg017 commented Sep 29, 2025

@joelpaulkoch lmk if I can help with that in anyway! A no-dep Makefile option would be awesome

@joelpaulkoch
Copy link
Author

So, here's the updated PR.
A few notable points:

  1. it's not no-dep, but elixir_make is a compile time only dependency, and pretty established (as you can see by the fact that it's in the elixir-lang org on github). It's for instance used by burrito to detect a native library and recompile it when doing cross compilation (that is to say, there are benefits to have elixir_make as dependency compared to manually calling make).
  2. To compile the library from the Makefile, we must have Makefile, VERSION, sqlite-vec.c and sqlite-vec.h.tmpl as part of the published package. I solved that by adding symlinks and a publish script which replaces the symlinks with the actual files before publishing the package.
  3. I had to update the Makefile so that the install prefix can be set from env.
  4. This approach requires that users have make and a c compiler available. There is the concept of precompilers in elixir_make that check for existing binaries to download before compilation but I think it's not quite fitting the scenario here where the native library is not only made for Elixir. So maybe we can later add a step to download from github releases and make the compilation with make and c compiler optional.

@asg017
Copy link
Owner

asg017 commented Oct 2, 2025

Thank you @joelpaulkoch for your hard work here!

One ask — instead of relying on the Makefile + sqlite-vec.h.tmpl from this project, can we instead use the "amalgamation" build, that has pre-processed sqlite-vec.c and sqlite-vec.h files? Here's a sample: https://github.com/asg017/sqlite-vec/releases/download/v0.1.7-alpha.2/sqlite-vec-0.1.7-alpha.2-amalgamation.zip

Hopefully that will mean we don't need to include Makefile/VERSION in here, and maybe avoid the need for the symlinking

Also — do you have a link to docs that talk about the "precompilers" for elixir make? I ask because the sqlite-dist project is for pre-compiling/bundling SQLite extensions for various platforms, so it might be possible to include elixir_make precompiled support in there (which sqlite-vec uses in it's publishing process)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants