Nix packaging requirements

[happysalada]

What type of enhancement is this?

User experience

What does the enhancement do?

First of all thank you for making greptimedb.

I'm trying to package this for nixpkgs (the package repo for nixos).

nixpkgs uses cargo vendor to build rust packages. Greptime-db has a dependency on a custom sqlparser 0.54, but also it uses several packages that depend on the vanilla sqlparser 0.54. it's challenge for the nix build. after checking, it seems all the custom commits of sqlparser 0.54 that are used have already been merged upstream. Therefore upgrading to the latest release 0.56 should have a single sqlparser version for the whole repo.

That would in turn enable packaging for nixpkgs.

Implementation challenges

I'm not sure what it entails to update sql parser to the latest stable.

[evenyag]

@sunng87 Do you have any ideas?

We have some custom syntax extensions (RANGE QUERY) on the parser that are hard to port to the upstream.

[sunng87]

tbh I know little about making package in nix. In this repo we use nix flake for managing development environment but it's a different story when it comes to packaging.

I think at least for now we still need to use custom sqlparser. Do you think there will be a solution for this? Also we may switch to customized datafusion at any time when there is a gap with upstream so I'm not sure if we can use all upstream mainline in a stable manner.

[happysalada]

If the version used in the custom parsers where different, then it would work i think.
Instead of keeping the same version if when using a custom version you incremented the patch number that would work.

Otherwise we could make a patch ourselves to do just that.

Last thing is we could wait for things to stabilize a bit.

Personally im not in a rush, i just wanted to release the package early to hopefully get help when i make a systemd service for it.

[happysalada]

if you have time to do the sqlparser update, feel free to post here.
In the meanwhile, I'll try again to package on every minor version upgrade.
Thank you for your time spent answering this, and thank you for making greptimedb!

[sunng87]

@happysalada Thank you. I'm curious when building rust based projects in nix, do you need all dependencies managed by nix instead of cargo? Another question, when building nix package for greptimedb, will you start from our current flake.nix? Is it possible the make the nix config works for development environment setup as well as packaging?

[happysalada]

Actualky yes, another way is to extend the current flake to add a package. Usually yes you need to let nix handle the build, but there are tools like crate2nix that help with that.
The only benefit of having something in nixpkgs is the visibility, but since the ecosystem is moving towards flakes anyways, and it looks like greptimedb will need a custom sql parser for sometime, that might be a better approach.

The best would be to add a service to the flake which would be a systemd service to run on nixos.

Ill try to have a look at something i did before to see if i cant reuse some parts.

[happysalada]

ok, after more consideration, here are the benefits I see in having greptimedb in nixpkgs.

• Better visibility for users, you can find greptimedb in the already used nixpkgs repo.
• Users can use nixpkgs binary cache (you can download the full built binary without having to build from source on your own system). This is a big plus in my opinion as getting the package from the flake would not have this benefit. This would mean time spent building greptimedb everytime someone updates their system.
• This goes with the first argument, but by having the a nixpkgs nixos service, you might get more feedback from users, as trying greptimedb out would just be to enable a switch. Users wouldn't have to add a custom flake to their setup.
• Maintaining a flake can be costly, you need to run it in the cicd to make sure nothing breaks. Having broken ci for a while in the repo is also not great (nix related packages can break from time to time due to random things). Whereas you can leave an older version in nixpkgs that builds until things are resolved on the latest version.

That being said, think also, the changes to greptimedb to enable nix build should be minimal.
I've made a proof of concept PR here #6111
Basically you just need to increment the patch version of the custom dependencies.
If you are open to this, I'm happy to make the relevant PRs in the different repos (I'm happy if you just push the changes directly yourself, I don't need any attribution).

Let me know what you think.

[sunng87]

Thank you @happysalada I totally agree having a greptimedb package in nixpkgs is great.

Basically you just need to increment the patch version of the custom dependencies.

But if the upstream of custom dependencies released same patch version, will there be a conflict?

[happysalada]

But if the upstream of custom dependencies released same patch version, will there be a conflict?

then you have two choices depending on if you want to rebase or not.

• If you rebase, you need to change the original patch update to increment one patch more.
• If you don't rebase, you can just increment the patch version again.

The version of the original crate and the fork just need to be different, that is it.

[sunng87]

I see. We can include a build identifier like 0.56.0+greptime in forked repo.

https://semver.org/#backusnaur-form-grammar-for-valid-semver-versions

[happysalada]

If cargo supports this, then its even better.
I already have a nix expression ready .
Im happy to test packaging as soon as there is a stable release with these changes.
Let me know what works best for you and i can help wherever needed.

[sunng87]

Yes I confirm the trick works. The build identifier is accepted by cargo.

[happysalada]

Perfect, let me know when things are ready and ill open a pr on nixpkgs. No rush on my side.