modules/nixos: add comin

[zowoq]

Polling the git repo looks much nicer than autoUpgrade that we're currently using.

I'd like to trial it on build01 but need to address a couple of thing first.

[nlewo]

That could still be a bit early since i didn't do any release yet. So i was still feeling free to break interfaces ;)

But do not hesitate to create issues with features you would like to see.

[Mic92]

How does it make deployment failures visible? Does it make the service fail? I also made the experience that many service upgrades failures can be mitigated by just adding one retry of nixos-rebuild. So it would be cool if it does that.

[nlewo]

How does it make deployment failures visible?

I'm planning to provide a Prometheus metrics. Currently, comin reports its status over http and when the head_commit_deployed attribute is false, it means an issue occurred during the deployment.

Does it make the service fail?

No (it is a daemon).

I also made the experience that many service upgrades failures can be mitigated by just adding one retry of nixos-rebuild. So it would be cool if it does that.

Noted (that should not be hard to add this option).
I also had planned to address this kind of needs by adding the force=true parameter to the http route /deploy (currently used by webhooks).

[zowoq]

We talked about deployments a bit more recently and looks like we'll switch cachix deploy as they going to add telemetry and have support for nix-darwin.

Sorry for the noise here @nlewo.

[nlewo]

@zowoq no worries and thank you for your explanations.
(Moreover, comin is definitively not ready for being used by others. Sometimes, i "forget" to publish my softwares... but for the first time, i published it too early :/)

[zowoq]

The cachix deploy telemetry didn't end up working out so I'd like to try this now that it has had a release.

Would need to use http for the status but that doesn't look to be hard to add to our monitoring.

{
  "Generation": {
    "BuildEndedAt": "0001-01-01T00:00:00Z",
    "BuildErr": null,
    "BuildStartedAt": "0001-01-01T00:00:00Z",
    "DrvPath": "",
    "EvalEndedAt": "0001-01-01T00:00:00Z",
    "EvalErr": null,
    "EvalMachineId": "",
    "EvalStartedAt": "0001-01-01T00:00:00Z",
    "OutPath": "",
    "RepositoryStatus": {
      "Error": null,
      "main_branch_name": "",
      "main_commit_id": "",
      "main_remote_name": "",
      "remotes": null,
      "selected_branch_is_testing": false,
      "selected_branch_name": "",
      "selected_commit_id": "",
      "selected_commit_msg": "",
      "selected_remote_name": ""
    },
    "Status": 0
  },
  "deployment": {
    "end_at": "0001-01-01T00:00:00Z",
    "error_msg": "",
    "generation": {
      "BuildEndedAt": "0001-01-01T00:00:00Z",
      "BuildErr": null,
      "BuildStartedAt": "0001-01-01T00:00:00Z",
      "DrvPath": "",
      "EvalEndedAt": "0001-01-01T00:00:00Z",
      "EvalErr": null,
      "EvalMachineId": "",
      "EvalStartedAt": "0001-01-01T00:00:00Z",
      "OutPath": "",
      "RepositoryStatus": {
        "Error": null,
        "main_branch_name": "",
        "main_commit_id": "",
        "main_remote_name": "",
        "remotes": null,
        "selected_branch_is_testing": false,
        "selected_branch_name": "",
        "selected_commit_id": "",
        "selected_commit_msg": "",
        "selected_remote_name": ""
      },
      "Status": 0
    },
    "operation": "",
    "restart_comin": false,
    "start_at": "0001-01-01T00:00:00Z",
    "status": 0
  },
  "hostname": "build01",
  "is_fetching": false,
  "is_running": false,
  "repository_status": {
    "Error": {},
    "main_branch_name": "",
    "main_commit_id": "",
    "main_remote_name": "",
    "remotes": [
      {
        "fetched": true,
        "fetched_at": "2024-03-07T00:57:43.502544262Z",
        "main": {
          "error_msg": "The branch 'origin/fail' doesn't exist",
          "name": "fail"
        },
        "name": "origin",
        "testing": {},
        "url": "https://github.com/nix-community/infra.git"
      }
    ],
    "selected_branch_is_testing": false,
    "selected_branch_name": "",
    "selected_commit_id": "",
    "selected_commit_msg": "",
    "selected_remote_name": ""
  }
}

time="2024-03-07T04:48:38Z" level=info msg="The manager is started"
time="2024-03-07T04:48:38Z" level=info msg="  hostname = build01"
time="2024-03-07T04:48:38Z" level=info msg="  machineId = 2a7f74e3b4f6409eb031a1e606fff1bc"
time="2024-03-07T04:48:38Z" level=info msg="Starting the poller for the remote 'origin' with period 300s"
time="2024-03-07T04:48:38Z" level=info msg="  repositoryPath = /var/lib/comin/repository"
time="2024-03-07T04:48:38Z" level=info msg="Starting the webhook server on 127.0.0.1:4242"
time="2024-03-07T04:48:39Z" level=info msg="New commits have been fetched from 'https://github.com/nix-community/infra.git'"
time="2024-03-07T04:48:39Z" level=info msg="Running 'nix --extra-experimental-features nix-command --extra-experimental-features flakes show-derivation /var/lib/comin/repository#nixosConfigurations.build01.config.system.build.toplevel -L'"
warning: 'show-derivation' is a deprecated alias for 'derivation show'
Pass '--accept-flake-config' to trust it
warning: ignoring untrusted flake configuration setting 'extra-trusted-public-keys'.
Pass '--accept-flake-config' to trust it
time="2024-03-07T04:48:50Z" level=info msg="The derivation path is /nix/store/7xd89npijmy73y286zzkw25qd379g7p7-nixos-system-build01-24.05.20240306.edf9f14.drv"
time="2024-03-07T04:48:50Z" level=info msg="The output path is /nix/store/zjvs12b8bc7x2hvmcq9rzmpdalr67yb0-nixos-system-build01-24.05.20240306.edf9f14"
time="2024-03-07T04:48:50Z" level=info msg="Running 'nix --extra-experimental-features nix-command --extra-experimental-features flakes eval /var/lib/comin/repository#nixosConfigurations.build01.config.services.comin.machineId --json'"
Pass '--accept-flake-config' to trust it
warning: ignoring untrusted flake configuration setting 'extra-trusted-public-keys'.
Pass '--accept-flake-config' to trust it
time="2024-03-07T04:48:51Z" level=info msg="Evaluation error: The defined comin.machineId (2a7f74e3b4f6409eb031a1e606fff1bc) is different to the machine id () of this machine"
time="2024-03-07T04:49:16Z" level=info msg="Getting status request /status from 127.0.0.1:40056"

[nlewo]

time="2024-03-07T04:48:51Z" level=info msg="Evaluation error: The defined comin.machineId (2a7f74e3b4f6409eb031a1e606fff1bc) is different to the machine id () of this machine"

This is an issue in comin which should be fixed by nlewo/comin#14.

Would need to use http for the status but that doesn't look to be hard to add to our monitoring.

Which metrics would you like to collect?
In case you use Prometheus, i created nlewo/comin#9 to collect a list of useful metrics. Maybe that could fit your need better than the unstable status output.

[zowoq]

This is an issue in comin which should be fixed by ...

Thank you!

Which metrics would you like to collect?

I think a status/exit code of the most recent deployment and the commit of the last successful deployment would be enough for us.

In case you use Prometheus ...

Yes, we're using prometheus so metrics in that format would be excellent.

[zowoq]

I'll test this on build01 for a bit and also so I have a live machine to work on while I figure out the the http status monitoring.

Follow up in #1166.