Error with pyarrow

[akaihola]

Describe the issue

If pyarrow is in [tool.poetry.dependencies], nix build fails with error: value is a set while a string was expected.

Additional context

To reproduce as simply as possibly, I added pyarrow = "^16.1.0" to pyproject.toml in poetry2nix/templates/app/ and ran nix build.

Output from `nix build --show-trace`:

error:
       … while evaluating the attribute 'drvPath'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/customisation.nix:365:7:

          364|     in commonAttrs // {
          365|       drvPath = assert condition; drv.drvPath;
             |       ^
          366|       outPath = assert condition; drv.outPath;

       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'python3.11-app-0.1.0'
         whose name attribute is located at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'propagatedBuildInputs' of derivation 'python3.11-app-0.1.0'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:385:7:

          384|       depsHostHostPropagated      = elemAt (elemAt propagatedDependencies 1) 0;
          385|       propagatedBuildInputs       = elemAt (elemAt propagatedDependencies 1) 1;
             |       ^
          386|       depsTargetTargetPropagated  = elemAt (elemAt propagatedDependencies 2) 0;

       … while calling anonymous lambda

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:321:13:

          320|       (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHostPropagated" depsHostHostPropagated))
          321|       (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "propagatedBuildInputs" propagatedBuildInputs))
             |             ^
          322|     ]

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:321:18:

          320|       (map (drv: getDev drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHostPropagated" depsHostHostPropagated))
          321|       (map (drv: getDev drv.__spliced.hostTarget or drv) (checkDependencyList "propagatedBuildInputs" propagatedBuildInputs))
             |                  ^
          322|     ]

       … while calling 'getOutput'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1796:23:

         1795|   */
         1796|   getOutput = output: pkg:
             |                       ^
         1797|     if ! pkg ? outputSpecified || ! pkg.outputSpecified

       … while calling anonymous lambda

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/lists.nix:334:29:

          333|   */
          334|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                             ^
          335|

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/lists.nix:334:32:

          333|   */
          334|   imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
             |                                ^
          335|

       … while calling anonymous lambda

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:276:15:

          275|     imap1
          276|       (index: dep:
             |               ^
          277|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/stdenv/generic/make-derivation.nix:277:12:

          276|       (index: dep:
          277|         if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep
             |            ^
          278|         else if isList dep then checkDependencyList' ([index] ++ positions) name dep

       … while calling 'isDerivation'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1283:5:

         1282|   isDerivation =
         1283|     value: value.type or null == "derivation";
             |     ^
         1284|

       … while calling 'checkDrv'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/mk-python-derivation.nix:206:26:

          205|
          206|     checkDrv = attrName: drv:
             |                          ^
          207|       if (isPythonModule drv) && (isMismatchedPython drv) then throwMismatch attrName drv

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/mk-python-derivation.nix:207:11:

          206|     checkDrv = attrName: drv:
          207|       if (isPythonModule drv) && (isMismatchedPython drv) then throwMismatch attrName drv
             |           ^
          208|       else drv;

       … while calling 'isPythonModule'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/mk-python-derivation.nix:42:20:

           41|
           42|   isPythonModule = drv:
             |                    ^
           43|     # all pythonModules have the pythonModule attribute

       … while calling anonymous lambda

         at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/default.nix:50:13:

           49|           (
           50|             dep:
             |             ^
           51|             let

       … while calling 'func'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/passthrufun.nix:26:20:

           25|       valid = value: pythonPackages.hasPythonModule value || providesSetupHook value || lib.elem value exceptions;
           26|       func = name: value:
             |                    ^
           27|         if lib.isDerivation value then

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/passthrufun.nix:27:12:

           26|       func = name: value:
           27|         if lib.isDerivation value then
             |            ^
           28|           lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value

       … while calling 'isDerivation'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1283:5:

         1282|   isDerivation =
         1283|     value: value.type or null == "derivation";
             |     ^
         1284|

       … from call site

         at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2230:17:

         2229|
         2230|       pyarrow = prev.pyarrow.overridePythonAttrs (
             |                 ^
         2231|         old: lib.optionalAttrs ((!old.src.isWheel or false) && lib.versionAtLeast old.version "0.16.0")

       … while calling 'overridePythonAttrs'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:21:35:

           20|           passthru = (previousAttrs.passthru or { }) // {
           21|             overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs);
             |                                   ^
           22|           };

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/trivial.nix:955:7:

          954|     { # TODO: Should we add call-time "type" checking like built in?
          955|       __functor = self: f;
             |       ^
          956|       __functionArgs = args;

       … while calling anonymous lambda

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:16:63:

           15|   # This function introduces `overridePythonAttrs` and it overrides the call to `buildPythonPackage`.
           16|   makeOverridablePythonPackage = f: lib.mirrorFunctionArgs f (origArgs:
             |                                                               ^
           17|     let

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/trivial.nix:955:7:

          954|     { # TODO: Should we add call-time "type" checking like built in?
          955|       __functor = self: f;
             |       ^
          956|       __functionArgs = args;

       … while calling anonymous lambda

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/customisation.nix:136:17:

          135|     in
          136|     mirrorArgs (origArgs:
             |                 ^
          137|     let

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:18:14:

           17|     let
           18|       args = lib.fix (lib.extends
             |              ^
           19|         (_: previousAttrs: {

       … while calling 'fix'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:93:9:

           92|   */
           93|   fix = f: let x = f x; in x;
             |         ^
           94|

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:93:20:

           92|   */
           93|   fix = f: let x = f x; in x;
             |                    ^
           94|

       … while calling 'extends'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:306:7:

          305|     (
          306|       final:
             |       ^
          307|       let

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/fixed-points.nix:308:16:

          307|       let
          308|         prev = f final;
             |                ^
          309|       in

       … while calling anonymous lambda

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:24:10:

           23|         })
           24|         (_: origArgs));
             |          ^
           25|       result = f args;

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:21:76:

           20|           passthru = (previousAttrs.passthru or { }) // {
           21|             overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs);
             |                                                                            ^
           22|           };

       … while calling 'overrideWith'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:26:22:

           25|       result = f args;
           26|       overrideWith = newArgs: args // (if pkgs.lib.isFunction newArgs then newArgs args else newArgs);
             |                      ^
           27|     in

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/pkgs/development/interpreters/python/python-packages-base.nix:26:76:

           25|       result = f args;
           26|       overrideWith = newArgs: args // (if pkgs.lib.isFunction newArgs then newArgs args else newArgs);
             |                                                                            ^
           27|     in

       … while calling anonymous lambda

         at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2231:9:

         2230|       pyarrow = prev.pyarrow.overridePythonAttrs (
         2231|         old: lib.optionalAttrs ((!old.src.isWheel or false) && lib.versionAtLeast old.version "0.16.0")
             |         ^
         2232|           (

       … from call site

         at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2231:14:

         2230|       pyarrow = prev.pyarrow.overridePythonAttrs (
         2231|         old: lib.optionalAttrs ((!old.src.isWheel or false) && lib.versionAtLeast old.version "0.16.0")
             |              ^
         2232|           (

       … while calling 'optionalAttrs'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/attrsets.nix:1352:5:

         1351|     cond:
         1352|     as:
             |     ^
         1353|     if cond then as else {};

       … from call site

         at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2247:13:

         2246|             in
         2247|             lib.throwIf (arrowCppVersion != pyArrowVersion) errorMessage {
             |             ^
         2248|               nativeBuildInputs = old.nativeBuildInputs or [ ] ++ [ pkg-config cmake ];

       … while calling 'throwIf'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/trivial.nix:879:19:

          878|   */
          879|   throwIf = cond: msg: if cond then throw msg else x: x;
             |                   ^
          880|

       … from call site

         at /nix/store/papjgkg8lwdad6qrqys31hx2rq87vd1y-source/overrides/default.nix:2243:33:

         2242|               ARROW_HOME = _arrow-cpp;
         2243|               arrowCppVersion = lib.versions.majorMinor _arrow-cpp;
             |                                 ^
         2244|               pyArrowVersion = lib.versions.majorMinor prev.pyarrow;

       … while calling 'majorMinor'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/versions.nix:45:16:

           44|   */
           45|   majorMinor = v:
             |                ^
           46|     builtins.concatStringsSep "."

       … from call site

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/versions.nix:47:6:

           46|     builtins.concatStringsSep "."
           47|     (lib.take 2 (splitVersion v));
             |      ^
           48|

       … while calling 'sublist'

         at /nix/store/sfha4fghdlh7xvbcm4gc3wj7k8q5csy7-source/lib/lists.nix:1606:5:

         1605|     count:
         1606|     list:
             |     ^
         1607|     let len = length list; in

       error: value is a set while a string was expected

poetry2nix/templates/app/flake.nix:

{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication;
      in
      {
        packages = {
          myapp = mkPoetryApplication { projectDir = self; };
          default = self.packages.${system}.myapp;
        };

        # Shell for app dependencies.
        #
        #     nix develop
        #
        # Use this shell for developing your app.
        devShells.default = pkgs.mkShell {
          inputsFrom = [ self.packages.${system}.myapp ];
        };

        # Shell for poetry.
        #
        #     nix develop .#poetry
        #
        # Use this shell for changes to pyproject.toml and poetry.lock.
        devShells.poetry = pkgs.mkShell {
          packages = [ pkgs.poetry ];
        };
      });
}

modified `poetry2nix/templates/app/pyproject.toml`:

[tool.poetry]
name = "app"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
license = "MIT"
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
pyarrow = "^16.1.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock:

# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.
package = []

[metadata]
lock-version = "2.0"
python-versions = "^3.11"
content-hash = "81b2fa642d7f2d1219cf80112ace12d689d053d81be7f7addb98144d56fc0fb2"

[TyberiusPrime]

So this is in a check that tries to see if the pyarrow version and the arrow-cpp version (from nixpkgs) match, and it's passing the arrow-cpp attrSet, instead of the verison, to lib.versions.majorMinor.

I have a fix in my mega work in progress PR, but in the meantime, have you tried to use to 'just use a wheel'? (pyarrow.override {preferWheel=True}...)

[akaihola]

have you tried to use to 'just use a wheel'? (pyarrow.override {preferWheel=True}...)

Thanks!

I didn't realize I could do that. Unfortunately I've moved on and don't really remember what I was packaging... but next time I try out poetry2nix I'm sure I'll rediscover this issue if I run to the same problem.

[bouk]

This was fixed in #1738