[bug] "conan download --list pkglist.json" downloads only recipes, but not binaries

[Vobraz]

Describe the bug

Using "conan download" with "--list" specified does not download any packages/binaries, even when they are listed inside pkglist.json.
Using just pattern "conan download zlib/1.3.1:*" works fine
Conan 2.20.1, also tested with conan 2.12.2

How to reproduce it

It can be reproduced following official tutorial
https://docs.conan.io/2/examples/commands/pkglists.html#examples-commands-pkglists
I change pattern and query from tutorial to decrease number of search results
In fresh empty environment with no profiles execute:

conan list "zlib/1.3.1#latest:*" -p "os=Windows AND options.shared=True" --format=json -r=conancenter > pkglist.json

pkglist.json

{
     "conancenter": {
         "zlib/1.3.1": {
            "revisions": {
                "b8bc2603263cf7eccbd6e17e66b0ed76": {
                    "timestamp": 1733936244.862,
                    "packages": {
                        "4a51572e3450257b106bc02ae2dcc4b6fd1de072": {
                            "info": {
                                "settings": {
                                    "arch": "armv8",
                                    "build_type": "Release",
                                    "compiler": "msvc",
                                    "compiler.runtime": "dynamic",
                                    "compiler.runtime_type": "Release",
                                    "compiler.version": "194",
                                    "os": "Windows"
                                },
                                "options": {
                                    "shared": "True"
                                }
                            }
                        },
                        "ae9eaf478e918e6470fe64a4d8d4d9552b0b3606": {
                            "info": {
                                "settings": {
                                    "arch": "x86_64",
                                    "build_type": "Release",
                                    "compiler": "msvc",
                                    "compiler.runtime": "dynamic",
                                    "compiler.runtime_type": "Release",
                                    "compiler.version": "192",
                                    "os": "Windows"
                                },
                                "options": {
                                    "shared": "True"
                                }
                            }
                        },
                        "f7dcd1cec4586c154a242a765a1de011300db8e2": {
                            "info": {
                                "settings": {
                                    "arch": "x86_64",
                                    "build_type": "Release",
                                    "compiler": "msvc",
                                    "compiler.runtime": "dynamic",
                                    "compiler.runtime_type": "Release",
                                    "compiler.version": "193",
                                    "os": "Windows"
                                },
                                "options": {
                                    "shared": "True"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

conan download --list=pkglist.json -r=conancenter -f json > downloaded.json
Note that "downloaded.json" is same as pkglist.json, it contains all packages, as if they are successfully downloaded

downloaded.json

{
    "Local Cache": {
        "zlib/1.3.1": {
            "revisions": {
                "b8bc2603263cf7eccbd6e17e66b0ed76": {
                    "timestamp": 1733936244.862,
                    "packages": {
                        "4a51572e3450257b106bc02ae2dcc4b6fd1de072": {
                            "info": {
                                "settings": {
                                    "arch": "armv8",
                                    "build_type": "Release",
                                    "compiler": "msvc",
                                    "compiler.runtime": "dynamic",
                                    "compiler.runtime_type": "Release",
                                    "compiler.version": "194",
                                    "os": "Windows"
                                },
                                "options": {
                                    "shared": "True"
                                }
                            }
                        },
                        "ae9eaf478e918e6470fe64a4d8d4d9552b0b3606": {
                            "info": {
                                "settings": {
                                    "arch": "x86_64",
                                    "build_type": "Release",
                                    "compiler": "msvc",
                                    "compiler.runtime": "dynamic",
                                    "compiler.runtime_type": "Release",
                                    "compiler.version": "192",
                                    "os": "Windows"
                                },
                                "options": {
                                    "shared": "True"
                                }
                            }
                        },
                        "f7dcd1cec4586c154a242a765a1de011300db8e2": {
                            "info": {
                                "settings": {
                                    "arch": "x86_64",
                                    "build_type": "Release",
                                    "compiler": "msvc",
                                    "compiler.runtime": "dynamic",
                                    "compiler.runtime_type": "Release",
                                    "compiler.version": "193",
                                    "os": "Windows"
                                },
                                "options": {
                                    "shared": "True"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

But in reality there are no packages:

conan list zlib/1.3.1:* -f json > cache.json

cache.json

{
    "Local Cache": {
        "zlib/1.3.1": {
            "revisions": {
                "b8bc2603263cf7eccbd6e17e66b0ed76": {
                    "timestamp": 1733936244.862,
                    "packages": {}
                }
            }
        }
    }
}

[AbrilRBS]

Hi @Vobraz thanks a lot for taking the time to report this.

This would not be a bug, but expected behaviour. The issue here is that the first conan list is a bit missleading when using it to download packages, you need to specify a package revision, because that's what the download command expects.

So the idea here is to add a #latest to the end of the first list pattern, like conan list "zlib/1.3.1#latest:*#latest" -p "os=Windows AND options.shared=True" --format=json -r=conancenter > pkglist.json

I'll check with the team to see if we might want to change something about this current behaviour, with conan-io/conan#18833 being worked on

[Vobraz]

Hi, @AbrilRBS , thanks for the reply

Wow, I really missed this note about package revision in the tutorial, sorry. It happened because I started not from the tutorial, but from just reading command's --help. And then looked into the tutorial when it didn't work, just for sanity check.
Current method is unintuitive. I'd expect that behaviour of "download" with pattern like "zlib/1.3.1:*" and pkglist, that was made with this same pattern, is identical,
And then when I saw list of "downloaded" packages in execution output, when in reality there was none - I was 100% sure that this is a bug.
Maybe some clarification in --help for --list would be useful.

[memsharded]

Current method is unintuitive. I'd expect that behaviour of "download" with pattern like "zlib/1.3.1:*" and pkglist, that was made with this same pattern, is identical,

this is something that we iterated, discussed and carefully considered for a long time while designing the apis and behavior of Conan 2.
The final conclusion was that the expectations of users for a conan download, a conan remove, conan upload or a conan list commands are not the same. Because for example the conan download and conan upload, being relative "heavy" operations with a lot of transfer, users don't expect that by default all the older revisions are downloaded or uploaded. While for other operations like conan list which is informational, users might want to be able to load the minimal information necessary, so the query is as fast as possible, and in this regard, unless explicitly requested for the package-revisions information, it won't retrieve. Note that if by default it would do it, this could make some queries to the servers much slower, because in the server the package-revisions has to be loaded, potentially for many versions+recipe-revisions+package-ids. And there wouldn't be an elegant syntax to opt-out and get only the package_id information.

I understand that this might be fully evident, and I agree it is worth to try to add some clarification in the command line help.

[memsharded]

Closed by #4232, thanks!

[Vobraz]

Thanks for the answers and for your work!