[python-package] support Python 3.13

[jameslamb]

Add support for Python 3.13, now that it's available: https://docs.python.org/3/whatsnew/3.13.html

This also proposes a new standard process for Python version upgrades in this project. Proposing that when a new Python version is generally available, we do the following:

• bump every Python version in all CI up by 1 major version (e.g., all 3.10 -> 3.11)
• add any end-of-life versions to the test-old-versions CI job (as I've done here with Python 3.8)
  • based on https://devguide.python.org/versions/

This makes the updates easy to make and review, and reduces the risk of issues like #5969 and #6680 leading to maintenance work.

[jameslamb]

We'll need to wait for the rest of LightGBM's dependencies to get Python 3.13 packages on conda-forge.

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package mypy-1.11.1-py310h5b4e0ec_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ mypy >=1.11.1  is installable with the potential options
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python_abi 3.12.* *_cp312, which requires
│  │     └─ python 3.12.* *_cpython, which can be installed;
│  ├─ mypy [1.11.1|1.11.2] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2] would require
│  │  └─ pypy3.9 >=7.3.15 , which requires
│  │     └─ python 3.9.* *_73_pypy, which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  └─ mypy 1.12.0 would require
│     └─ python_abi 3.13.* *_cp313 with the potential options
│        ├─ python_abi 3.13 would require
│        │  └─ python 3.13.* *_cpython, which can be installed;
│        └─ python_abi 3.13 would require
│           └─ python 3.13.* *_cp313, which can be installed;
├─ pyarrow-core >=17.0  is installable with the potential options
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ __cuda >=11.8 , which is missing on the system;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python_abi 3.12.* *_cp312, which can be installed (as previously explained);
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  └─ pyarrow-core 17.0.0 would require
│     └─ python >=3.9,<3.10.0a0 , which can be installed;
└─ python 3.13 *_cp* is not installable because it conflicts with any installable versions previously reported.

(build link)

But hey, at least lightgbm itself already has Python 3.13 packages on conda-forge! (conda-forge/lightgbm-feedstock#63). That's thanks to lightgbm having such a small set of required dependencies... a good reminder that it's worthwhile to keep it that way 😁

We can check the state of LightGBM's dependencies here: https://conda-forge.org/status/migration/?name=python313

[jameslamb]

We can check the state of LightGBM's dependencies here: https://conda-forge.org/status/migration/?name=python313

pyarrow now has Python 3.13 packages (conda-forge/pyarrow-feedstock#139). I just merged latest master into this... let's see if all of LightGBM's recursive dependencies are now updated.

[jameslamb]

The conda-forge Python 3.13 migration is making progress, but looks like not all of lightgbm's dependencies have made it across yet. Looks like the next one we need is cffi.

PR to subscribe to: conda-forge/cffi-feedstock#53

[jameslamb]

good news: conda-forge migrations we needed for Python 3.13 appear to be done! (successful macOS Python 3.13 build)

bad news: seems like conda isn't able to find a solution for our CI environment on x86_64 Python 3.9 (failed macOS x86_64 Python 3.9 build, failed Linux x86_64 Python 3.9 job

Not sure about the root cause yet, will try to investigate.

[StrikerRUS]

Thanks a lot!
I checked that there are no any other places to bump Python version.
I left some comments for your consideration below.

[StrikerRUS]

Why not to bump version in Appveyor?

LightGBM/.appveyor.yml

Line 6 in 3654eca

- '3.8'

[jameslamb]

I was thinking that because we use Appveyor to test such an old version of VS Code:

LightGBM/.appveyor.yml

Line 3 in 3654eca

image: Visual Studio 2015

That it should stay frozen on an old Python version too, until supporting that older Python version on Appveyor breaks. Given that, do you think we should make increasing this version to 3.9 part of this update?

[StrikerRUS]

OK, makes sense! I don't mind leaving 3.8.

[StrikerRUS]

Thanks for making Python 3.13 work here!

I'm approving because the diff looks good to me and CI is green. I didn't check all CI logs to ensure that correct Python version is used there.

Please update Ubuntu version in docs:

LightGBM/docs/README.rst

Line 38 in 3aac25a

readthedocs/build:ubuntu-20.04-2021.09.23 \

[jameslamb]

Please update Ubuntu version in docs:

Just pushed 6af26b1 doing that. I tested this locally and it worked well for me.

Thanks as always for your thoroughness!