Didn't find out why yet but Python is downgraded from 3.13 to 3.11 in the updated lock file. This is what's causing the CI to fail, because the wheel is built for Python 3.13.

I still get Python 3.13 when updating the lock-file locally. Honestly this is the kind of things, where I would wait until next Monday because the problem may disappear by itself ...

Also in an ideal world we would use the same conda environment in the job that builds the wheel and the one that tests it ...

Also in an ideal world we would use the same conda environment in the job that builds the wheel and the one that tests it ...

I think this is quite tricky because we use cibuildwheel, which in turn uses some build tools and they setup their own environment. At least I was discussing "frozen dependencies in CI jobs" in the context of cuml and was told that the jobs that build the wheels would need to use the same frozen versions but that is very difficult to do (pip can use --no-build-isolation but there is nothing like that for building conda packages).

But, given that we hard wire Python 3.13 in the build job in the CUDA CI, could we make it so that producing the lockfile fails if it can't produce one that contains Python 3.13? Or is all this too much effort for a problem that occurs rarely, doesn't lead to silent failures and (hopefully) goes away by waiting a bit?

edit: I'm fine with waiting till next Monday to see if it fixes itself.

My thinking right now is that we don't have to use cibuildwheel in principle. If we use the same conda lock-file to create the env for building and testing the wheel, we should be fine I think.

The right incantation for building the wheel would be something like:

python -m build . --no-isolation

True. Though I am not sure how much we'd gain. In the case of this PR we want things to fail. Maybe with a easier to understand error message. I think if we used a lockfile for building as well things would have succeeded and we'd have quietly switched to using a different Python version

Let's wait until next Monday and revisit if the problem is still there, hopefully not 🤞.

In case the problem is still there, one possible work-around would be to add a constraint for the CUDA build in build_tools/update_environment_and_lock_files.py to have python=3.13 (and add a comment in both places to be explicit that the Python version needs to be synchronized between update_environment_and_lock_files.py and cuda-ci.yml).