SHA256 does not match when using multiple extra index urls containing same package #2631
Comments
|
Is I'll dig deeper on the details you provide on 12/19. |
|
Individual representatives may be eventually identical but does not necessarily have to be. Snapshot index is for development - we may publish snapshot which is mutable and use this snapshot in another projects. When the feature is finalized, we publish release to different index which is immutable. Usually the last snapshot has identical content to release version. But sometimes one may add some last changes and release the package. Snapshot and release would then diverge. To make it clear, we use only [python-repos]
indexes.add = [
# for example
# contains release of A==1.0.0 and B==0.0.1
"https://myindex/nexus/repository/pypi-hosted/simple/",
# contains old snapshot of A==1.0.0 and WIP snapshot B=0.0.2
"https://myindex/nexus/repository/pypi-hosted-snapshots/simple/",
]I want So my idea was to change the order to make sure [python-repos]
indexes.add = [
# for example
# contains old snapshot of A==1.0.0 and WIP snapshot B=0.0.2
"https://myindex/nexus/repository/pypi-hosted-snapshots/simple/",
# contains release of A==1.0.0 and B==0.0.1
"https://myindex/nexus/repository/pypi-hosted/simple/",
]Thank you a lot for looking into this. |
I want to focus here. This is not OK. Each "snapshot" should have its own version; that's what a local version identifier suffix tends to be used for: https://peps.python.org/pep-0440/#local-version-identifiers Re-using the same version number for mutating content is the root issue here. This is Maven-style -SNAPSHOT shenanigans applied to the Python ecosystem. Are there practical issues preventing use of a local version identifier? As I said, I will look at the secondary index / lock / download hash alignment issues on the 19th, but I'd like to have full understanding of this primary issue 1st. |
|
Yes, you are right. We actually use Python and Scala, so I can confirm this approach is definitely Maven-style. I shared PEP 440 with the team after researching what is the "pythonic" approach - few minutes before your response which confirmed it 😄
Not really. This approach was set up few years ago and adopted by most of our Python teams (including ours). Not a problem for Poetry teams, problem for Pants team. We will try to migrate. |
|
Ok, thanks for the confirmation. I still would like to leave this issue open to investigate the indexes / hashes mismatch. There may be something better Pex could be doing. |
|
If there was some improvement that could help us in the meantime (before we migrate), it would be awesome. But as we are doing something non-standard, I am prepared for "can't be resolved" outcome as well 😄 Thank you once again and especially for the quick response time. I really appreciate it. |
|
@MajerMartin one missing piece of information here is what Actually, to be complete, I'll need:
|
|
One other note - I referenced using local version identifiers when creating snapshots, but that's just one of several schemes you could use. Since these are pre-releases, you could also use 4.0.0.aN with N strictly increasing, etc. There are many approaches. The only thing to avoid being ever publishing anything ever in any language ecosystem with the same version as something already published but with different contents. This was always a bad idea. I worked in the java ecosystem for the 1st half of my career and |
|
I tried with Pants 2.24.0 (Pex 2.20.3 and Pip 24.2) and 2.25.0.dev1 (Pex 2.27.1 and Pip 24.2). Attached Pip log was gathered using Output after |
|
Hm, it seems the fastest workaround for us could be to just re-publish snapshot from "tag" CI so the snapshot is identical to release and gets same SHA256. And it then wouldn't matter which one is picked. Sorry for raising this issue as a result of our non-standard approach. There are workarounds for us, so if there is nothing you can/want to improve, feel free to close this anytime 🙂 |
Hi, we are using Pants build system with two private repositories - first one is for release versions, second one is for snapshots. Each of these repositories can contain package with same name and same version.
Pants setup:
Example package in our repos:
When I try to create lock file using
pants generate-lockfiles, Pants generates PEXcreate lockcommand which includes:and which is then translated into these for Pip:
This command crashes with
The problem is that PEX takes hash of package from the first index, where the (resolvable) package is found, but Pip downloads from the last index, where the package is found.
PEX log:
Pip log:
When I reverse the indexes in Pants, I get reversed hash results:
I know this is well known Pip limitation (all indexes have same priority), but is there something that can be done on Pants/PEX side? Use hash of artifact from last index, where the package is found, to match Pip's selection process?
The text was updated successfully, but these errors were encountered: