devX tags get special treatment

[sajith]

I'm not sure if this is a bug or feature, but it does not seem to be documented.

I have been using devX tags in a couple of projects that happen to use setuptools_scm, and it's been working just fine so far. Lately they've just started emitting these errors when pip install-ing them:

ValueError: choosing custom numbers for the `.devX` distance is not supported.
 The 2.0.6.dev1 can't be bumped
Please drop the tag or create a new supported one ending in .dev0

That caused some head-scratching. Why devX tags get this treatment, and why not, say, rcX tags? I'm sure there are good reasons, but it would be nice to have this behavior documented.

[RonnyPfannschmidt]

It's the most safe/sane component of the version as by the python pep for package versions to put under control

Any deliberate release ought to come from a tag

[sajith]

I'm sorry, I'm afraid that I do not understand that reasoning. :-)

By python pep, you mean https://peps.python.org/pep-0440/? That PEP only seem to document certain usage patterns of .devN tags, which is what I thought I was doing when creating .devN tags. Shouldn't setuptools_scm users be free to use whatever .devN tag they want to use? If setuptools_scm would like to make that decision and reserve .devN tags for itself, could that be documented?

Speaking for myself, the whole point of choosing setuptools_scm was the ability to make more ad-hoc versions with less deliberation between releases. Perhaps I misunderstood...

[RonnyPfannschmidt]

Setuptools-scm is more about using the scm as the source of truth

The default version shemes use .dev to create pseudo unique versions per commit

Most quick/ad hoc releases would then simply get a non dev version tgged

[sajith]

Wouldn't git commit hash be enough to create pseudo unique versions? The previous behavior was creating packages with names like <package-name>-<tag>-<shorthash>, regardless of tag name, and it worked just fine for me, until it turned out that .devN tags are out of bounds. It broke a couple of packages for me. I will have to go around and remove a bunch of old .devN tags from a couple of repositories.

I sincerely appreciate the work done on setuptools_scm, but I also want to register a little protest here: it does not seem right for build tools to get in the way, or break existing builds, or spring surprises like this. If it must be done, it should be documented.

Perhaps I should have pinned setuptools_scm, or perhaps I should not rely on it anymore. :-/

[jziolkowski]

Please help me understand something, maybe I don't get how this tool does its thing.

I have develop and master branches. Master is used only for "big" releases, f.e. to PyPI. The last commit pushed to master has a manually added tag, so I can control when semver changes happen. This one is clear.

But I also want to have github prereleases for every PR merged to develop. Last time I setup this tool it was correctly bumping the .devX tags with the expected distance from the last non-dev tag. But recently it shows the same error as above.

So how should I configure that tool so it allows me to achieve that I want?

[jziolkowski]

or to put it differently: when the tool sees the latest tag is x.y.z.devX, why it doesn't drop the .devX part and re-calculate the distance from x.y.z again, to make a new devY?

[RonnyPfannschmidt]

The tool refuses to hide incorrect usage and incorrect data

If a specific dev version is tagged,then guessing is no longer safe

A non numbered dev tag can be used to denote the start of a new line

A explicit dev tag means we no longer know the intent and refuse to guess

[jziolkowski]

So it means that there can be only one .devX tag after a "full" tag, yes? F.e. if I already have 1.2.3.dev12, I then need to manually add 1.2.3 in develop, so it's then versioned with .dev0?

I'm confused with the logic behind it or the what is the use case.

Under what circumstances after I have a release 1.2.3 I could then get multiple 1.2.4.devX pre-releases?

[RonnyPfannschmidt]

Every commit after the tag gets a own version, x being the distance from the Tag

[jziolkowski]

That's what I thought. But when it tries to tag the next commit after an already present prior .devX, it throws the error.

[RonnyPfannschmidt]

The dev version ought not to be tagged

[RonnyPfannschmidt]

The dev version is implied by the latest tag and the distance

A dev tag to denote the start of a new base version explicitly ought not to have a distance

[jziolkowski]

ok then, we're getting somewhere. So if I want to have prereleases in GH, I should simply tag them with anything that does not match the tool pattern, so it does not bump version from something it should not.

Correct?

[RonnyPfannschmidt]

What kinda pre release do you want

Pep440 has a lot of painful Nuance

[jziolkowski]

I want each commit to develop to be a GH prerelease (I need built binaries from CI). I don't care about how its tag is defined, as long as it doesn't break the flow of this tool.

[jziolkowski]

Using PEP440 nomenclature I basically need developmental releases, which I will then publish on GH as a pre-release. The naming might have been confusing on my side.

[RonnyPfannschmidt]

The .dev we add are development releases, as far as I understand the default already does what you want

[jziolkowski]

it doesn't. Any tag that contains numbers between the last "full" tag and the current one is being parsed with regex as potential candidate. When it doesn't match the process fails.

f.e.:

2cb4357 (HEAD -> develop, origin/develop, origin/HEAD) Misc rule fixes (#291)
fa68be7 Update tooling
872224a Shutter controls (#290)
6a0cabe CI scm tags
e2a0c62 Revert "Update release.yml"
a6761ae Update release.yml
3615fee Redesign relay/color/shutter state display (#289)
db64437 Update ci (#287)
535633b Fix ethernet ip address not showing (#286)
e911924 Update macos build
b9217d1 make StatusSHT:TiltConfig optional
bb433e5 update CI
1088554 Update release.yml
65b3a3f add schema for STATUS13 (StatusSHT) (#284)
b7bc865 add release.yml
0790ab9 add Shift-F<x> shortcuts for relays >8 (#282)
6a48bc2 Update README.md
97afc2a (tag: v2024.9.3, origin/master, master, filter_sort_proxy) use version tag in (pre)release

when I run the tool I get the expected new version: 2024.9.4.dev17

When I add any alpha-only tag in between HEAD and the current tag, it will always show the properly generated .dev17 tag.

But as soon as a tag contains any digits, it's being parsed with regex and the process fails, like so:

2cb4357 (HEAD -> develop, origin/develop, origin/HEAD) Misc rule fixes (#291)
fa68be7 Update tooling
872224a Shutter controls (#290)
6a0cabe CI scm tags
e2a0c62 Revert "Update release.yml"
a6761ae Update release.yml
3615fee Redesign relay/color/shutter state display (#289)
db64437 Update ci (#287)
535633b (tag: pre202409211346) Fix ethernet ip address not showing (#286)
e911924 Update macos build
b9217d1 make StatusSHT:TiltConfig optional
bb433e5 update CI
1088554 Update release.yml
65b3a3f add schema for STATUS13 (StatusSHT) (#284)
b7bc865 add release.yml
0790ab9 add Shift-F<x> shortcuts for relays >8 (#282)
6a48bc2 Update README.md
97afc2a (tag: v2024.9.3, origin/master, master, filter_sort_proxy) use version tag in (pre)release

DEBUG setuptools_scm.overrides dist name: tdmgr
DEBUG setuptools_scm.config check absolute root=. relative_to=C:\Users\faza\PycharmProjects\tdm\pyproject.toml
DEBUG setuptools_scm.config file C:\Users\faza\PycharmProjects\tdm\pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in C:\Users\faza\PycharmProjects\tdm
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in C:\Users\faza\PycharmProjects\tdm
DEBUG setuptools_scm.discover found ep EntryPoint(name='.git', value='setuptools_scm.git:parse', group='setuptools_scm.parse_scm') in C:\Users\faza\PycharmProjects\tdm
DEBUG setuptools_scm.run_cmd at .
    $ git version
DEBUG setuptools_scm.run_cmd out:
    git version 2.39.1.windows.1
DEBUG setuptools_scm.run_cmd at C:\Users\faza\PycharmProjects\tdm
    $ git --git-dir "C:\Users\faza\PycharmProjects\tdm\.git" rev-parse --show-prefix
DEBUG setuptools_scm.git real root C:\Users\faza\PycharmProjects\tdm
DEBUG setuptools_scm.run_cmd at C:\Users\faza\PycharmProjects\tdm
    $ git --git-dir "C:\Users\faza\PycharmProjects\tdm\.git" describe --dirty --tags --long --match "*[0-9]*"
DEBUG setuptools_scm.run_cmd out:
    pre202409211346-8-g2cb4357
DEBUG setuptools_scm.version tag pre202409211346
DEBUG setuptools_scm.version tag 'pre202409211346' did not parse

and after some traceback it ends up with assertion error assert version is not None

It would seem that this tool now effectively prevents from having other tags in the history.

Shouldn't it skip the unmatched tag and try on the next one if present, and then still return the .dev17?

[RonnyPfannschmidt]

Is there any reason for not using a valid pre release dev version as tag

In your example v2024.09.21.alpha1346.dev0 seem like a valid version matching the rough intentions

[RonnyPfannschmidt]

A key reason why it's not skipping tags easily is that it requires very specific errors prone git describe invocation plus no availability in Export archives, so expecting users to stay within pep440 enures more robust and wider usage options

[jziolkowski]

Fine, I can alter my tagging to make it compatible. I've tested this manual tag and it works.

Now I need to find out how to automate that and/or which configuration settings to change. The docs are quite confusing for me...

[jziolkowski]

Anyway, I have a feeling that something so simple that I want to achieve is unnecessarily complicated.

[RonnyPfannschmidt]

It's either that or having to explain to people why their metadata is Sometimes broken and sometimes not

[jziolkowski]

I need to rethink all that. Thanks a lot for the explanation.

[gri38]

Hello @RonnyPfannschmidt

It seems that the question has not been fully addressed yet.

Let me explain my development process, which I believe is quite similar to @jziolkowski 's, or @sajith 's, as well as to others who have raised similar questions on forums.

I am developing a package, and during the development phase (leading up to the 1.0.0 release), I publish several versions of the package for use in other projects: 1.0.0.dev0, 1.0.0.dev1, 1.0.0.dev2, ..., up to 1.0.0.dev19.

This practice follows the guidelines of PEP 440 (although PEP 440 recommends using x.y.devN rather than x.y.z.devN, which is not directly relevant here).

However, when working with a local workspace and a "dirty" Git state (where the index/staging area is not clean), I encounter an issue when trying to locally build the package for testing. Running:

hatch build

results in the following error:

ValueError: Error getting the version from source `vcs`: choosing custom numbers for the `.devX` distance is not supported.
The 0.1.dev20 can't be bumped.
Please drop the tag or create a new supported one ending in .dev.

To work around this, my only option so far has been to create a temporary fake tag, which forces me to commit changes I do not wish to commit now.

Are there any fallback solutions or Hatch command-line options that I might not be aware of? If so, I would greatly appreciate any guidance.

Thank you for your tool and your support!

[RonnyPfannschmidt]

The contract for setuptools scm is that it manages the dev versions

Currently it's intentionally unsupported to manage them as a developer

[gri38]

Thanks @RonnyPfannschmidt for clarifying the limitations of setuptools_scm.
However, I'm struggling to understand what the recommended workflow would look like for my use case. Is the idea to avoid using x.y.devN tags entirely?

If so, when publishing a dev release, should I rely on setuptools_scm to automatically generate versions like x.y.dev0+something_obscure? This feels counterintuitive for managing multiple dev iterations. Any suggestions on how to align my workflow with the intended usage?

[RonnyPfannschmidt]

the general suggested workflow would be to just have a automated release from the primary branch

one detail feature for that is missing however - not rendering the node if on the protected main branch

a workaround is configuring the dirty-taglocal scheme - which then would generate the dev versions

[RonnyPfannschmidt]

my recommendation would be to make alphas/betas as releases to hit pypi and tagging them - my persional oppinion is that alpha/beta is more matching for releases that hit a index while .dev is purely development

(key reason why i didn't add the handling of the main branch for releasable artifacts is that i dont need it after all)

[gri38]

Hi again.

Please be sure I insist only to try to ease the life of possible other users.

Today I decided to change to another version "tagger" for hatch, because just the fact I have a x.y.z.dev18 tag in my git, it makes fails other jobs like unit tests.

Illustration:

[...]

It means that if my last tag, a few commits before, was 0.1.0.dev19, it makes fails all my future CI jobs.

This seems like a major challenge in certain workflows, in my opinion that could impact other users as well

[RonnyPfannschmidt]

As per the error that kinda workflow is intentionally not supported