21.0.0: pytest baed test suite is failing

[kloczek]

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-hyperlink-21.0.0-3.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-hyperlink-21.0.0-3.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/python3 -Bm pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /home/tkloczko/rpmbuild/BUILD/hyperlink-21.0.0, configfile: pytest.ini
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, asyncio-0.14.0, expect-1.1.0, cov-2.11.1, mock-3.5.1, httpbin-1.0.0, xdist-2.2.1, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, pyfakefs-4.4.0, freezegun-0.4.2, flaky-3.7.0, cases-3.4.6, hypothesis-6.10.0
collected 113 items

src/hyperlink/test/test_common.py ......                                                                                                                             [  5%]
src/hyperlink/test/test_decoded_url.py ............                                                                                                                  [ 15%]
src/hyperlink/test/test_hypothesis.py ......F.......                                                                                                                 [ 28%]
src/hyperlink/test/test_parse.py .                                                                                                                                   [ 29%]
src/hyperlink/test/test_scheme_registration.py .......                                                                                                               [ 35%]
src/hyperlink/test/test_url.py .........................................................................                                                             [100%]

================================================================================= FAILURES =================================================================================
___________________________________________________________ TestHypothesisStrategies.test_hostnames_ascii_nolead ___________________________________________________________

self = <hyperlink.test.test_hypothesis.TestHypothesisStrategies testMethod=test_hostnames_ascii_nolead>

    @given(hostnames(allow_leading_digit=False, allow_idn=False))
>   def test_hostnames_ascii_nolead(self, hostname):
E   hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 8 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.

src/hyperlink/test/test_hypothesis.py:177: FailedHealthCheck
-------------------------------------------------------------------------------- Hypothesis --------------------------------------------------------------------------------
You can add @seed(316402225955689492760055034870421002781) to this test or run pytest with --hypothesis-seed=316402225955689492760055034870421002781 to reproduce this failure.
========================================================================= short test summary info ==========================================================================
FAILED src/hyperlink/test/test_hypothesis.py::TestHypothesisStrategies::test_hostnames_ascii_nolead - hypothesis.errors.FailedHealthCheck: It looks like your strategy is...
====================================================================== 1 failed, 112 passed in 27.97s ======================================================================

[justinkb]

ran into the same thing, seems to be intermittent. tried again and no such fault occured the second time

[kloczek]

I cannot reporoduce that issue but time to time I see another one:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-hyperlink-21.0.0-5.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-hyperlink-21.0.0-5.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=3918079823
rootdir: /home/tkloczko/rpmbuild/BUILD/hyperlink-21.0.0, configfile: pytest.ini
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, trio-0.7.0, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, randomly-3.8.0, Faker-8.12.1, nose2pytest-1.0.8
collected 113 items

src/hyperlink/test/test_decoded_url.py ............                                                                                                                  [ 10%]
src/hyperlink/test/test_scheme_registration.py .......                                                                                                               [ 16%]
src/hyperlink/test/test_url.py .........................................................................                                                             [ 81%]
src/hyperlink/test/test_common.py ......                                                                                                                             [ 86%]
src/hyperlink/test/test_hypothesis.py ......F.......                                                                                                                 [ 99%]
src/hyperlink/test/test_parse.py .                                                                                                                                   [100%]

================================================================================= FAILURES =================================================================================
___________________________________________________________ TestHypothesisStrategies.test_hostnames_ascii_nolead ___________________________________________________________

self = <hyperlink.test.test_hypothesis.TestHypothesisStrategies testMethod=test_hostnames_ascii_nolead>

    @given(hostnames(allow_leading_digit=False, allow_idn=False))
>   def test_hostnames_ascii_nolead(self, hostname):
E   hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 6 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.

src/hyperlink/test/test_hypothesis.py:177: FailedHealthCheck
-------------------------------------------------------------------------------- Hypothesis --------------------------------------------------------------------------------
You can add @seed(182165199112656175031944200410016344689) to this test or run pytest with --hypothesis-seed=182165199112656175031944200410016344689 to reproduce this failure.
========================================================================= short test summary info ==========================================================================
FAILED src/hyperlink/test/test_hypothesis.py::TestHypothesisStrategies::test_hostnames_ascii_nolead - hypothesis.errors.FailedHealthCheck: It looks like your strategy is...
====================================================================== 1 failed, 112 passed in 23.96s ======================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.

But that is caused by use `pytest-randonly.
In other words looks there are some units order dependencies.
Thi scan be diagnosed by https://github.com/mrbean-bremen/pytest-find-dependencies/

[kloczek]

Clean output with disabled pytest-randownly

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-hyperlink-21.0.0-5.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-hyperlink-21.0.0-5.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -p no:randomly
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/tkloczko/rpmbuild/BUILD/hyperlink-21.0.0, configfile: pytest.ini
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, aspectlib-1.5.2, toolbox-0.5, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, flaky-3.7.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, datadir-1.3.1, regressions-2.2.0, cases-3.6.3, xprocess-0.18.1, black-0.3.12, anyio-3.3.0, asyncio-0.15.1, trio-0.7.0, subtests-0.5.0, isort-2.0.0, hypothesis-6.14.6, mock-3.6.1, profiling-1.7.0, Faker-8.12.1, nose2pytest-1.0.8
collected 113 items

src/hyperlink/test/test_common.py ......                                                                                                                             [  5%]
src/hyperlink/test/test_decoded_url.py ............                                                                                                                  [ 15%]
src/hyperlink/test/test_hypothesis.py ..............                                                                                                                 [ 28%]
src/hyperlink/test/test_parse.py .                                                                                                                                   [ 29%]
src/hyperlink/test/test_scheme_registration.py .......                                                                                                               [ 35%]
src/hyperlink/test/test_url.py .........................................................................                                                             [100%]

=========================================================================== 113 passed in 37.88s ===========================================================================
pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'.