[tests] Attempt to uncover and resolve test flakiness in 'test_linkcheck_exclude_documents'.

[jayaddison]

Feature or Bugfix

• Bugfix (test suite)

Purpose

• As reported in flaky unit test: test_build_linkcheck.test_linkcheck_exclude_documents #12188, the test_linkcheck_exclude_documents test case failed recently. It appears that it makes an assumption about the order that linkchecking results will be received in -- linkchecking uses threading, so we cannot rely on the ordering of those results.

Detail

• Initially this pull request should update the application code to make the occurrence of the problem more likely -- ideally 50% occurrence rate in our workflow jobs -- so that the problem becomes easier to confirm.
• After we have found a reliable way to replicate and observe the problem, we can apply a fix. Continuous integration workflows should begin passing reliably again.
• Finally, the temporary changes to the application code should be reverted.

Relates

• flaky unit test: test_build_linkcheck.test_linkcheck_exclude_documents #12188.

[jayaddison]

Initially this pull request should update the application code to make the occurrence of the problem more likely -- ideally 50% occurrence rate in our workflow jobs -- so that the problem becomes easier to confirm.

Unit test results:

Passes (expected to be ~50%):

• Python 3.9 (Docutils 0.18)
• Python 3.10 (Docutils 0.18)
• Python 3.11 (Docutils 0.18)
• Python 3.12 (Docutils 0.18)
• Python 3.12 (Docutils 0.20)
• Windows

Failures (expected to be ~50%):

• Python 3.9 (Docutils 0.20) - (one failure - test_linkcheck_exclude_documents).
• Python 3.10 (Docutils 0.20) - (one failure - test_linkcheck_exclude_documents).
• Python 3.11 (Docutils 0.20) - (one failure - test_linkcheck_exclude_documents).
• Python 3.12 (Docutils 0.19) - (one failure - test_linkcheck_exclude_documents).
• Docutils HEAD - (one failure - test_linkcheck_exclude_documents).
• LaTeX - (one failure - test_linkcheck_exclude_documents).

[jayaddison]

After we have found a reliable way to replicate and observe the problem, we can apply a fix. Continuous integration workflows should begin passing reliably again.

This appears true for the test results of 6a3d3d9 (essentially: confirming that a in results and then b in results, instead of confirming results == [a, b]).

[picnixz]

Thank you!

[jayaddison]

Thanks!