Add option to limit tests and checks to only selected provider (and depending providers)

[potiuk]

While preparing fab-providers/v1-5 branch and making the branch green - we had to backport a number of commits from main in order to make that older branch of fab provider green: See #45185

This is a bit annoying and time-consuming - and also the problem is that just "running" the whole suite of docs, tests, checks etc. takes a lot of time in CI.

We could add an option to selective checks to only limit tests to selected provider and it's depending providers in such branch. That should be relatively simple task - where we could add a flag --limit-to-providers fab for example and whenever we determine which providers are affected, we shuld just limit it to those that are limited and providers that depend on them.

Then the process of preparing branch for past version of provider would look like this:

1. create a branch
2. add LIMIT_TO_PROVIDERS=<PROVIDER> variable in "ci.yml` where selective checks are run

This could save quite some time on making the branch green.

Note! It's likely it will not cover all cases - there are certain cases where just test collection might stop working because newer version of dependencies retrieved might cause collection to fail and we will not have a chance to "skip" those tests if they fail during collection, but it should severly limit the needs of back porting some main changes for:

a) runnning the tests
b) runninng the compatibility tests

It should also significantly speed up docs building time for PRS for such "old provider versions" branches.

[eladkal]

What direction of depended are we talking about?
For example: If I make changes to presto provider. It has dependency on common.sql so I guess we expect tests for presto and common.sql to run and we have the list of dependencies in the provider.yaml of presto, but what about the different direction? Say we change something in common.sql. Souldn't we run also presto tests? To make sure we didn't break anything in the common.sql interface? In this case we don't have the information of presto dependent on common.sql because we scan only the provider.yaml of common.sql.

[potiuk]

It's both directions. This is the same code that we run currently in selective checks in "regular" PRs - to be on a safe side we select both sides - the stuff we depend on and the stuff that depends on us (and it's recursive).

https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/utils/selective_checks.py#L1443

We also have a few other rules there that trigger different providers when we know there are implicit dependencies.

[potiuk]

This is the actual code:

airflow/dev/breeze/src/airflow_breeze/utils/provider_dependencies.py

Line 31 in f56038e

def get_related_providers(

-> both upstream and downstream cross-provider dependencies are calculated here.