Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different behavior for DagBag.dagbag_stats.file on Linux vs Windows #45172

Open
2 tasks done
Dev-iL opened this issue Dec 23, 2024 · 0 comments
Open
2 tasks done

Different behavior for DagBag.dagbag_stats.file on Linux vs Windows #45172

Dev-iL opened this issue Dec 23, 2024 · 0 comments
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet

Comments

@Dev-iL
Copy link
Contributor

Dev-iL commented Dec 23, 2024

Apache Airflow version

2.10.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

On Windows, dagbag.dagbag_stats.file contains an absolute path, whereas on Linux, this field contains a path relative to dagbag.dag_folder.

What you think should happen instead?

The behavior should be the same regardless of the OS.

How to reproduce

Run the below on Windows and Linux and compare the results:

from os import environ
from pathlib import Path

from airflow.models import DagBag

repo_root: Path = next(p for p in Path(__file__).parents if p.name == "my_proj")
home: Path = repo_root / "airflow"
environ["AIRFLOW_HOME"] = home.as_posix()
dagbag = DagBag(include_examples=False)

Operating System

Win10 + RL9.3

Versions of Apache Airflow Providers

Irrelevant

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

I believe the reason for this difference is that

file=filepath.replace(settings.DAGS_FOLDER, ""),

Doesn't normalize the folder separators, so in the case of

# These are values copied from the debugger on Win
settings.DAGS_FOLDER == 'C:\\repositories\\my_proj\\airflow/dags'
filepath == "C:\\repositories\\my_proj\\airflow\\dags\\my_dag.py"

nothing gets replaced. A possible solution for this is applying Path(raw_path).as_posix() to all paths involved.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@Dev-iL Dev-iL added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Development

No branches or pull requests

1 participant