-
Notifications
You must be signed in to change notification settings - Fork 888
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split pandas pytests to prepare for GPU vs CPU metrics reporting #16743
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no PR description or linked issue so I can't tell what you aim to accomplish in this. We should find a solution that does not require git LFS.
@@ -0,0 +1 @@ | |||
python/cudf/cudf/pandas/scripts/.test_durations filter=lfs diff=lfs merge=lfs -text |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this change intentional? Why? I strongly oppose introducing LFS requirements to this repository.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Responded here: #16743 (comment)
@bdice I just updated the PR title. My apologies for hitting ready for review when it isn't. Now it is :) The reason for using git LFS is the file size. I can move the file to s3 but this file will require future updates if the test suite changes drastically and I wanted to avoid having to ask ops to upload a file manually. Do you see any downsides for using GIT LFS ? I followed instructions from ops as mentioned here: https://docs.rapids.ai/resources/git/#large-files-and-git |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove pytest-split
if we are using pytest-shard
instead.
Please run all shards in one worker (serially) to avoid paying for the job setup cost on multiple GPU workers (scarce resources). Then we don't need any LFS, S3, or .gitattributes
changes.
Description
This PR splits the pandas tests job into 3 groups using
pytest-split
. The purpose of this PR is to reduce the runtime and avoid hangs being faced in running all the tests in one job along with profiling that will detect and extract metrics of GPU vs CPU execution in: #16754As part of splitting the tests into separate groups we need
.test_durations
file which is 25 MB, hence I've added this file using git lfs.Checklist