adapt CI tests to use compiled_rmsnorm #451
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Jul 10, 2024
ghstack-source-id: 357f6104859f6d2d9372622a8284330502a4db77 Pull Request resolved: #451
facebook-github-bot
added
the
CLA Signed
awgu
reviewed
Jul 11, 2024
| @@ -54,7 +54,7 @@ def build_test_list(): | |||
| "--experimental.pipeline_parallel_split_points layers.4", | |||
| "--experimental.pipeline_parallel_schedule 1f1b", | |||
| "--training.data_parallel_degree 1", | |||
| "--model.norm_type rmsnorm", # fused_rmsnorm crashes with PP | |||
| "--model.norm_type rmsnorm", # compiled_rmsnorm / fused_rmsnorm crashes with PP | |||
There was a problem hiding this comment.
curious what is the reason that compiled RMSNorm is incompatible with PP
test_runner.py
Outdated
| @@ -257,7 +253,7 @@ def build_test_list(): | |||
| "--experimental.pipeline_parallel_degree 4", | |||
| "--experimental.pipeline_parallel_split_points layers.1,layers.2,layers.3,layers.4,layers.5,layers.6,layers.7", | |||
| "--experimental.pipeline_parallel_schedule interleaved_1f1b", | |||
| "--model.norm_type rmsnorm", # fused_rmsnorm throws cuda context error with pp | |||
| "--model.norm_type rmsnorm", # compiled_rmsnorm / fused_rmsnorm throws cuda context error with pp | |||
There was a problem hiding this comment.
e.g. is this cuda context error the same reason as above for "crashes with PP"?
There was a problem hiding this comment.
Yes. It seems nobody understands why the error happens at this moment.
There was a problem hiding this comment.
If it is the same error, maybe we should make the commit uniform?
[ghstack-poisoned]
tianyu-l
added a commit
that referenced
this pull request
Aug 2, 2024
ghstack-source-id: fa38b2261667b182e0b05699d2f04772ec5852d3 Pull Request resolved: #451
tianyu-l
force-pushed
the
gh/tianyu-l/15/head
branch
from
a437389
to
a757a84
Compare
tianyu-l
added a commit
that referenced
this pull request
Aug 16, 2024
ghstack-source-id: fa38b2261667b182e0b05699d2f04772ec5852d3 Pull Request resolved: #451
|
see updates in #535 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):