Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust MFU to account for FP8 #560

Open
lessw2020 opened this issue Aug 23, 2024 · 1 comment
Open

Adjust MFU to account for FP8 #560

lessw2020 opened this issue Aug 23, 2024 · 1 comment

Comments

@lessw2020
Copy link
Contributor

From internal discussions, logging an issue around updating our MFU calculations so that if FP8 is used, we can generate an accurate MFU number.

Atm - FP8 replaces wq/wk/wv/wo in Attention, and w1/w2/w3 in the MLP.

Thus, need an adjusted calculation.

In addition, would like to correctly pull the proper MFU (fp8 or bf16) based on the training config being run so this is handled automatically for the user.

@raghukiran1224
Copy link

yes, this needs to be updated. The MFU computations for fp8 are too good to be true :)
CC: @lchu-ibm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants