You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
From internal discussions, logging an issue around updating our MFU calculations so that if FP8 is used, we can generate an accurate MFU number.
Atm - FP8 replaces wq/wk/wv/wo in Attention, and w1/w2/w3 in the MLP.
Thus, need an adjusted calculation.
In addition, would like to correctly pull the proper MFU (fp8 or bf16) based on the training config being run so this is handled automatically for the user.
The text was updated successfully, but these errors were encountered:
From internal discussions, logging an issue around updating our MFU calculations so that if FP8 is used, we can generate an accurate MFU number.
Atm - FP8 replaces wq/wk/wv/wo in Attention, and w1/w2/w3 in the MLP.
Thus, need an adjusted calculation.
In addition, would like to correctly pull the proper MFU (fp8 or bf16) based on the training config being run so this is handled automatically for the user.
The text was updated successfully, but these errors were encountered: