Support for llvm __bf16 type in DPC++

[JackAKirk]

DPC++ does not currently support __bf16 in device code, see https://reviews.llvm.org/D141375

However it is supported for the NVPTX llvm backend: https://reviews.llvm.org/D136311 , and it is supported on amdgpu as a storage type https://reviews.llvm.org/D139398 (I believe it is only used for a single instruction in amdgpu currently: mfma).

Based on my current understanding, I don't think it is a good idea to switch the storage type of bfloat16 here https://github.com/intel/llvm/blob/sycl/sycl/include/sycl/ext/oneapi/bfloat16.hpp#L27 from uint16_t to __bf16, even conditionally for certain backends that could support it (e.g. NVPTX), until full support is available for all backends. Note that a recent llvm-project pull down had to be partially reverted due to a switch of some NVPTX builtins from using uint16_t to __bf16. However in the DPC++ cuda backend we can easily go around this by using inline ptx (that uses the uint16_t storage type) as a replacement for calling these builtins, at least as a temporary solution. This would avoid any problems with past/future pulldowns that use __bf16 in NVPTX builtins, without breaking DPC++ code.

However I thought it would be a good idea to raise a discussion item here in case people have plans/thoughts on the issue of future __bf16 support in DPC++. Is there a plan to support the __bf16 type for SPIRV backends for example?

[bader]

Note that a recent llvm-project pull down had to be partially reverted due to a switch of some NVPTX builtins from using uint16_t to __bf16.

We should do our best to avoid this as maintenance of the project quickly becomes very costly.

I don't think it is a good idea to switch the storage type of bfloat16 here https://github.com/intel/llvm/blob/sycl/sycl/include/sycl/ext/oneapi/bfloat16.hpp#L27 from uint16_t to __bf16, even conditionally for certain backends that could support it (e.g. NVPTX), until full support is available for all backends.

Can you clarify why? I see bfloat16 is customized for NVPTX already, so it's not clear why changing the storage type is unwanted? Is it just code divergence for different targets? If so, I suppose the proposed solution with inline asm should be easier to maintain. Am I right?

Is there a plan to support the __bf16 type for SPIRV backends for example?

+@bashbaug, @MrSidims to follow-up on this question.

[JackAKirk]

Can you clarify why? I see bfloat16 is customized for NVPTX already, so it's not clear why changing the storage type is unwanted? Is it just code divergence for different targets?

Yeah exactly. We could support it if we diverge for different targets. I think the fact that there are a lot of generic implementations of bfloat16 math functions using shorts might be remain problematic if we do this.

If so, I suppose the proposed solution with inline asm should be easier to maintain. Am I right?

Yeah I thought the simplest solution would be to operate with inline asm directly on the shorts, which avoids us having to make the switch over to __bf16 in NVPTX right now. However I can see why there might be an argument for using __bf16 in backends where it is supported. So I wanted to get some other opinions on this. It would be useful to know what the plans of other backends are regarding __bf16 in order to make the best decision.

[asudarsa]

Hi @JackAKirk

I have started looking at this issue. i wanted to quickly clarify if I am looking at relevant portions of bfloat16 support. I suppose we are specifically referring to __bf16 added here. Is that correct? Thanks

commit ecd682b
Author: Ties Stuij [email protected]
Date: Fri Jun 5 00:20:02 2020 +0100
[ARM] Add __bf16 as new Bfloat16 C Type
Summary:
This patch upstreams support for a new storage only bfloat16 C type.
This type is used to implement primitive support for bfloat16 data, in
line with the Bfloat16 extension of the Armv8.6-a architecture, as
detailed here.

[JackAKirk]

Yeah that's right.

[MrSidims]

Yes, we do plan to add this type in SPIR-V and map it to the appropriate LLVM IR type and/or SPIR-V friendly IR construct. Our main motivation here is to extend cooperative/joint matrix extension capabilities.

We don't have timeline for this yet though.

[JackAKirk]

OK thanks. Interestingly the potential NVPTX __bf16 usage that was not added in llvm/llvm-project@250f2bb was the matrix tensor cores capabilities. I expect that they might update this again.

Until __bf16 is supported in more backends such that you will want to expose it in the dpc++ runtime implementation, we can just swap usage of NVPTX builtins that use __bf16 with equivalent inline ptx.

[MrSidims]

For matrices I don't think the type is needed, if you use a short integer as storage and have mechanisms to say, how to interpret this storage passing it to the special hardware (for scalars and vectors as well, but with the special type you can utilize more from LLVM framework). Can't tell what NVIDIA uses though.