quant support

[zhyncs]

Hi @yzh119 Thank you for your excellent work. Are there any current plans to support quantization, such as AWQ, SmoothQuant, KV Cache Int8, KV Cache FP8？Thanks.

[yzh119]

Hi @zhyncs , KV Cache Int8, KV Cache FP8 are mentioned in #125 .

Regarding AWQ and SmoothQuant, I suppose the most critical operators are fused dequant+gemv/fused dequant+gemm, and existing libraries have good support for them. I want to avoid duplicate work, and I'm glad to implement the missing operators in these libraries.