sdxl-quant-int8 / math_model.py

Commit History

Remove potential overflow / saturation error.
161df88

nickfraser commited on

Added comments - highlight possible overflow situation
3f5851c

nickfraser commited on

Updated math model to target int8 x int8 kernels.
4024f9d

nickfraser commited on

Updated QOp model to fuse SmoothQuant scales with input quantization
dca9b6e

nickfraser commited on

Fix: set `keepdim=True`
9ab1060

nickfraser commited on

[math_model/test] Added "QOp" implementation and basic tests.
eb5a5f6

nickfraser commited on

Upload math_model.py with huggingface_hub
049c65f
verified

GiusFra commited on

Upload math_model.py with huggingface_hub
d5dfd96
verified

GiusFra commited on