|
--- |
|
license: mit |
|
--- |
|
|
|
# DALL-E Mini Running in the Browser (work in progress) |
|
|
|
|
|
### Notes: |
|
* Working tflite conversion: https://colab.research.google.com/gist/josephrocca/f427377f76c574f1c1e8e4d6d62c34b6/tflite-dalle-mini-conversion-separated-encoder-and-decoder.ipynb |
|
* Note that the encoder and decoder need to be converted separately for some reason. More info: https://github.com/tensorflow/tensorflow/issues/56629#issuecomment-1172928973 |
|
* But these models currently require TF Select operators due to bitwise operations that aren't supported in the default tflite opset. [This issue](https://github.com/tensorflow/tfjs/issues/5844) tracks the potential inclusion of TF Select ops in tfjs-tflite, so models like this can be used on the web. |
|
* The encoder requires the TF Select `Erf` op. The decoder requires these TF Select ops: Bitcast, BitwiseOr, BitwiseXor, ConcatV2, Erf, LeftShift, RightShift, ScatterNd, SelectV2, StridedSlice |
|
* Here are the two models: |
|
* https://huggingface.co./rocca/dalle-mini-js/blob/main/debug/dalle_bart_encoder_flax_model.tflite |
|
* https://huggingface.co./rocca/dalle-mini-js/blob/main/debug/dalle_bart_decoder_flax_model.tflite |
|
* ONNX conversion is blocked on support for `Bitcast`, `BitwiseOr` and `BitwiseXor`: https://github.com/onnx/tensorflow-onnx/issues/1985 The need for the `BitwiseAnd` op can be avoided by pulling `jax.random.PRNGKey` outside of the encode-decode function, but I'm not sure where the other ops are coming from - netron doesn't seem to "expand" certain types of nodes, so I think they're hidden within some other nodes. |
|
|
