Hugging Face's logo

Seiriryu
/
VToonify

PyTorch
style-transfer
face-stylization
Model card Files Community
Seiriryu commited on
Commit
503849c
·
1 Parent(s): 93b4d7a

Upload 28 files

Browse files
Files changed (28) hide show
  1. .gitattributes +1 -2
  2. README.md +41 -0
  3. models/encoder.pt +3 -0
  4. models/faceparsing.pth +3 -0
  5. models/readme.md +0 -0
  6. models/shape_predictor_68_face_landmarks.dat +3 -0
  7. models/vtoonify_d_arcane/exstyle_code.npy +3 -0
  8. models/vtoonify_d_arcane/vtoonify_s000_d0.5.pt +3 -0
  9. models/vtoonify_d_arcane/vtoonify_s077_d0.5.pt +3 -0
  10. models/vtoonify_d_arcane/vtoonify_s_d.pt +3 -0
  11. models/vtoonify_d_caricature/exstyle_code.npy +3 -0
  12. models/vtoonify_d_caricature/vtoonify_s039_d0.5.pt +3 -0
  13. models/vtoonify_d_caricature/vtoonify_s068_d0.5.pt +3 -0
  14. models/vtoonify_d_cartoon/exstyle_code.npy +3 -0
  15. models/vtoonify_d_cartoon/vtoonify_s026_d0.5.pt +3 -0
  16. models/vtoonify_d_cartoon/vtoonify_s299_d0.5.pt +3 -0
  17. models/vtoonify_d_cartoon/vtoonify_s_d.pt +3 -0
  18. models/vtoonify_d_comic/exstyle_code.npy +3 -0
  19. models/vtoonify_d_comic/vtoonify_s_d.pt +3 -0
  20. models/vtoonify_d_illustration/exstyle_code.npy +3 -0
  21. models/vtoonify_d_illustration/vtoonify_s004_d_c.pt +3 -0
  22. models/vtoonify_d_illustration/vtoonify_s009_d_c.pt +3 -0
  23. models/vtoonify_d_illustration/vtoonify_s043_d_c.pt +3 -0
  24. models/vtoonify_d_illustration/vtoonify_s054_d_c.pt +3 -0
  25. models/vtoonify_d_illustration/vtoonify_s086_d_c.pt +3 -0
  26. models/vtoonify_d_pixar/exstyle_code.npy +3 -0
  27. models/vtoonify_d_pixar/vtoonify_s052_d0.5.pt +3 -0
  28. models/vtoonify_d_pixar/vtoonify_s_d.pt +3 -0
.gitattributes CHANGED
@@ -2,7 +2,6 @@
2
  *.arrow filter=lfs diff=lfs merge=lfs -text
3
  *.bin filter=lfs diff=lfs merge=lfs -text
4
  *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
  *.ftz filter=lfs diff=lfs merge=lfs -text
7
  *.gz filter=lfs diff=lfs merge=lfs -text
8
  *.h5 filter=lfs diff=lfs merge=lfs -text
@@ -22,7 +21,6 @@
22
  *.pt filter=lfs diff=lfs merge=lfs -text
23
  *.pth filter=lfs diff=lfs merge=lfs -text
24
  *.rar filter=lfs diff=lfs merge=lfs -text
25
- *.safetensors filter=lfs diff=lfs merge=lfs -text
26
  saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
  *.tar.* filter=lfs diff=lfs merge=lfs -text
28
  *.tflite filter=lfs diff=lfs merge=lfs -text
@@ -32,3 +30,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
32
  *.zip filter=lfs diff=lfs merge=lfs -text
33
  *.zst filter=lfs diff=lfs merge=lfs -text
34
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
2
  *.arrow filter=lfs diff=lfs merge=lfs -text
3
  *.bin filter=lfs diff=lfs merge=lfs -text
4
  *.bz2 filter=lfs diff=lfs merge=lfs -text
 
5
  *.ftz filter=lfs diff=lfs merge=lfs -text
6
  *.gz filter=lfs diff=lfs merge=lfs -text
7
  *.h5 filter=lfs diff=lfs merge=lfs -text
 
21
  *.pt filter=lfs diff=lfs merge=lfs -text
22
  *.pth filter=lfs diff=lfs merge=lfs -text
23
  *.rar filter=lfs diff=lfs merge=lfs -text
 
24
  saved_model/**/* filter=lfs diff=lfs merge=lfs -text
25
  *.tar.* filter=lfs diff=lfs merge=lfs -text
26
  *.tflite filter=lfs diff=lfs merge=lfs -text
 
30
  *.zip filter=lfs diff=lfs merge=lfs -text
31
  *.zst filter=lfs diff=lfs merge=lfs -text
32
  *tfevents* filter=lfs diff=lfs merge=lfs -text
33
+ models/shape_predictor_68_face_landmarks.dat filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ tags:
4
+ - style-transfer
5
+ - face-stylization
6
+ ---
7
+
8
+ ## Model Details
9
+
10
+ This system provides a web demo for the following paper:
11
+
12
+ **VToonify: Controllable High-Resolution Portrait Video Style Transfer (TOG/SIGGRAPH Asia 2022)**
13
+
14
+ - Developed by: Shuai Yang, Liming Jiang, Ziwei Liu and Chen Change Loy
15
+ - Resources for more information:
16
+ - [Project Page](https://www.mmlab-ntu.com/project/vtoonify/)
17
+ - [Research Paper](https://arxiv.org/abs/2209.11224)
18
+ - [GitHub Repo](https://github.com/williamyang1991/VToonify)
19
+
20
+ **Abstract**
21
+ > Generating high-quality artistic portrait videos is an important and desirable task in computer graphics and vision. Although a series of successful portrait image toonification models built upon the powerful StyleGAN have been proposed, these image-oriented methods have obvious limitations when applied to videos, such as the fixed frame size, the requirement of face alignment, missing non-facial details and temporal inconsistency. In this work, we investigate the challenging controllable high-resolution portrait video style transfer by introducing a novel **VToonify** framework. Specifically, VToonify leverages the mid- and high-resolution layers of StyleGAN to render high-quality artistic portraits based on the multi-scale content features extracted by an encoder to better preserve the frame details. The resulting fully convolutional architecture accepts non-aligned faces in videos of variable size as input, contributing to complete face regions with natural motions in the output. Our framework is compatible with existing StyleGAN-based image toonification models to extend them to video toonification, and inherits appealing features of these models for flexible style control on color and intensity. This work presents two instantiations of VToonify built upon Toonify and DualStyleGAN for collection-based and exemplar-based portrait video style transfer, respectively. Extensive experimental results demonstrate the effectiveness of our proposed VToonify framework over existing methods in generating high-quality and temporally-coherent artistic portrait videos with flexible style controls.
22
+
23
+
24
+ ## Citation Information
25
+ ```bibtex
26
+ @article{yang2022Vtoonify,
27
+ title={VToonify: Controllable High-Resolution Portrait Video Style Transfer},
28
+ author={Yang, Shuai and Jiang, Liming and Liu, Ziwei and Loy, Chen Change},
29
+ journal={ACM Transactions on Graphics (TOG)},
30
+ volume={41},
31
+ number={6},
32
+ articleno={203},
33
+ pages={1--15},
34
+ year={2022},
35
+ publisher={ACM New York, NY, USA},
36
+ doi={10.1145/3550454.3555437},
37
+ }
38
+ ```
39
+
40
+ ## License
41
+ [S-Lab License 1.0](https://github.com/williamyang1991/VToonify/blob/main/LICENSE.md)
models/encoder.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:264f263987efb847f1ff2f2bf729178d0ee25a6d65731f5a4d194de2c3ba3507
3
+ size 1201660560
models/faceparsing.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:468e13ca13a9b43cc0881a9f99083a430e9c0a38abd935431d1c28ee94b26567
3
+ size 53289463
models/readme.md ADDED
File without changes
models/shape_predictor_68_face_landmarks.dat ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fbdc2cb80eb9aa7a758672cbfdda32ba6300efe9b6e6c7a299ff7e736b11b92f
3
+ size 99693937
models/vtoonify_d_arcane/exstyle_code.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:437d2fdd0bb9bf7337a59db515a356de0527fe49e926ab10052e16a01153d026
3
+ size 3694721
models/vtoonify_d_arcane/vtoonify_s000_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:74bad518e0a9fd44c6745d97f7d98f84c16bbc7f2b6484d85e8dbb1b8a5a394b
3
+ size 666594870
models/vtoonify_d_arcane/vtoonify_s077_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae8546dbbcc0f10160f6e87613510cbe833932235b2f55ab61a4572fad66fd39
3
+ size 666594870
models/vtoonify_d_arcane/vtoonify_s_d.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f40236328e890b9acf93dac66873ca634aeb014ca7c56920d8f974288d8e98f0
3
+ size 666594870
models/vtoonify_d_caricature/exstyle_code.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:29fd9d6609458690fb4ea85a2456dd251c2db6bf41f0f9a9ca868e5100f08b49
3
+ size 7354601
models/vtoonify_d_caricature/vtoonify_s039_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f3c493a8585261159b4784a38c19ae433b89faa7a3529dab9ee5e53123c08965
3
+ size 666594870
models/vtoonify_d_caricature/vtoonify_s068_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6299167862c175e17c6cb71340170681cb02f47409d46fa2c56e8773f63b6164
3
+ size 666594870
models/vtoonify_d_cartoon/exstyle_code.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b8ccd6be0154c233920b7694dbb20c8353ab3f5bd33a426a18eb2702e2f8d068
3
+ size 11714990
models/vtoonify_d_cartoon/vtoonify_s026_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed43f4eb0b2e7732dfca4c11edab79329afffb27c0fd41beac86baa04defe672
3
+ size 666594870
models/vtoonify_d_cartoon/vtoonify_s299_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:66f2cc0f19d3ea31a4ca5f7752106244507f89763defe9ced713153907e46a4b
3
+ size 666594870
models/vtoonify_d_cartoon/vtoonify_s_d.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9f938e99995045ac400f2ba35ee2b6aa482e5811765adbb28d00790bfb8aeca
3
+ size 666594870
models/vtoonify_d_comic/exstyle_code.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b901ffeed34bfae885a3481cabb7574481f3d9de57bb0e000c916f969bcd871a
3
+ size 3731118
models/vtoonify_d_comic/vtoonify_s_d.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d08a82d69d7580265b0a0889ba4f952ad12e30fc7edaa5ff50116775a6174ef8
3
+ size 666594870
models/vtoonify_d_illustration/exstyle_code.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ab7bb530bce73a3a97dd9494f98bd5127d65bc78a62dd75c22593cff44e31154
3
+ size 5764913
models/vtoonify_d_illustration/vtoonify_s004_d_c.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a40b556acb644c5b6e05250dfe2965bcdb23b43f308f3a8e2950e04d72aa12b6
3
+ size 666594870
models/vtoonify_d_illustration/vtoonify_s009_d_c.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35d3b7e3e70fa334dbe225d88fd0d1abff377c98929dd9028dbe5b1925ccbf39
3
+ size 666594870
models/vtoonify_d_illustration/vtoonify_s043_d_c.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:706800ecc525d4268b22dafd99e84505b88a57b6c3aec978d5a4838c63deae12
3
+ size 666594870
models/vtoonify_d_illustration/vtoonify_s054_d_c.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca0051cf694a93c28260b86cd608caf99d3023492af3b939e0edaf40af3ac508
3
+ size 666594870
models/vtoonify_d_illustration/vtoonify_s086_d_c.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f4d3724a8a9330fbc05fba3a59576b618fd3b27912e7977b91c5225cfb74ac11
3
+ size 666594870
models/vtoonify_d_pixar/exstyle_code.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0bd9447e7bbccc918106f780e471bd309ad0d416752e82e7781f60a4558e34f
3
+ size 4506989
models/vtoonify_d_pixar/vtoonify_s052_d0.5.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51067b2e97a657200993cdea765ef7c0888c3b5b0f6714507fcd6a392c4294ce
3
+ size 666594870
models/vtoonify_d_pixar/vtoonify_s_d.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e2c0fbe4b0f8de5dad51e9d9d803c194ec8a8631036b9e48037727106d7399f
3
+ size 666594870