File size: 7,365 Bytes
b72ab63
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b6ee1ede",
   "metadata": {},
   "source": [
    "## Voice Style Control Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7f043ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from openvoice import se_extractor\n",
    "from openvoice.api import BaseSpeakerTTS, ToneColorConverter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15116b59",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aacad912",
   "metadata": {},
   "outputs": [],
   "source": [
    "ckpt_base = 'checkpoints/base_speakers/EN'\n",
    "ckpt_converter = 'checkpoints/converter'\n",
    "device=\"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n",
    "output_dir = 'outputs'\n",
    "\n",
    "base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)\n",
    "base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')\n",
    "\n",
    "tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)\n",
    "tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')\n",
    "\n",
    "os.makedirs(output_dir, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f67740c",
   "metadata": {},
   "source": [
    "### Obtain Tone Color Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8add279",
   "metadata": {},
   "source": [
    "The `source_se` is the tone color embedding of the base speaker. \n",
    "It is an average of multiple sentences generated by the base speaker. We directly provide the result here but\n",
    "the readers feel free to extract `source_se` by themselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63ff6273",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_se = torch.load(f'{ckpt_base}/en_default_se.pth').to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f71fcc3",
   "metadata": {},
   "source": [
    "The `reference_speaker.mp3` below points to the short audio clip of the reference whose voice we want to clone. We provide an example here. If you use your own reference speakers, please **make sure each speaker has a unique filename.** The `se_extractor` will save the `targeted_se` using the filename of the audio and **will not automatically overwrite.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55105eae",
   "metadata": {},
   "outputs": [],
   "source": [
    "reference_speaker = 'resources/example_reference.mp3' # This is the voice you want to clone\n",
    "target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter, target_dir='processed', vad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40284aa",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73dc1259",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = f'{output_dir}/output_en_default.wav'\n",
    "\n",
    "# Run the base speaker tts\n",
    "text = \"This audio is generated by OpenVoice.\"\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "base_speaker_tts.tts(text, src_path, speaker='default', language='English', speed=1.0)\n",
    "\n",
    "# Run the tone color converter\n",
    "encode_message = \"@MyShell\"\n",
    "tone_color_converter.convert(\n",
    "    audio_src_path=src_path, \n",
    "    src_se=source_se, \n",
    "    tgt_se=target_se, \n",
    "    output_path=save_path,\n",
    "    message=encode_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e3ea28a",
   "metadata": {},
   "source": [
    "**Try with different styles and speed.** The style can be controlled by the `speaker` parameter in the `base_speaker_tts.tts` method. Available choices: friendly, cheerful, excited, sad, angry, terrified, shouting, whispering. Note that the tone color embedding need to be updated. The speed can be controlled by the `speed` parameter. Let's try whispering with speed 0.9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd022d38",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_se = torch.load(f'{ckpt_base}/en_style_se.pth').to(device)\n",
    "save_path = f'{output_dir}/output_whispering.wav'\n",
    "\n",
    "# Run the base speaker tts\n",
    "text = \"This audio is generated by OpenVoice.\"\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "base_speaker_tts.tts(text, src_path, speaker='whispering', language='English', speed=0.9)\n",
    "\n",
    "# Run the tone color converter\n",
    "encode_message = \"@MyShell\"\n",
    "tone_color_converter.convert(\n",
    "    audio_src_path=src_path, \n",
    "    src_se=source_se, \n",
    "    tgt_se=target_se, \n",
    "    output_path=save_path,\n",
    "    message=encode_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fcfc70b",
   "metadata": {},
   "source": [
    "**Try with different languages.** OpenVoice can achieve multi-lingual voice cloning by simply replace the base speaker. We provide an example with a Chinese base speaker here and we encourage the readers to try `demo_part2.ipynb` for a detailed demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a71d1387",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "ckpt_base = 'checkpoints/base_speakers/ZH'\n",
    "base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)\n",
    "base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')\n",
    "\n",
    "source_se = torch.load(f'{ckpt_base}/zh_default_se.pth').to(device)\n",
    "save_path = f'{output_dir}/output_chinese.wav'\n",
    "\n",
    "# Run the base speaker tts\n",
    "text = \"今天天气真好,我们一起出去吃饭吧。\"\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "base_speaker_tts.tts(text, src_path, speaker='default', language='Chinese', speed=1.0)\n",
    "\n",
    "# Run the tone color converter\n",
    "encode_message = \"@MyShell\"\n",
    "tone_color_converter.convert(\n",
    "    audio_src_path=src_path, \n",
    "    src_se=source_se, \n",
    "    tgt_se=target_se, \n",
    "    output_path=save_path,\n",
    "    message=encode_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e513094",
   "metadata": {},
   "source": [
    "**Tech for good.** For people who will deploy OpenVoice for public usage: We offer you the option to add watermark to avoid potential misuse. Please see the ToneColorConverter class. **MyShell reserves the ability to detect whether an audio is generated by OpenVoice**, no matter whether the watermark is added or not."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9d70c38e1c0b038dbdffdaa4f8bfa1f6767c43760905c87a9fbe7800d18c6c35"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}