Upload kohya-sdxl-lora-training-on-a-free-kaggle-notebook_v19.ipynb

kohya-sdxl-lora-training-on-a-free-kaggle-notebook_v19.ipynb

@@ -0,0 +1 @@
+{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":7299093,"sourceType":"datasetVersion","datasetId":4025881},{"sourceId":7677229,"sourceType":"datasetVersion","datasetId":4478544}],"dockerImageVersionId":30683,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"* **Free Kaggle Notebook V19 - 18 April 2024**\n* **Preffered GPU is GPU T4 x2 - SDXL Base 1.0 DreamBooth Working**\n* **If You Get VRAM Error On Your SDXL Model, Pick GPU P100 but It will be slower and only Single GPU**\n* T4 GPU works faster and literally doubles batch size since there are 2 GPUs given to us\n* So 2000 steps with GPU T4 x2 becomes 4000 steps of single GPU\n* Also according to my tests, SD 1.5 better than SDXL for Kaggle since on Kaggle, we can only do FP16 SDXL training opposed to BF16\n* But With SD 1.5, we can do FP32 which means no quality loss\n* SD 1.5 config loads best realism model automatically that I have found\n* Please read this article carefully for learning all newest findings and configurations : https://www.patreon.com/posts/full-workflow-sd-98620163\n* Please read instructions carefully\n* Most updated version with config of this notebook will be on > \nhttps://www.patreon.com/posts/kohya-sd-1-5-88397937\n* Tutorial Video 2023-11-24 > https://youtu.be/16-b1AjvyBE\n* Open https://www.kaggle.com/ and click + icon and click Create button and upload this notebook\n* Verify your Kaggle Phone number from here to be able to use GPUs : https://www.kaggle.com/settings\n* Support me on Patreon please : https://www.patreon.com/SECourses\n* [Patreon Exclusive Resources Index](https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/Patreon-Posts-Index.md)\n* SECourses YouTube : https://www.youtube.com/SECourses\n* All reg / class images links are shared here (for both man and woman) : https://www.patreon.com/posts/massive-4k-woman-87700469\n* SDXL repo name : ```stabilityai/stable-diffusion-xl-base-1.0```\n* For example to use RealVisXL V2.0 change repo name to : ```SG161222/RealVisXL_V2.0```\n* Use the Patreon post attached SDXL DreamBooth Config. SDXL DreamBooth much better than LoRA.\n* If you need LoRA do DreamBooth and then extract LoRA.","metadata":{}},{"cell_type":"code","source":"!yes | apt-get update --allow-releaseinfo-change\n!yes | apt install libgoogle-perftools-dev\n\n%cd /kaggle/working\n\n!git clone https://github.com/bmaltais/kohya_ss.git\n\n%cd /kaggle/working/kohya_ss\n\n#!pip install fastapi==0.99.1\n\n!./setup.sh -n\n\n!pip install huggingface_hub\n\n!pip install flask\n!pip install pyngrok\n\n# Load SD 1.5 or SDXL config and prepare your dataset and start training\n# Don't forget to save changes you made on the loaded config. \n# If any error happens, start GUI again\n\n!wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/Kaggle_SDXL_DreamBooth.json -O \"/kaggle/working/Kaggle_SDXL_DreamBooth_Best.json\"\n!wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/Kaggle_SD_15_DreamBooth.json -O \"/kaggle/working/Kaggle_SD_15.json\"\n  ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Below code will download and extract 1024x1024 regularization / classification images. Replace link and file name accordingly to your need.**\n\n**For 768x768 just uncomment them and comment 1024x1024. I suggest 768x768 for SD 1.5 config.**\n\n**All reg / class images links are shared here : https://www.patreon.com/posts/massive-4k-woman-87700469**\n\n**Right click the desired resolution and copy link and replace below link**\n\n**This one is for Man class / reg images dataset for 1024x1024 pixels. Use the exact same pixels as your training images.**","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n!wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/man_5200_imgs_1024x1024.zip\n#!wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/man_5200_imgs_768x768.zip\n%cd /kaggle/working/kohya_ss\n\nimport zipfile\n\nzip_file_path = \"/kaggle/working/man_5200_imgs_1024x1024.zip\"\n#zip_file_path = \"/kaggle/working/man_5200_imgs_768x768.zip\"\nextract_path = \"/kaggle/working/\"\n\nwith zipfile.ZipFile(zip_file_path, 'r') as zip_ref:\n    zip_ref.extractall(extract_path)\n    \n!rm -r /kaggle/working/man_5200_imgs_1024x1024.zip\n!rm -r /kaggle/working/man_5200_imgs_768x768.zip\n\nprint(\"Zip file extracted successfully.\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Below code will download and extract 1024x1024 regularization / classification images. Replace link and file name accordingly to your need.**\n\n**For 768x768 just uncomment them and comment 1024x1024. I suggest 768x768 for SD 1.5 config.**\n\n**All reg / class images links are shared here : https://www.patreon.com/posts/massive-4k-woman-87700469**\n\n**Right click the desired resolution and copy link and replace below link**\n\n**This one is for Woman class / reg images dataset for 1024x1024 pixels. Use the exact same pixels as your training images.**","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working\n!wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/woman_5200_imgs_1024x1024.zip\n#!wget https://huggingface.co/MonsterMMORPG/SECourses/resolve/main/woman_5200_imgs_768x768.zip\n%cd /kaggle/working/kohya_ss\n\nimport zipfile\n\nzip_file_path = \"/kaggle/working/woman_5200_imgs_1024x1024.zip\"\n#zip_file_path = \"/kaggle/working/woman_5200_imgs_768x768.zip\"\nextract_path = \"/kaggle/working/\"\n\nwith zipfile.ZipFile(zip_file_path, 'r') as zip_ref:\n    zip_ref.extractall(extract_path)\n    \n!rm -r /kaggle/working/woman_5200_imgs_1024x1024.zip\n!rm -r /kaggle/working/woman_5200_imgs_768x768.zip\n\nprint(\"Zip file extracted successfully.\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### First you need to register and get a token. \n### I used gmail register >\n### https://dashboard.ngrok.com/get-started/your-authtoken\n### Skipped authenticator not necessary\n### How to use NGROK on Kaggle video : https://youtu.be/iBT6rhH0Fjs\n### [Click here](https://cdn-uploads.huggingface.co/production/uploads/6345bd89fe134dfd7a0dba40/eRmm7HjCcKhqsTNTJXy9E.png) to see how a token looks like you need to get","metadata":{}},{"cell_type":"code","source":"\n## first put your ngrok token to the below and then run this code\n## it will give a link like this at below : https://2fc5-34-134-226-xxx.ngrok-free.app\n## open it and then run web ui and once web ui started that link will start working\n## How to use NGROK on Kaggle new video : https://youtu.be/iBT6rhH0Fjs\nimport os\nimport threading\n\nfrom flask import Flask\nfrom pyngrok import ngrok, conf\n\nconf.get_default().auth_token = \"replace_your_ngrok_token_here\"\n\nos.environ[\"FLASK_ENV\"] = \"development\"\n\napp = Flask(__name__)\n\npublic_url = ngrok.connect(7860).public_url\nprint(\" * ngrok tunnel \\\"{}\\\" -> \\\"http://127.0.0.1:{}/\\\"\".format(public_url, 7860))\n\napp.config[\"BASE_URL\"] = public_url\n\[email protected](\"/\")\ndef index():\n    return \"Hello from Colab!\"\n\nthreading.Thread(target=app.run, kwargs={\"use_reloader\": False}).start()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Best SD 1.5 and SDXL configs are already downloaded into working directory when installing first cell\n* Don't forget to load and use them\n* I suggest SD 1.5 config for Kaggle\n* Read more info here : https://www.patreon.com/posts/full-workflow-sd-98620163","metadata":{}},{"cell_type":"code","source":"%cd /kaggle/working/kohya_ss\n!mkdir -p /kaggle/temp/models\n!bash gui.sh --do_not_share","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Below code will delete class / reg images folder to free up space. Execute it after you have prepared your training data via Kohya GUI. Don't forget to change folder path according to yours.**","metadata":{}},{"cell_type":"code","source":"!rm -r /kaggle/working/man_5200_imgs_1024x1024\n!rm -r /kaggle/working/woman_5200_imgs_1024x1024\n!rm -r /kaggle/working/man_5200_imgs_768x768\n!rm -r /kaggle/working/woman_5200_imgs_768x768","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## You can move model files and download 1 by 1","metadata":{}},{"cell_type":"code","source":"#below code will move into /kaggle/temp/models folder and list you models generated there\n#for SDXL DreamBooth we have to save models into that folder because of limited space\n%cd /kaggle/temp/models\n!dir","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#then by using the below code copy model into the working directory and download 1 by 1\n#don't forget to change My_DB_Kaggle_1300 with the model you have\n!cp /kaggle/temp/models/My_DB_Kaggle-step00000520.safetensors /kaggle/working/My_DB_Kaggle-step00000520.safetensors","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#after downloading model delete it from working directory so that you can copy new model\n!rm -r /kaggle/working/My_DB_Kaggle-step00000520.safetensors","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A better way is uploading models into Hugging Face and downloading from there","metadata":{}},{"cell_type":"code","source":"from huggingface_hub import login\nlogin()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from huggingface_hub import HfApi\napi = HfApi()\napi.upload_folder(\n    folder_path=r\"/kaggle/temp/models\",\n    repo_id=\"VideoTutorial/my_checkpoints\",\n    repo_type=\"model\",\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The below code is for LoRA training","metadata":{}},{"cell_type":"code","source":"!accelerate launch \\\n  --num_cpu_threads_per_process=2 \\\n  \"./sdxl_train_network.py\" \\\n  --pretrained_model_name_or_path=\"stabilityai/stable-diffusion-xl-base-1.0\" \\\n  --train_data_dir=\"/kaggle/working/results/img\" \\\n  --reg_data_dir=\"/kaggle/working/results/reg\" \\\n  --resolution=\"1024,1024\" \\\n  --output_dir=\"/kaggle/working/results/model\" \\\n  --logging_dir=\"/kaggle/working/results/log\" \\\n  --network_alpha=\"1\" \\\n  --save_model_as=safetensors \\\n  --network_module=networks.lora \\\n  --text_encoder_lr=0.0004 \\\n  --unet_lr=0.0004 \\\n  --network_dim=32 \\\n  --output_name=\"test_kaggle\" \\\n  --lr_scheduler_num_cycles=\"8\" \\\n  --no_half_vae \\\n  --learning_rate=\"0.0004\" \\\n  --lr_scheduler=\"constant\" \\\n  --train_batch_size=\"1\" \\\n  --max_train_steps=\"5200\" \\\n  --save_every_n_epochs=\"1\" \\\n  --mixed_precision=\"fp16\" \\\n  --save_precision=\"fp16\" \\\n  --optimizer_type=\"Adafactor\" \\\n  --optimizer_args scale_parameter=False relative_step=False warmup_init=False \\\n  --max_data_loader_n_workers=\"0\" \\\n  --bucket_reso_steps=64 \\\n  --gradient_checkpointing \\\n  --full_fp16 \\\n  --xformers \\\n  --bucket_no_upscale \\\n  --noise_offset=0.0 \\\n  --lowram \\\n  --cache_latents \\\n  --cache_latents_to_disk\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**This below code will zip entire output folder of your training. Don't forget to change path according to your naming. Also be sure that you have sufficient workspace to zip it. Easier to download all saved LoRA checkpoints**","metadata":{}},{"cell_type":"code","source":"!tar -czf /kaggle/working/LoRAs.tar.gz -C /kaggle/working/results/model .","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}