Upload folder using huggingface_hub

.ipynb_checkpoints/llm-question-answering-Copy1 (1)-checkpoint.ipynb

@@ -0,0 +1,1136 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f6aa8c52-9bde-41a6-a5f7-4fa93d5c2a6c",
+   "metadata": {},
+   "source": [
+    "# LLM Instruction-following pipeline with OpenVINO \n",
+    "\n",
+    "LLM stands for “Large Language Model,” which refers to a type of artificial intelligence model that is designed to understand and generate human-like text based on the input it receives. LLMs are trained on large datasets of text to learn patterns, grammar, and semantic relationships, allowing them to generate coherent and contextually relevant responses. One core capability of Large Language Models (LLMs) is to follow natural language instructions. Instruction-following models are capable of generating text in response to prompts and are often used for tasks like writing assistance, chatbots, and content generation.\n",
+    "\n",
+    "In this tutorial, we consider how to run an instruction-following text generation pipeline using popular LLMs and OpenVINO. We will use pre-trained models from the [Hugging Face Transformers](https://huggingface.co/docs/transformers/index) library. To simplify the user experience, the [Hugging Face Optimum Intel](https://huggingface.co/docs/optimum/intel/index) library converts the models to OpenVINO™ IR format.\n",
+    "\n",
+    "The tutorial consists of the following steps:\n",
+    "\n",
+    "- Install prerequisites\n",
+    "- Download and convert the model from a public source using the [OpenVINO integration with Hugging Face Optimum](https://huggingface.co/blog/openvino).\n",
+    "- Compress model weights to INT8 and INT4 with [OpenVINO NNCF](https://github.com/openvinotoolkit/nncf)\n",
+    "- Create an instruction-following inference pipeline\n",
+    "- Run instruction-following pipeline\n",
+    "\n",
+    "\n",
+    "#### Table of contents:\n",
+    "\n",
+    "- [Prerequisites](#Prerequisites)\n",
+    "- [Select model for inference](#Select-model-for-inference)\n",
+    "- [Instantiate Model using Optimum Intel](#Instantiate-Model-using-Optimum-Intel)\n",
+    "- [Compress model weights](#Compress-model-weights)\n",
+    "    - [Weights Compression using Optimum Intel](#Weights-Compression-using-Optimum-Intel)\n",
+    "    - [Weights Compression using NNCF](#Weights-Compression-using-NNCF)\n",
+    "- [Select device for inference and model variant](#Select-device-for-inference-and-model-variant)\n",
+    "- [Create an instruction-following inference pipeline](#Create-an-instruction-following-inference-pipeline)\n",
+    "    - [Setup imports](#Setup-imports)\n",
+    "    - [Prepare template for user prompt](#Prepare-template-for-user-prompt)\n",
+    "    - [Main generation function](#Main-generation-function)\n",
+    "    - [Helpers for application](#Helpers-for-application)\n",
+    "- [Run instruction-following pipeline](#Run-instruction-following-pipeline)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "027108c2-1fbe-4be5-9e23-3fc359185a42",
+   "metadata": {},
+   "source": [
+    "## Prerequisites\n",
+    "[back to top ⬆️](#Table-of-contents:)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5d0473c6-3734-422d-a370-2e39d576be0e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install -Uq pip\n",
+    "%pip uninstall -q -y optimum optimum-intel\n",
+    "%pip install --pre -Uq openvino openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n",
+    "%pip install -q \"torch>=2.1\" \"nncf>=2.7\" \"transformers>=4.36.0\" onnx \"optimum>=1.16.1\" \"accelerate\" \"datasets>=2.14.6\" \"gradio>=4.19\" \"git+https://github.com/huggingface/optimum-intel.git\" --extra-index-url https://download.pytorch.org/whl/cpu"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "611cc777-d5bc-4c7b-92e4-a4befa13b2ce",
+   "metadata": {},
+   "source": [
+    "## Select model for inference\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "The tutorial supports different models, you can select one from the provided options to compare the quality of open source LLM solutions.\n",
+    ">**Note**: conversion of some models can require additional actions from user side and at least 64GB RAM for conversion.\n",
+    "\n",
+    "The available options are:\n",
+    "\n",
+    "* **tiny-llama-1b-chat** - This is the chat model finetuned on top of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T). The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens with the adoption of the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint. More details about model can be found in [model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)\n",
+    "* **phi-2** - Phi-2 is a Transformer with 2.7 billion parameters. It was trained using the same data sources as [Phi-1.5](https://huggingface.co/microsoft/phi-1_5), augmented with a new data source that consists of various NLP synthetic texts and filtered websites (for safety and educational value). When assessed against benchmarks testing common sense, language understanding, and logical reasoning, Phi-2 showcased a nearly state-of-the-art performance among models with less than 13 billion parameters. More details about model can be found in [model card](https://huggingface.co/microsoft/phi-2#limitations-of-phi-2).\n",
+    "* **dolly-v2-3b** - Dolly 2.0 is an instruction-following large language model trained on the Databricks machine-learning platform that is licensed for commercial use. It is based on [Pythia](https://github.com/EleutherAI/pythia) and is trained on ~15k instruction/response fine-tuning records generated by Databricks employees in various capability domains, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization. Dolly 2.0 works by processing natural language instructions and generating responses that follow the given instructions. It can be used for a wide range of applications, including closed question-answering, summarization, and generation. More details about model can be found in [model card](https://huggingface.co/databricks/dolly-v2-3b).\n",
+    "* **red-pajama-3b-instruct** -  A 2.8B parameter pre-trained language model based on GPT-NEOX architecture. The model was fine-tuned for few-shot applications on the data of [GPT-JT](https://huggingface.co/togethercomputer/GPT-JT-6B-v1), with exclusion of tasks that overlap with the HELM core scenarios.More details about model can be found in [model card](https://huggingface.co/togethercomputer/RedPajama-INCITE-Instruct-3B-v1).\n",
+    "* **mistral-7b** - The Mistral-7B-v0.2 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. You can find more details about model in the [model card](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), [paper](https://arxiv.org/abs/2310.06825) and [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n",
+    "* **llama-3-8b-instruct** - Llama 3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. The Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks. More details about model can be found in [Meta blog post](https://ai.meta.com/blog/meta-llama-3/), [model website](https://llama.meta.com/llama3) and [model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct).\n",
+    ">**Note**: run model with demo, you will need to accept license agreement. \n",
+    ">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), carefully read terms of usage and click accept button.  You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
+    ">You can login on Hugging Face Hub in notebook environment, using following code:\n",
+    " \n",
+    "```python\n",
+    "    ## login to huggingfacehub to get access to pretrained model \n",
+    "\n",
+    "    from huggingface_hub import notebook_login, whoami\n",
+    "\n",
+    "    try:\n",
+    "        whoami()\n",
+    "        print('Authorization token already provided')\n",
+    "    except OSError:\n",
+    "        notebook_login()\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "51920510-effc-49ef-8c6f-81c951d96a9b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import requests\n",
+    "\n",
+    "# Fetch `notebook_utils` module\n",
+    "r = requests.get(\n",
+    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
+    ")\n",
+    "open(\"notebook_utils.py\", \"w\").write(r.text)\n",
+    "from notebook_utils import download_file\n",
+    "\n",
+    "if not Path(\"./config.py\").exists():\n",
+    "    download_file(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/llm-question-answering/config.py\")\n",
+    "from config import SUPPORTED_LLM_MODELS\n",
+    "import ipywidgets as widgets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "27b42290-a9b5-4453-9a4c-ffa44bbd966d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "f3928b82e676449b8a6bfbded7f4686d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Model:', index=1, options=('tiny-llama-1b', 'phi-2', 'dolly-v2-3b', 'red-pajama-instruct…"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model_ids = list(SUPPORTED_LLM_MODELS)\n",
+    "\n",
+    "model_id = widgets.Dropdown(\n",
+    "    options=model_ids,\n",
+    "    value=model_ids[1],\n",
+    "    description=\"Model:\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "model_id"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "37e9634f-4fc7-4d9c-9ade-b3e8684a0828",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Selected model phi-2\n"
+     ]
+    }
+   ],
+   "source": [
+    "model_configuration = SUPPORTED_LLM_MODELS[model_id.value]\n",
+    "print(f\"Selected model {model_id.value}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "7937959a-d2a1-49bd-bf12-35554fa901d1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'model_id': 'susnato/phi-2',\n",
+       " 'prompt_template': 'Instruct:{instruction}\\nOutput:'}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model_configuration = SUPPORTED_LLM_MODELS[model_id.value]\n",
+    "model_configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e4fd394-b4fb-4eef-8bdc-d116572aa8f0",
+   "metadata": {},
+   "source": [
+    "## Instantiate Model using Optimum Intel\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "Optimum Intel can be used to load optimized models from the [Hugging Face Hub](https://huggingface.co/docs/optimum/intel/hf.co/models) and create pipelines to run an inference with OpenVINO Runtime using Hugging Face APIs. The Optimum Inference models are API compatible with Hugging Face Transformers models.  This means we just need to replace `AutoModelForXxx` class with the corresponding `OVModelForXxx` class.\n",
+    "\n",
+    "Below is an example of the RedPajama model\n",
+    "\n",
+    "```diff\n",
+    "-from transformers import AutoModelForCausalLM\n",
+    "+from optimum.intel.openvino import OVModelForCausalLM\n",
+    "from transformers import AutoTokenizer, pipeline\n",
+    "\n",
+    "model_id = \"togethercomputer/RedPajama-INCITE-Chat-3B-v1\"\n",
+    "-model = AutoModelForCausalLM.from_pretrained(model_id)\n",
+    "+model = OVModelForCausalLM.from_pretrained(model_id, export=True)\n",
+    "```\n",
+    "\n",
+    "Model class initialization starts with calling `from_pretrained` method. When downloading and converting the Transformers model, the parameter `export=True` should be added. We can save the converted model for the next usage with the `save_pretrained` method.\n",
+    "Tokenizer class and pipelines API are compatible with Optimum models.\n",
+    "\n",
+    "To optimize the generation process and use memory more efficiently, the `use_cache=True` option is enabled. Since the output side is auto-regressive, an output token hidden state remains the same once computed for every further generation step. Therefore, recomputing it every time you want to generate a new token seems wasteful. With the cache, the model saves the hidden state once it has been computed. The model only computes the one for the most recently generated output token at each time step, re-using the saved ones for hidden tokens. This reduces the generation complexity from $O(n^3)$ to $O(n^2)$ for a transformer model. More details about how it works can be found in this [article](https://scale.com/blog/pytorch-improvements#Text%20Translation). With this option, the model gets the previous step's hidden states (cached attention keys and values) as input and additionally provides hidden states for the current step as output. It means for all next iterations, it is enough to provide only a new token obtained from the previous step and cached key values to get the next token prediction. \n",
+    "\n",
+    "## Compress model weights\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "The Weights Compression algorithm is aimed at compressing the weights of the models and can be used to optimize the model footprint and performance of large models where the size of weights is relatively larger than the size of activations, for example, Large Language Models (LLM). Compared to INT8 compression, INT4 compression improves performance even more but introduces a minor drop in prediction quality.\n",
+    "\n",
+    "\n",
+    "### Weights Compression using Optimum Intel\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "Optimum Intel supports weight compression via NNCF out of the box. For 8-bit compression we pass `load_in_8bit=True` to `from_pretrained()` method of `OVModelForCausalLM`. For 4 bit compression we provide `quantization_config=OVWeightQuantizationConfig(bits=4, ...)` argument containing number of bits and other compression parameters. An example of this approach usage you can find in [llm-chatbot notebook](../llm-chatbot)\n",
+    "\n",
+    "### Weights Compression using NNCF\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "You also can perform weights compression for OpenVINO models using NNCF directly. `nncf.compress_weights` function accepts the OpenVINO model instance and compresses its weights for Linear and Embedding layers. We will consider this variant in this notebook for both int4 and int8 compression.\n",
+    "\n",
+    "\n",
+    ">**Note**: This tutorial involves conversion model for FP16 and INT4/INT8 weights compression scenarios. It may be memory and time-consuming in the first run. You can manually control the compression precision below.\n",
+    ">**Note**: There may be no speedup for INT4/INT8 compressed models on dGPU"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "f81602ca-4674-4b61-b2c8-ca11631428b1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "f0ca5ccd8cbf49169f5286150875cf5f",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Checkbox(value=True, description='Prepare INT4 model')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "23304b6eea4f4f23ab2e56867f362471",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Checkbox(value=False, description='Prepare INT8 model')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "babfd2f23029448ca8144cc2554599cf",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Checkbox(value=False, description='Prepare FP16 model')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from IPython.display import display\n",
+    "\n",
+    "prepare_int4_model = widgets.Checkbox(\n",
+    "    value=True,\n",
+    "    description=\"Prepare INT4 model\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "prepare_int8_model = widgets.Checkbox(\n",
+    "    value=False,\n",
+    "    description=\"Prepare INT8 model\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "prepare_fp16_model = widgets.Checkbox(\n",
+    "    value=False,\n",
+    "    description=\"Prepare FP16 model\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "display(prepare_int4_model)\n",
+    "display(prepare_int8_model)\n",
+    "display(prepare_fp16_model)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "0066fbec-b89b-4caa-94fb-9ea8598c22e0",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino\n"
+     ]
+    },
+    {
+     "ename": "NameError",
+     "evalue": "name 'model_configuration' is not defined",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[3], line 11\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mgc\u001b[39;00m\n\u001b[0;32m      9\u001b[0m nncf\u001b[38;5;241m.\u001b[39mset_log_level(logging\u001b[38;5;241m.\u001b[39mERROR)\n\u001b[1;32m---> 11\u001b[0m pt_model_id \u001b[38;5;241m=\u001b[39m \u001b[43mmodel_configuration\u001b[49m[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel_id\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n\u001b[0;32m     12\u001b[0m fp16_model_dir \u001b[38;5;241m=\u001b[39m Path(model_id\u001b[38;5;241m.\u001b[39mvalue) \u001b[38;5;241m/\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFP16\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m     13\u001b[0m int8_model_dir \u001b[38;5;241m=\u001b[39m Path(model_id\u001b[38;5;241m.\u001b[39mvalue) \u001b[38;5;241m/\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mINT8_compressed_weights\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
+      "\u001b[1;31mNameError\u001b[0m: name 'model_configuration' is not defined"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "import logging\n",
+    "import openvino as ov\n",
+    "import nncf\n",
+    "from optimum.intel.openvino import OVModelForCausalLM, OVWeightQuantizationConfig\n",
+    "import gc\n",
+    "\n",
+    "\n",
+    "nncf.set_log_level(logging.ERROR)\n",
+    "\n",
+    "pt_model_id = model_configuration[\"model_id\"]\n",
+    "fp16_model_dir = Path(model_id.value) / \"FP16\"\n",
+    "int8_model_dir = Path(model_id.value) / \"INT8_compressed_weights\"\n",
+    "int4_model_dir = Path(model_id.value) / \"INT4_compressed_weights\"\n",
+    "\n",
+    "core = ov.Core()\n",
+    "\n",
+    "\n",
+    "def convert_to_fp16():\n",
+    "    if (fp16_model_dir / \"openvino_model.xml\").exists():\n",
+    "        return\n",
+    "    ov_model = OVModelForCausalLM.from_pretrained(pt_model_id, export=True, compile=False, load_in_8bit=False)\n",
+    "    ov_model.half()\n",
+    "    ov_model.save_pretrained(fp16_model_dir)\n",
+    "    del ov_model\n",
+    "    gc.collect()\n",
+    "\n",
+    "\n",
+    "def convert_to_int8():\n",
+    "    if (int8_model_dir / \"openvino_model.xml\").exists():\n",
+    "        return\n",
+    "    ov_model = OVModelForCausalLM.from_pretrained(pt_model_id, export=True, compile=False, load_in_8bit=True)\n",
+    "    ov_model.save_pretrained(int8_model_dir)\n",
+    "    del ov_model\n",
+    "    gc.collect()\n",
+    "\n",
+    "\n",
+    "def convert_to_int4():\n",
+    "    compression_configs = {\n",
+    "        \"mistral-7b\": {\n",
+    "            \"sym\": True,\n",
+    "            \"group_size\": 64,\n",
+    "            \"ratio\": 0.6,\n",
+    "        },\n",
+    "        \"red-pajama-3b-instruct\": {\n",
+    "            \"sym\": False,\n",
+    "            \"group_size\": 128,\n",
+    "            \"ratio\": 0.5,\n",
+    "        },\n",
+    "        \"dolly-v2-3b\": {\"sym\": False, \"group_size\": 32, \"ratio\": 0.5},\n",
+    "        \"llama-3-8b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
+    "        \"default\": {\n",
+    "            \"sym\": False,\n",
+    "            \"group_size\": 128,\n",
+    "            \"ratio\": 0.8,\n",
+    "        },\n",
+    "    }\n",
+    "\n",
+    "    model_compression_params = compression_configs.get(model_id.value, compression_configs[\"default\"])\n",
+    "    if (int4_model_dir / \"openvino_model.xml\").exists():\n",
+    "        return\n",
+    "    ov_model = OVModelForCausalLM.from_pretrained(\n",
+    "        pt_model_id,\n",
+    "        export=True,\n",
+    "        compile=False,\n",
+    "        quantization_config=OVWeightQuantizationConfig(bits=4, **model_compression_params),\n",
+    "    )\n",
+    "    ov_model.save_pretrained(int4_model_dir)\n",
+    "    del ov_model\n",
+    "    gc.collect()\n",
+    "\n",
+    "\n",
+    "if prepare_fp16_model.value:\n",
+    "    convert_to_fp16()\n",
+    "if prepare_int8_model.value:\n",
+    "    convert_to_int8()\n",
+    "if prepare_int4_model.value:\n",
+    "    convert_to_int4()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b5f31838-c7d6-4f52-aa2d-1f29c6f3b397",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "60355f86-2250-4ebe-82ac-950f2d4fb01b",
+   "metadata": {},
+   "source": [
+    "Let's compare model size for different compression types"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "42c7b254-1ce4-4f23-813a-9bdc23aed327",
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "NameError",
+     "evalue": "name 'fp16_model_dir' is not defined",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[2], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m fp16_weights \u001b[38;5;241m=\u001b[39m \u001b[43mfp16_model_dir\u001b[49m \u001b[38;5;241m/\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mopenvino_model.bin\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m      2\u001b[0m int8_weights \u001b[38;5;241m=\u001b[39m int8_model_dir \u001b[38;5;241m/\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mopenvino_model.bin\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m      3\u001b[0m int4_weights \u001b[38;5;241m=\u001b[39m int4_model_dir \u001b[38;5;241m/\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mopenvino_model.bin\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
+      "\u001b[1;31mNameError\u001b[0m: name 'fp16_model_dir' is not defined"
+     ]
+    }
+   ],
+   "source": [
+    "fp16_weights = fp16_model_dir / \"openvino_model.bin\"\n",
+    "int8_weights = int8_model_dir / \"openvino_model.bin\"\n",
+    "int4_weights = int4_model_dir / \"openvino_model.bin\"\n",
+    "\n",
+    "if fp16_weights.exists():\n",
+    "    print(f\"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB\")\n",
+    "for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):\n",
+    "    if compressed_weights.exists():\n",
+    "        print(f\"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB\")\n",
+    "    if compressed_weights.exists() and fp16_weights.exists():\n",
+    "        print(f\"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3df73379-bccc-41b1-9c94-c3040819805b",
+   "metadata": {},
+   "source": [
+    "## Select device for inference and model variant\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    ">**Note**: There may be no speedup for INT4/INT8 compressed models on dGPU."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "d2d7bf5b-8a05-4c3b-a36b-631af5c197e9",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a934a422dd6f4a2ea9ab918f0c41b437",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "core = ov.Core()\n",
+    "\n",
+    "support_devices = core.available_devices\n",
+    "if \"NPU\" in support_devices:\n",
+    "    support_devices.remove(\"NPU\")\n",
+    "\n",
+    "device = widgets.Dropdown(\n",
+    "    options=support_devices + [\"AUTO\"],\n",
+    "    value=\"CPU\",\n",
+    "    description=\"Device:\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "device"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "01673d97-6645-4d2a-8306-293b8064b317",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a934a422dd6f4a2ea9ab918f0c41b437",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "device"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "24532480-80a5-4953-9cd6-78ac51a1cd8f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "4a81c4bf7a4d45b89c23aac543b9e027",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Model to run:', options=('INT8',), value='INT8')"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "available_models = []\n",
+    "if int4_model_dir.exists():\n",
+    "    available_models.append(\"INT4\")\n",
+    "if int8_model_dir.exists():\n",
+    "    available_models.append(\"INT8\")\n",
+    "if fp16_model_dir.exists():\n",
+    "    available_models.append(\"FP16\")\n",
+    "\n",
+    "model_to_run = widgets.Dropdown(\n",
+    "    options=available_models,\n",
+    "    value=available_models[0],\n",
+    "    description=\"Model to run:\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "model_to_run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5259c1c5-4128-4210-9ad2-faf33ee40e86",
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "NameError",
+     "evalue": "name 'model_to_run' is not defined",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[1], line 3\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtransformers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m AutoTokenizer\n\u001b[1;32m----> 3\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[43mmodel_to_run\u001b[49m\u001b[38;5;241m.\u001b[39mvalue \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mINT4\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m      4\u001b[0m     model_dir \u001b[38;5;241m=\u001b[39m int4_model_dir\n\u001b[0;32m      5\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m model_to_run\u001b[38;5;241m.\u001b[39mvalue \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mINT8\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n",
+      "\u001b[1;31mNameError\u001b[0m: name 'model_to_run' is not defined"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "if model_to_run.value == \"INT4\":\n",
+    "    model_dir = int4_model_dir\n",
+    "elif model_to_run.value == \"INT8\":\n",
+    "    model_dir = int8_model_dir\n",
+    "else:\n",
+    "    model_dir = fp16_model_dir\n",
+    "print(f\"Loading model from {model_dir}\")\n",
+    "\n",
+    "model_name = model_configuration[\"model_id\"]\n",
+    "print(model_name)\n",
+    "ov_config = {\"PERFORMANCE_HINT\": \"LATENCY\", \"NUM_STREAMS\": \"1\", \"CACHE_DIR\": \"\"}\n",
+    "\n",
+    "tok = AutoTokenizer.from_pretrained(model_name)\n",
+    "\n",
+    "ov_model = OVModelForCausalLM.from_pretrained(\n",
+    "    model_dir,\n",
+    "    device=device.value,\n",
+    "    ov_config=ov_config,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bf13f6c3-6671-408e-ae0e-aaa3d8a6eaac",
+   "metadata": {},
+   "source": [
+    "## Create an instruction-following inference pipeline\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    " \n",
+    " The `run_generation` function accepts user-provided text input, tokenizes it, and runs the generation process. Text generation is an iterative process, where each next token depends on previously generated until a maximum number of tokens or stop generation condition is not reached. To obtain intermediate generation results without waiting until when generation is finished, we will use [`TextIteratorStreamer`](https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.TextIteratorStreamer), provided as part of HuggingFace [Streaming API](https://huggingface.co/docs/transformers/main/en/generation_strategies#streaming).\n",
+    " \n",
+    "The diagram below illustrates how the instruction-following pipeline works\n",
+    "\n",
+    "![generation pipeline)](https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/e881f4a4-fcc8-427a-afe1-7dd80aebd66e)\n",
+    "\n",
+    "As can be seen, on the first iteration, the user provided instructions converted to token ids using a tokenizer, then prepared input provided to the model. The model generates probabilities for all tokens in logits format  The way the next token will be selected over predicted probabilities is driven by the selected decoding methodology. You can find more information about the most popular decoding methods in this [blog](https://huggingface.co/blog/how-to-generate).\n",
+    "\n",
+    "There are several parameters that can control text generation quality:\n",
+    "\n",
+    "  * `Temperature` is a parameter used to control the level of creativity in AI-generated text. By adjusting the `temperature`, you can influence the AI model's probability distribution, making the text more focused or diverse.  \n",
+    "  Consider the following example: The AI model has to complete the sentence \"The cat is ____.\" with the following token probabilities:  \n",
+    "\n",
+    "    playing: 0.5  \n",
+    "    sleeping: 0.25  \n",
+    "    eating: 0.15  \n",
+    "    driving: 0.05  \n",
+    "    flying: 0.05  \n",
+    "\n",
+    "    - **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as \"playing.\"  \n",
+    "    - **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as \"playing,\" \"sleeping,\" or \"eating.\"  \n",
+    "    - **High temperature** (e.g., 2.0): The AI model becomes more adventurous, increasing the chances of selecting less likely tokens, such as \"driving\" and \"flying.\"\n",
+    "  * `Top-p`, also known as nucleus sampling, is a parameter used to control the range of tokens considered by the AI model based on their cumulative probability. By adjusting the `top-p` value, you can influence the AI model's token selection, making it more focused or diverse.\n",
+    "  Using the same example with the cat, consider the following top_p settings:  \n",
+    "    - **Low top_p** (e.g., 0.5): The AI model considers only tokens with the highest cumulative probability, such as \"playing.\"  \n",
+    "    - **Medium top_p** (e.g., 0.8): The AI model considers tokens with a higher cumulative probability, such as \"playing,\" \"sleeping,\" and \"eating.\"  \n",
+    "    - **High top_p** (e.g., 1.0): The AI model considers all tokens, including those with lower probabilities, such as \"driving\" and \"flying.\" \n",
+    "  * `Top-k` is another popular sampling strategy. In comparison with Top-P, which chooses from the smallest possible set of words whose cumulative probability exceeds the probability P, in Top-K sampling K most likely next words are filtered and the probability mass is redistributed among only those K next words. In our example with cat, if k=3, then only \"playing\", \"sleeping\" and \"eating\" will be taken into account as possible next word.\n",
+    "\n",
+    "To optimize the generation process and use memory more efficiently, the `use_cache=True` option is enabled. Since the output side is auto-regressive, an output token hidden state remains the same once computed for every further generation step. Therefore, recomputing it every time you want to generate a new token seems wasteful. With the cache, the model saves the hidden state once it has been computed. The model only computes the one for the most recently generated output token at each time step, re-using the saved ones for hidden tokens. This reduces the generation complexity from O(n^3) to O(n^2) for a transformer model. More details about how it works can be found in this [article](https://scale.com/blog/pytorch-improvements#Text%20Translation). With this option, the model gets the previous step's hidden states (cached attention keys and values) as input and additionally provides hidden states for the current step as output. It means for all next iterations, it is enough to provide only a new token obtained from the previous step and cached key values to get the next token prediction. \n",
+    "\n",
+    "The generation cycle repeats until the end of the sequence token is reached or it also can be interrupted when maximum tokens will be generated. As already mentioned before, we can enable printing current generated tokens without waiting until when the whole generation is finished using Streaming API, it adds a new token to the output queue and then prints them when they are ready."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eb7af692-0a15-4493-86ef-a80cda21551c",
+   "metadata": {},
+   "source": [
+    "### Setup imports\n",
+    "[back to top ⬆️](#Table-of-contents:)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "19d23a99-3284-42db-b7dc-5805d219f70d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from threading import Thread\n",
+    "from time import perf_counter\n",
+    "from typing import List\n",
+    "import gradio as gr\n",
+    "from transformers import AutoTokenizer, TextIteratorStreamer\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b341a9e6-290d-4780-90da-1ee64cee436d",
+   "metadata": {},
+   "source": [
+    "### Prepare template for user prompt\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "For effective generation, model expects to have input in specific format. The code below prepare template for passing user instruction into model with providing additional context."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "e2638c5b-47ad-4213-80da-8cfc2659b3aa",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
+     ]
+    }
+   ],
+   "source": [
+    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
+    "tokenizer_kwargs = model_configuration.get(\"toeknizer_kwargs\", {})\n",
+    "\n",
+    "\n",
+    "def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:\n",
+    "    \"\"\"\n",
+    "    Gets the token ID for a given string that has been added to the tokenizer as a special token.\n",
+    "\n",
+    "    Args:\n",
+    "        tokenizer (PreTrainedTokenizer): the tokenizer\n",
+    "        key (str): the key to convert to a single token\n",
+    "\n",
+    "    Raises:\n",
+    "        RuntimeError: if more than one ID was generated\n",
+    "\n",
+    "    Returns:\n",
+    "        int: the token ID for the given key\n",
+    "    \"\"\"\n",
+    "    token_ids = tokenizer.encode(key)\n",
+    "    if len(token_ids) > 1:\n",
+    "        raise ValueError(f\"Expected only a single token for '{key}' but found {token_ids}\")\n",
+    "    return token_ids[0]\n",
+    "\n",
+    "\n",
+    "response_key = model_configuration.get(\"response_key\")\n",
+    "tokenizer_response_key = None\n",
+    "\n",
+    "if response_key is not None:\n",
+    "    tokenizer_response_key = next(\n",
+    "        (token for token in tokenizer.additional_special_tokens if token.startswith(response_key)),\n",
+    "        None,\n",
+    "    )\n",
+    "\n",
+    "end_key_token_id = None\n",
+    "if tokenizer_response_key:\n",
+    "    try:\n",
+    "        end_key = model_configuration.get(\"end_key\")\n",
+    "        if end_key:\n",
+    "            end_key_token_id = get_special_token_id(tokenizer, end_key)\n",
+    "        # Ensure generation stops once it generates \"### End\"\n",
+    "    except ValueError:\n",
+    "        pass\n",
+    "\n",
+    "prompt_template = model_configuration.get(\"prompt_template\", \"{instruction}\")\n",
+    "end_key_token_id = end_key_token_id or tokenizer.eos_token_id\n",
+    "pad_token_id = end_key_token_id or tokenizer.pad_token_id"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55dbc4ae-da28-4be8-b928-3dd68c197937",
+   "metadata": {},
+   "source": [
+    "### Main generation function\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "As it was discussed above, `run_generation` function is the entry point for starting generation. It gets provided input instruction as parameter and returns model response."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "27802e81-9d42-4d71-99ea-5f76db5237f1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def run_generation(\n",
+    "    user_text: str,\n",
+    "    top_p: float,\n",
+    "    temperature: float,\n",
+    "    top_k: int,\n",
+    "    max_new_tokens: int,\n",
+    "    perf_text: str,\n",
+    "):\n",
+    "    \"\"\"\n",
+    "    Text generation function\n",
+    "\n",
+    "    Parameters:\n",
+    "      user_text (str): User-provided instruction for a generation.\n",
+    "      top_p (float):  Nucleus sampling. If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.\n",
+    "      temperature (float): The value used to module the logits distribution.\n",
+    "      top_k (int): The number of highest probability vocabulary tokens to keep for top-k-filtering.\n",
+    "      max_new_tokens (int): Maximum length of generated sequence.\n",
+    "      perf_text (str): Content of text field for printing performance results.\n",
+    "    Returns:\n",
+    "      model_output (str) - model-generated text\n",
+    "      perf_text (str) - updated perf text filed content\n",
+    "    \"\"\"\n",
+    "\n",
+    "    # Prepare input prompt according to model expected template\n",
+    "    prompt_text = prompt_template.format(instruction=user_text)\n",
+    "\n",
+    "    # Tokenize the user text.\n",
+    "    model_inputs = tokenizer(prompt_text, return_tensors=\"pt\", **tokenizer_kwargs)\n",
+    "\n",
+    "    # Start generation on a separate thread, so that we don't block the UI. The text is pulled from the streamer\n",
+    "    # in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.\n",
+    "    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)\n",
+    "    generate_kwargs = dict(\n",
+    "        model_inputs,\n",
+    "        streamer=streamer,\n",
+    "        max_new_tokens=max_new_tokens,\n",
+    "        do_sample=True,\n",
+    "        top_p=top_p,\n",
+    "        temperature=float(temperature),\n",
+    "        top_k=top_k,\n",
+    "        eos_token_id=end_key_token_id,\n",
+    "        pad_token_id=pad_token_id,\n",
+    "    )\n",
+    "    t = Thread(target=ov_model.generate, kwargs=generate_kwargs)\n",
+    "    t.start()\n",
+    "\n",
+    "    # Pull the generated text from the streamer, and update the model output.\n",
+    "    model_output = \"\"\n",
+    "    per_token_time = []\n",
+    "    num_tokens = 0\n",
+    "    start = perf_counter()\n",
+    "    for new_text in streamer:\n",
+    "        current_time = perf_counter() - start\n",
+    "        model_output += new_text\n",
+    "        perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)\n",
+    "        yield model_output, perf_text\n",
+    "        start = perf_counter()\n",
+    "    return model_output, perf_text"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6d7b7182-1d9b-485b-81f2-1a9fab6d5cd4",
+   "metadata": {},
+   "source": [
+    "### Helpers for application\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "For making interactive user interface we will use Gradio library. The code bellow provides useful functions used for communication with UI elements."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "c9b2a2b7-5a91-470f-98d7-7355664bc1be",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def estimate_latency(\n",
+    "    current_time: float,\n",
+    "    current_perf_text: str,\n",
+    "    new_gen_text: str,\n",
+    "    per_token_time: List[float],\n",
+    "    num_tokens: int,\n",
+    "):\n",
+    "    \"\"\"\n",
+    "    Helper function for performance estimation\n",
+    "\n",
+    "    Parameters:\n",
+    "      current_time (float): This step time in seconds.\n",
+    "      current_perf_text (str): Current content of performance UI field.\n",
+    "      new_gen_text (str): New generated text.\n",
+    "      per_token_time (List[float]): history of performance from previous steps.\n",
+    "      num_tokens (int): Total number of generated tokens.\n",
+    "\n",
+    "    Returns:\n",
+    "      update for performance text field\n",
+    "      update for a total number of tokens\n",
+    "    \"\"\"\n",
+    "    num_current_toks = len(tokenizer.encode(new_gen_text))\n",
+    "    num_tokens += num_current_toks\n",
+    "    per_token_time.append(num_current_toks / current_time)\n",
+    "    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:\n",
+    "        current_bucket = per_token_time[:-10]\n",
+    "        return (\n",
+    "            f\"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}\",\n",
+    "            num_tokens,\n",
+    "        )\n",
+    "    return current_perf_text, num_tokens\n",
+    "\n",
+    "\n",
+    "def reset_textbox(instruction: str, response: str, perf: str):\n",
+    "    \"\"\"\n",
+    "    Helper function for resetting content of all text fields\n",
+    "\n",
+    "    Parameters:\n",
+    "      instruction (str): Content of user instruction field.\n",
+    "      response (str): Content of model response field.\n",
+    "      perf (str): Content of performance info filed\n",
+    "\n",
+    "    Returns:\n",
+    "      empty string for each placeholder\n",
+    "    \"\"\"\n",
+    "    return \"\", \"\", \"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "31ebb167-0e55-4271-aedd-13814c2356d2",
+   "metadata": {},
+   "source": [
+    "## Run instruction-following pipeline\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "Now, we are ready to explore model capabilities. This demo provides a simple interface that allows communication with a model using text instruction. Type your instruction into the `User instruction` field or select one from predefined examples and click on the `Submit` button to start generation. Additionally, you can modify advanced generation parameters:\n",
+    "\n",
+    "* `Device` - allows switching inference device. Please note, every time when new device is selected, model will be recompiled and this takes some time.\n",
+    "* `Max New Tokens` - maximum size of generated text.\n",
+    "* `Top-p (nucleus sampling)` -  if set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.\n",
+    "* `Top-k` - the number of highest probability vocabulary tokens to keep for top-k-filtering.\n",
+    "* `Temperature` - the value used to module the logits distribution."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "9f222d02-847a-490f-8d66-02608a53259b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running on local URL:  http://127.0.0.1:7860\n",
+      "\n",
+      "To create a public link, set `share=True` in `launch()`.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div><iframe src=\"http://127.0.0.1:7860/\" width=\"100%\" height=\"800\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "examples = [\n",
+    "    \"Give me a recipe for pizza with pineapple\",\n",
+    "    \"Write me a tweet about the new OpenVINO release\",\n",
+    "    \"Explain the difference between CPU and GPU\",\n",
+    "    \"Give five ideas for a great weekend with family\",\n",
+    "    \"Do Androids dream of Electric sheep?\",\n",
+    "    \"Who is Dolly?\",\n",
+    "    \"Please give me advice on how to write resume?\",\n",
+    "    \"Name 3 advantages to being a cat\",\n",
+    "    \"Write instructions on how to become a good AI engineer\",\n",
+    "    \"Write a love letter to my best friend\",\n",
+    "]\n",
+    "\n",
+    "\n",
+    "with gr.Blocks() as demo:\n",
+    "    gr.Markdown(\n",
+    "        \"# Question Answering with \" + model_id.value + \" and OpenVINO.\\n\"\n",
+    "        \"Provide instruction which describes a task below or select among predefined examples and model writes response that performs requested task.\"\n",
+    "    )\n",
+    "\n",
+    "    with gr.Row():\n",
+    "        with gr.Column(scale=4):\n",
+    "            user_text = gr.Textbox(\n",
+    "                placeholder=\"Write an email about an alpaca that likes flan\",\n",
+    "                label=\"User instruction\",\n",
+    "            )\n",
+    "            model_output = gr.Textbox(label=\"Model response\", interactive=False)\n",
+    "            performance = gr.Textbox(label=\"Performance\", lines=1, interactive=False)\n",
+    "            with gr.Column(scale=1):\n",
+    "                button_clear = gr.Button(value=\"Clear\")\n",
+    "                button_submit = gr.Button(value=\"Submit\")\n",
+    "            gr.Examples(examples, user_text)\n",
+    "        with gr.Column(scale=1):\n",
+    "            max_new_tokens = gr.Slider(\n",
+    "                minimum=1,\n",
+    "                maximum=1000,\n",
+    "                value=256,\n",
+    "                step=1,\n",
+    "                interactive=True,\n",
+    "                label=\"Max New Tokens\",\n",
+    "            )\n",
+    "            top_p = gr.Slider(\n",
+    "                minimum=0.05,\n",
+    "                maximum=1.0,\n",
+    "                value=0.92,\n",
+    "                step=0.05,\n",
+    "                interactive=True,\n",
+    "                label=\"Top-p (nucleus sampling)\",\n",
+    "            )\n",
+    "            top_k = gr.Slider(\n",
+    "                minimum=0,\n",
+    "                maximum=50,\n",
+    "                value=0,\n",
+    "                step=1,\n",
+    "                interactive=True,\n",
+    "                label=\"Top-k\",\n",
+    "            )\n",
+    "            temperature = gr.Slider(\n",
+    "                minimum=0.1,\n",
+    "                maximum=5.0,\n",
+    "                value=0.8,\n",
+    "                step=0.1,\n",
+    "                interactive=True,\n",
+    "                label=\"Temperature\",\n",
+    "            )\n",
+    "\n",
+    "    user_text.submit(\n",
+    "        run_generation,\n",
+    "        [user_text, top_p, temperature, top_k, max_new_tokens, performance],\n",
+    "        [model_output, performance],\n",
+    "    )\n",
+    "    button_submit.click(\n",
+    "        run_generation,\n",
+    "        [user_text, top_p, temperature, top_k, max_new_tokens, performance],\n",
+    "        [model_output, performance],\n",
+    "    )\n",
+    "    button_clear.click(\n",
+    "        reset_textbox,\n",
+    "        [user_text, model_output, performance],\n",
+    "        [user_text, model_output, performance],\n",
+    "    )\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    demo.queue()\n",
+    "    try:\n",
+    "        demo.launch(height=800)\n",
+    "    except Exception:\n",
+    "        demo.launch(share=True, height=800)\n",
+    "\n",
+    "# If you are launching remotely, specify server_name and server_port\n",
+    "# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
+    "# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "440417f8-d2e2-4d8d-8311-882d133bb572",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dd9f90e4-0ec6-4164-8a31-044b1079d3e7",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7297ddd6-2c3d-4540-aa9b-f1c48c274a86",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openvino_env",
+   "language": "python",
+   "name": "openvino_env"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  },
+  "openvino_notebooks": {
+   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/daafd702-5a42-4f54-ae72-2e4480d73501",
+   "tags": {
+    "categories": [
+     "Model Demos",
+     "AI Trends"
+    ],
+    "libraries": [],
+    "other": [
+     "LLM"
+    ],
+    "tasks": [
+     "Text Generation"
+    ]
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {},
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

README.md

@@ -1,12 +1,6 @@
 ---
 title: INTEL
-emoji: ⚡
-colorFrom: green
-colorTo: pink
-sdk: gradio
-sdk_version: 4.37.2
 app_file: app.py
-pinned: false
+sdk: gradio
+sdk_version: 4.36.1
 ---
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

app.py

@@ -0,0 +1,340 @@
+import os
+from transformers import AutoTokenizer, AutoConfig
+from optimum.intel.openvino import OVModelForCausalLM
+from generation_utils import run_generation, estimate_latency, reset_textbox,get_special_token_id
+from config import SUPPORTED_LLM_MODELS
+import gradio as gr
+from threading import Thread
+from time import perf_counter
+from typing import List
+from transformers import AutoTokenizer, TextIteratorStreamer
+import numpy as np
+import os
+from flask import Flask, render_template, redirect, url_for, request, flash
+from flask_sqlalchemy import SQLAlchemy
+from flask_login import LoginManager, UserMixin, login_user, login_required, logout_user, current_user
+from werkzeug.security import generate_password_hash, check_password_hash
+
+app = Flask(__name__)
+app.config['SECRET_KEY'] = 'your_secret_key'
+app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///users.db'
+db = SQLAlchemy(app)
+login_manager = LoginManager()
+login_manager.init_app(app)
+login_manager.login_view = 'login'
+
+class User(db.Model):
+    id = db.Column(db.Integer, primary_key=True)
+    username = db.Column(db.String(80), unique=True, nullable=False)
+    email = db.Column(db.String(120), unique=True, nullable=False)
+
+    def __repr__(self):
+        return '<User %r>' % self.username
+
+# Create the database tables
+with app.app_context():
+    db.create_all()
+
+@login_manager.user_loader
+def load_user(user_id):
+    return User.query.get(int(user_id))
+
+@app.route('/signup', methods=['GET', 'POST'])
+def signup():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        hashed_password = generate_password_hash(password, method='sha256')
+
+        new_user = User(username=username, password=hashed_password)
+        db.session.add(new_user)
+        db.session.commit()
+        flash('Signup successful!', 'success')
+        return redirect(url_for('login'))
+
+    return render_template('signup.html')
+
+@app.route('/login', methods=['GET', 'POST'])
+def login():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        user = User.query.filter_by(username=username).first()
+        if user and check_password_hash(user.password, password):
+            login_user(user)
+            return redirect(url_for('dashboard'))
+        flash('Invalid username or password', 'danger')
+
+    return render_template('login.html')
+
+@app.route('/dashboard')
+@login_required
+def dashboard():
+    return render_template('dashboard.html', name=current_user.username)
+
+@app.route('/logout')
+@login_required
+def logout():
+    logout_user()
+    return redirect(url_for('login'))
+
+if __name__ == '__main__':
+    app.run(debug=True)
+model_dir = "C:/Users/KIIT/OneDrive/Desktop/INTEL/phi-2/INT8_compressed_weights"
+print(f"Checking model directory: {model_dir}")
+print(f"Contents: {os.listdir(model_dir)}")  # Check contents of the directory
+
+print(f"Loading model from {model_dir}")
+
+
+model_name = "susnato/phi-2"
+model_configuration = SUPPORTED_LLM_MODELS["phi-2"]
+ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
+
+tok = AutoTokenizer.from_pretrained(model_name)
+
+ov_model = OVModelForCausalLM.from_pretrained(
+    model_dir,
+    device="CPU",
+    ov_config=ov_config,
+)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+tokenizer_kwargs = model_configuration.get("toeknizer_kwargs", {})
+# Continue with your tokenizer usage
+response_key = model_configuration.get("response_key")
+tokenizer_response_key = None
+
+def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:
+    """
+    Gets the token ID for a given string that has been added to the tokenizer as a special token.
+
+    Args:
+        tokenizer (PreTrainedTokenizer): the tokenizer
+        key (str): the key to convert to a single token
+
+    Raises:
+        ValueError: if more than one ID was generated
+
+    Returns:
+        int: the token ID for the given key
+    """
+    token_ids = tokenizer.encode(key)
+    if len(token_ids) > 1:
+        raise ValueError(f"Expected only a single token for '{key}' but found {token_ids}")
+    return token_ids[0]
+if response_key is not None:
+    tokenizer_response_key = next(
+        (token for token in tokenizer.additional_special_tokens if token.startswith(response_key)),
+        None,
+    )
+
+end_key_token_id = None
+if tokenizer_response_key:
+    try:
+        end_key = model_configuration.get("end_key")
+        if end_key:
+            end_key_token_id =get_special_token_id(tokenizer, end_key)
+        # Ensure generation stops once it generates "### End"
+    except ValueError:
+        pass
+
+prompt_template = model_configuration.get("prompt_template", "{instruction}")
+end_key_token_id = end_key_token_id or tokenizer.eos_token_id
+pad_token_id = end_key_token_id or tokenizer.pad_token_id
+
+def estimate_latency(
+    current_time: float,
+    current_perf_text: str,
+    new_gen_text: str,
+    per_token_time: List[float],
+    num_tokens: int,
+):
+    """
+    Helper function for performance estimation
+
+    Parameters:
+      current_time (float): This step time in seconds.
+      current_perf_text (str): Current content of performance UI field.
+      new_gen_text (str): New generated text.
+      per_token_time (List[float]): history of performance from previous steps.
+      num_tokens (int): Total number of generated tokens.
+
+    Returns:
+      update for performance text field
+      update for a total number of tokens
+    """
+    num_current_toks = len(tokenizer.encode(new_gen_text))
+    num_tokens += num_current_toks
+    per_token_time.append(num_current_toks / current_time)
+    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:
+        current_bucket = per_token_time[:-10]
+        return (
+            f"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}",
+            num_tokens,
+        )
+    return current_perf_text, num_tokens
+def run_generation(
+    user_text: str,
+    top_p: float,
+    temperature: float,
+    top_k: int,
+    max_new_tokens: int,
+    perf_text: str,
+):
+    """
+    Text generation function
+
+    Parameters:
+      user_text (str): User-provided instruction for a generation.
+      top_p (float):  Nucleus sampling. If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.
+      temperature (float): The value used to module the logits distribution.
+      top_k (int): The number of highest probability vocabulary tokens to keep for top-k-filtering.
+      max_new_tokens (int): Maximum length of generated sequence.
+      perf_text (str): Content of text field for printing performance results.
+    Returns:
+      model_output (str) - model-generated text
+      perf_text (str) - updated perf text filed content
+    """
+
+    # Prepare input prompt according to model expected template
+    prompt_text = prompt_template.format(instruction=user_text)
+
+    # Tokenize the user text.
+    model_inputs = tokenizer(prompt_text, return_tensors="pt", **tokenizer_kwargs)
+
+    # Start generation on a separate thread, so that we don't block the UI. The text is pulled from the streamer
+    # in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.
+    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
+    generate_kwargs = dict(
+        model_inputs,
+        streamer=streamer,
+        max_new_tokens=max_new_tokens,
+        do_sample=True,
+        top_p=top_p,
+        temperature=float(temperature),
+        top_k=top_k,
+        eos_token_id=end_key_token_id,
+        pad_token_id=pad_token_id,
+    )
+    t = Thread(target=ov_model.generate, kwargs=generate_kwargs)
+    t.start()
+
+    # Pull the generated text from the streamer, and update the model output.
+    model_output = ""
+    per_token_time = []
+    num_tokens = 0
+    start = perf_counter()
+    for new_text in streamer:
+        current_time = perf_counter() - start
+        model_output += new_text
+        perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)
+        yield model_output, perf_text
+        start = perf_counter()
+    return model_output, perf_text
+def reset_textbox(instruction: str, response: str, perf: str):
+    """
+    Helper function for resetting content of all text fields
+
+    Parameters:
+      instruction (str): Content of user instruction field.
+      response (str): Content of model response field.
+      perf (str): Content of performance info filed
+
+    Returns:
+      empty string for each placeholder
+    """
+    return "", "", ""
+
+
+
+examples = [
+    "Give me a recipe for pizza with pineapple",
+    "Write me a tweet about the new OpenVINO release",
+    "Explain the difference between CPU and GPU",
+    "Give five ideas for a great weekend with family",
+    "Do Androids dream of Electric sheep?",
+    "Who is Dolly?",
+    "Please give me advice on how to write resume?",
+    "Name 3 advantages to being a cat",
+    "Write instructions on how to become a good AI engineer",
+    "Write a love letter to my best friend",
+]
+
+def main():
+    with gr.Blocks() as demo:
+        gr.Markdown(
+            "# Question Answering with Model and OpenVINO.\n"
+            "Provide instruction which describes a task below or select among predefined examples and model writes response that performs requested task."
+        )
+
+        with gr.Row():
+            with gr.Column(scale=4):
+                user_text = gr.Textbox(
+                    placeholder="Write an email about an alpaca that likes flan",
+                    label="User instruction",
+                )
+                model_output = gr.Textbox(label="Model response", interactive=False)
+                performance = gr.Textbox(label="Performance", lines=1, interactive=False)
+                with gr.Column(scale=1):
+                    button_clear = gr.Button(value="Clear")
+                    button_submit = gr.Button(value="Submit")
+                gr.Examples(examples, user_text)
+            with gr.Column(scale=1):
+                max_new_tokens = gr.Slider(
+                    minimum=1,
+                    maximum=1000,
+                    value=256,
+                    step=1,
+                    interactive=True,
+                    label="Max New Tokens",
+                )
+                top_p = gr.Slider(
+                    minimum=0.05,
+                    maximum=1.0,
+                    value=0.92,
+                    step=0.05,
+                    interactive=True,
+                    label="Top-p (nucleus sampling)",
+                )
+                top_k = gr.Slider(
+                    minimum=0,
+                    maximum=50,
+                    value=0,
+                    step=1,
+                    interactive=True,
+                    label="Top-k",
+                )
+                temperature = gr.Slider(
+                    minimum=0.1,
+                    maximum=5.0,
+                    value=0.8,
+                    step=0.1,
+                    interactive=True,
+                    label="Temperature",
+                )
+
+        user_text.submit(
+            run_generation,
+            [user_text, top_p, temperature, top_k, max_new_tokens, performance],
+            [model_output, performance],
+        )
+        button_submit.click(
+            run_generation,
+            [user_text, top_p, temperature, top_k, max_new_tokens, performance],
+            [model_output, performance],
+        )
+        button_clear.click(
+            reset_textbox,
+            [user_text, model_output, performance],
+            [user_text, model_output, performance],
+        )
+
+    if __name__ == "__main__":
+        demo.queue()
+        try:
+            demo.launch(height=800)
+        except Exception:
+            demo.launch(share=True, height=800)
+
+# Call main function to start Gradio interface
+main()

app1.py

@@ -0,0 +1,78 @@
+# app.py
+
+import os
+from flask import Flask, render_template, redirect, url_for, request, flash
+from flask_sqlalchemy import SQLAlchemy
+from flask_login import LoginManager, UserMixin, login_user, login_required, logout_user, current_user
+from werkzeug.security import generate_password_hash, check_password_hash
+from app import run_generation
+
+app = Flask(__name__)
+app.config['SECRET_KEY'] = 'your_secret_key'
+app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///users.db'
+db = SQLAlchemy(app)
+login_manager = LoginManager()
+login_manager.init_app(app)
+login_manager.login_view = 'login'
+
+class User(db.Model, UserMixin):
+    id = db.Column(db.Integer, primary_key=True)
+    username = db.Column(db.String(80), unique=True, nullable=False)
+    password = db.Column(db.String(120), nullable=False)
+
+    def __repr__(self):
+        return '<User %r>' % self.username
+
+# Create the database tables
+with app.app_context():
+    db.drop_all()
+    db.create_all()
+
+@login_manager.user_loader
+def load_user(user_id):
+    return User.query.get(int(user_id))
+
+@app.route('/', methods=['GET', 'POST'])
+def signup():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        hashed_password = generate_password_hash(password, method='pbkdf2:sha256')
+
+        new_user = User(username=username, password=hashed_password)
+        db.session.add(new_user)
+        db.session.commit()
+        flash('Signup successful!', 'success')
+        return redirect(url_for('login'))
+
+    return render_template('signup.html')
+
+@app.route('/login', methods=['GET', 'POST'])
+def login():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        user = User.query.filter_by(username=username).first()
+        if user and check_password_hash(user.password, password):
+            login_user(user)
+            return redirect(url_for('dashboard'))
+        flash('Invalid username or password', 'danger')
+
+    return render_template('login.html')
+
+@app.route('/app.run_generation')
+@login_required
+
+@app.route('/dashboard')
+@login_required
+def dashboard():
+    return render_template('dashboard.html', name=current_user.username)
+
+@app.route('/logout')
+@login_required
+def logout():
+    logout_user()
+    return redirect(url_for('login'))
+
+if __name__ == '__main__':
+    app.run(debug=True)

app2.py

@@ -0,0 +1,78 @@
+from flask import Flask, render_template, redirect, url_for, request, flash
+from flask_sqlalchemy import SQLAlchemy
+from flask_login import LoginManager, UserMixin, login_user, login_required, logout_user, current_user
+from werkzeug.security import generate_password_hash, check_password_hash
+
+app = Flask(__name__)
+app.config['SECRET_KEY'] = 'your_secret_key'
+app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///users.db'
+db = SQLAlchemy(app)
+login_manager = LoginManager()
+login_manager.init_app(app)
+login_manager.login_view = 'login'
+
+class User(db.Model, UserMixin):
+    id = db.Column(db.Integer, primary_key=True)
+    username = db.Column(db.String(80), unique=True, nullable=False)
+    password = db.Column(db.String(120), nullable=False)
+
+    def __repr__(self):
+        return '<User %r>' % self.username
+
+# Create the database tables
+with app.app_context():
+    db.drop_all()
+    db.create_all()
+
+@login_manager.user_loader
+def load_user(user_id):
+    return User.query.get(int(user_id))
+
+@app.route('/', methods=['GET', 'POST'])
+def signup():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        hashed_password = generate_password_hash(password, method='pbkdf2:sha256')
+
+        new_user = User(username=username, password=hashed_password)
+        db.session.add(new_user)
+        db.session.commit()
+        flash('Signup successful!', 'success')
+        return redirect(url_for('login'))
+
+    return render_template('signup.html')
+
+@app.route('/login', methods=['GET', 'POST'])
+def login():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        user = User.query.filter_by(username=username).first()
+        if user and check_password_hash(user.password, password):
+            login_user(user)
+            return redirect(url_for('dashboard'))
+        flash('Invalid username or password', 'danger')
+
+    return render_template('login.html')
+
+@app.route('/run_generation')
+@login_required
+def run_generation_route():
+    # Call the run_generation function here
+    result = run_generation()
+    return render_template('generation.html', result=result)
+
+@app.route('/dashboard')
+@login_required
+def dashboard():
+    return render_template('dashboard.html', name=current_user.username)
+
+@app.route('/logout')
+@login_required
+def logout():
+    logout_user()
+    return redirect(url_for('login'))
+
+if __name__ == '__main__':
+    app.run(debug=True)

app3.py

@@ -0,0 +1,171 @@
+import os
+from flask import Flask, render_template, redirect, url_for, request, flash
+from flask_sqlalchemy import SQLAlchemy
+from flask_login import LoginManager, UserMixin, login_user, login_required, logout_user, current_user
+from werkzeug.security import generate_password_hash, check_password_hash
+from transformers import AutoTokenizer
+from optimum.intel.openvino import OVModelForCausalLM
+import gradio as gr
+from threading import Thread
+from time import perf_counter
+from typing import List
+import numpy as np
+
+app = Flask(__name__)
+app.config['SECRET_KEY'] = 'your_secret_key'
+app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///users.db'
+db = SQLAlchemy(app)
+login_manager = LoginManager()
+login_manager.init_app(app)
+login_manager.login_view = 'login'
+
+class User(db.Model, UserMixin):
+    id = db.Column(db.Integer, primary_key=True)
+    username = db.Column(db.String(80), unique=True, nullable=False)
+    password = db.Column(db.String(120), nullable=False)
+
+    def __repr__(self):
+        return '<User %r>' % self.username
+
+# Create the database tables
+with app.app_context():
+    db.create_all()
+
+@login_manager.user_loader
+def load_user(user_id):
+    return User.query.get(int(user_id))
+
+@app.route('/', methods=['GET', 'POST'])
+def signup():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        hashed_password = generate_password_hash(password, method='pbkdf2:sha256')
+
+        new_user = User(username=username, password=hashed_password)
+        db.session.add(new_user)
+        db.session.commit()
+        flash('Signup successful!', 'success')
+        return redirect(url_for('login'))
+
+    return render_template('signup.html')
+
+@app.route('/login', methods=['GET', 'POST'])
+def login():
+    if request.method == 'POST':
+        username = request.form['username']
+        password = request.form['password']
+        user = User.query.filter_by(username=username).first()
+        if user and check_password_hash(user.password, password):
+            login_user(user)
+            return redirect(url_for('dashboard'))
+        flash('Invalid username or password', 'danger')
+
+    return render_template('login.html')
+
+@app.route('/dashboard')
+@login_required
+def dashboard():
+    return render_template('dashboard.html', name=current_user.username)
+
+@app.route('/logout')
+@login_required
+def logout():
+    logout_user()
+    return redirect(url_for('login'))
+
+# Gradio app integration
+model_dir = "C:/Users/KIIT/OneDrive/Desktop/INTEL/phi-2/INT8_compressed_weights"
+model_name = "susnato/phi-2"
+ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""}
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+ov_model = OVModelForCausalLM.from_pretrained(model_dir, device="CPU", ov_config=ov_config)
+
+prompt_template = "{instruction}"
+end_key_token_id = tokenizer.eos_token_id
+pad_token_id = tokenizer.pad_token_id
+
+def estimate_latency(current_time, current_perf_text, new_gen_text, per_token_time, num_tokens):
+    num_current_toks = len(tokenizer.encode(new_gen_text))
+    num_tokens += num_current_toks
+    per_token_time.append(num_current_toks / current_time)
+    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:
+        current_bucket = per_token_time[:-10]
+        return f"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}", num_tokens
+    return current_perf_text, num_tokens
+
+def run_generation(user_text, top_p, temperature, top_k, max_new_tokens, perf_text):
+    prompt_text = prompt_template.format(instruction=user_text)
+    model_inputs = tokenizer(prompt_text, return_tensors="pt")
+
+    streamer = gr.utils.TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
+    generate_kwargs = dict(
+        model_inputs,
+        streamer=streamer,
+        max_new_tokens=max_new_tokens,
+        do_sample=True,
+        top_p=top_p,
+        temperature=temperature,
+        top_k=top_k,
+        eos_token_id=end_key_token_id,
+        pad_token_id=pad_token_id,
+    )
+    t = Thread(target=ov_model.generate, kwargs=generate_kwargs)
+    t.start()
+
+    model_output = ""
+    per_token_time = []
+    num_tokens = 0
+    start = perf_counter()
+    for new_text in streamer:
+        current_time = perf_counter() - start
+        model_output += new_text
+        perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)
+        yield model_output, perf_text
+        start = perf_counter()
+    return model_output, perf_text
+
+def reset_textbox(instruction, response, perf):
+    return "", "", ""
+
+examples = [
+    "Give me a recipe for pizza with pineapple",
+    "Write me a tweet about the new OpenVINO release",
+    "Explain the difference between CPU and GPU",
+    "Give five ideas for a great weekend with family",
+    "Do Androids dream of Electric sheep?",
+    "Who is Dolly?",
+    "Please give me advice on how to write resume?",
+    "Name 3 advantages to being a cat",
+    "Write instructions on how to become a good AI engineer",
+    "Write a love letter to my best friend",
+]
+
+@app.route('/gradio')
+@login_required
+def gradio():
+    with gr.Blocks() as demo:
+        gr.Markdown("# Question Answering with Model and OpenVINO.\nProvide instruction which describes a task below or select among predefined examples and model writes response that performs requested task.")
+        with gr.Row():
+            with gr.Column(scale=4):
+                user_text = gr.Textbox(placeholder="Write an email about an alpaca that likes flan", label="User instruction")
+                model_output = gr.Textbox(label="Model response", interactive=False)
+                performance = gr.Textbox(label="Performance", lines=1, interactive=False)
+                with gr.Column(scale=1):
+                    button_clear = gr.Button(value="Clear")
+                    button_submit = gr.Button(value="Submit")
+                gr.Examples(examples, user_text)
+            with gr.Column(scale=1):
+                max_new_tokens = gr.Slider(minimum=1, maximum=1000, value=256, step=1, interactive=True, label="Max New Tokens")
+                top_p = gr.Slider(minimum=0.05, maximum=1.0, value=0.92, step=0.05, interactive=True, label="Top-p (nucleus sampling)")
+                top_k = gr.Slider(minimum=0, maximum=50, value=0, step=1, interactive=True, label="Top-k")
+                temperature = gr.Slider(minimum=0.1, maximum=5.0, value=0.8, step=0.1, interactive=True, label="Temperature")
+
+        user_text.submit(run_generation, [user_text, top_p, temperature, top_k, max_new_tokens, performance], [model_output, performance])
+        button_submit.click(run_generation, [user_text, top_p, temperature, top_k, max_new_tokens, performance], [model_output, performance])
+        button_clear.click(reset_textbox, [user_text, model_output, performance], [user_text, model_output, performance])
+
+    return demo.launch(share=True)
+
+if __name__ == '__main__':
+    app.run(debug=True)

config.py

@@ -0,0 +1,38 @@
+SUPPORTED_LLM_MODELS = {
+    "tiny-llama-1b": {
+        "model_id": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+        "prompt_template": "<|user|>\n{instruction}</s> \n<|assistant|>\n",
+        "tokenizer_kwargs": {"add_special_tokens": False},
+    },
+    "phi-2": {
+        "model_id": "susnato/phi-2",
+        "prompt_template": "Instruct:{instruction}\nOutput:",
+    },
+    "dolly-v2-3b": {
+        "model_id": "databricks/dolly-v2-3b",
+        "instriction_key": "### Instruction:",
+        "response_key": "### Response:",
+        "end_key": "### End",
+        "prompt_template": """Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{instruction}
+
+### Response:
+""",
+    },
+    "red-pajama-instruct-3b": {
+        "model_id": "togethercomputer/RedPajama-INCITE-Instruct-3B-v1",
+        "prompt_template": "Q: {instruction}\nA:",
+    },
+    "mistral-7b": {
+        "model_id": "mistralai/Mistral-7B-Instruct-v0.2",
+        "prompt_template": "<s> [INST] {instruction} [/INST] </s>",
+        "tokenizer_kwargs": {"add_special_tokens": False},
+    },
+    "llama-3-8b-instruct": {
+        "model_id": "meta-llama/Meta-Llama-3-8B-Instruct",
+        "end_key": "<|eot_id|>",
+        "prompt_template": "<|start_header_id|>system<|end_header_id|>\n\nBelow is an instruction that describes a task. Write a response that appropriately completes the request.<|eot_id|><|start_header_id|>user<|end_header_id|>Instruction: {instruction} Answer:<|eot_id|><|start_header_id|>assistant<|end_header_id|>",
+    },
+}

generation_utils.py

@@ -0,0 +1,201 @@
+# generation_utils.py
+
+from threading import Thread
+from time import perf_counter
+from typing import List
+import gradio as gr
+from transformers import AutoTokenizer, TextIteratorStreamer
+import numpy as np
+import os
+
+def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:
+    """
+    Gets the token ID for a given string that has been added to the tokenizer as a special token.
+
+    Args:
+        tokenizer (PreTrainedTokenizer): the tokenizer
+        key (str): the key to convert to a single token
+
+    Raises:
+        ValueError: if more than one ID was generated
+
+    Returns:
+        int: the token ID for the given key
+    """
+    token_ids = tokenizer.encode(key)
+    if len(token_ids) > 1:
+        raise ValueError(f"Expected only a single token for '{key}' but found {token_ids}")
+    return token_ids[0]
+
+
+def estimate_latency(
+    current_time: float,
+    current_perf_text: str,
+    new_gen_text: str,
+    per_token_time: List[float],
+    num_tokens: int,
+) -> tuple:
+    """
+    Helper function for performance estimation
+
+    Parameters:
+      current_time (float): This step time in seconds.
+      current_perf_text (str): Current content of performance UI field.
+      new_gen_text (str): New generated text.
+      per_token_time (List[float]): history of performance from previous steps.
+      num_tokens (int): Total number of generated tokens.
+
+    Returns:
+      update for performance text field
+      update for a total number of tokens
+    """
+    num_current_toks = len(tokenizer.encode(new_gen_text))
+    num_tokens += num_current_toks
+    per_token_time.append(num_current_toks / current_time)
+    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:
+        current_bucket = per_token_time[:-10]
+        return (
+            f"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}",
+            num_tokens,
+        )
+    return current_perf_text, num_tokens
+
+
+def run_generation(
+    user_text: str,
+    top_p: float,
+    temperature: float,
+    top_k: int,
+    max_new_tokens: int,
+    perf_text: str,
+    tokenizer: AutoTokenizer,
+    tokenizer_kwargs: dict,
+    model_configuration: dict,
+    ov_model,
+) -> tuple:
+    """
+    Text generation function
+
+    Parameters:
+      user_text (str): User-provided instruction for generation.
+      top_p (float): Nucleus sampling. If < 1, keeps smallest set of most probable tokens.
+      temperature (float): Modulates logits distribution.
+      top_k (int): Number of highest probability vocabulary tokens to keep for top-k-filtering.
+      max_new_tokens (int): Maximum length of generated sequence.
+      perf_text (str): Content of text field for performance results.
+      tokenizer (AutoTokenizer): The tokenizer object.
+      tokenizer_kwargs (dict): Additional kwargs for tokenizer.
+      model_configuration (dict): Configuration for the model.
+      ov_model: Your OpenVINO model object.
+
+    Returns:
+      model_output (str): Model-generated text.
+      perf_text (str): Updated performance text.
+    """
+
+    # Extract necessary configurations from model_configuration
+    response_key = model_configuration.get("response_key")
+    prompt_template = model_configuration.get("prompt_template", "{instruction}")
+    end_key = model_configuration.get("end_key")
+    end_key_token_id = None
+
+    # Handle special tokens
+    if response_key:
+        tokenizer_response_key = next(
+            (token for token in tokenizer.additional_special_tokens if token.startswith(response_key)),
+            None,
+        )
+        if tokenizer_response_key and end_key:
+            try:
+                end_key_token_id = get_special_token_id(tokenizer, end_key)
+            except ValueError:
+                pass
+
+    # Ensure defaults for token IDs
+    end_key_token_id = end_key_token_id or tokenizer.eos_token_id
+    pad_token_id = end_key_token_id or tokenizer.pad_token_id
+
+    # Prepare input prompt according to model expected template
+    prompt_text = prompt_template.format(instruction=user_text)
+
+    # Tokenize the user text.
+    model_inputs = tokenizer(prompt_text, return_tensors="pt", **tokenizer_kwargs)
+
+    # Start generation on a separate thread, so that we don't block the UI.
+    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
+    generate_kwargs = {
+        **model_inputs,
+        "streamer": streamer,
+        "max_new_tokens": max_new_tokens,
+        "do_sample": True,
+        "top_p": top_p,
+        "temperature": float(temperature),
+        "top_k": top_k,
+        "eos_token_id": end_key_token_id,
+        "pad_token_id": pad_token_id,
+    }
+
+    # Start generation in a separate thread
+    t = Thread(target=ov_model.generate, kwargs=generate_kwargs)
+    t.start()
+
+    # Pull the generated text from the streamer and update model output
+    model_output = ""
+    per_token_time = []
+    num_tokens = 0
+    start = perf_counter()
+
+    for new_text in streamer:
+        current_time = perf_counter() - start
+        model_output += new_text
+        perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)
+        yield model_output, perf_text
+        start = perf_counter()
+
+    return model_output, perf_text
+def estimate_latency(
+    current_time: float,
+    current_perf_text: str,
+    new_gen_text: str,
+    per_token_time: List[float],
+    num_tokens: int,
+):
+    """
+    Helper function for performance estimation
+
+    Parameters:
+      current_time (float): This step time in seconds.
+      current_perf_text (str): Current content of performance UI field.
+      new_gen_text (str): New generated text.
+      per_token_time (List[float]): history of performance from previous steps.
+      num_tokens (int): Total number of generated tokens.
+
+    Returns:
+      update for performance text field
+      update for a total number of tokens
+    """
+    num_current_toks = len(tokenizer.encode(new_gen_text))
+    num_tokens += num_current_toks
+    per_token_time.append(num_current_toks / current_time)
+    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:
+        current_bucket = per_token_time[:-10]
+        return (
+            f"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}",
+            num_tokens,
+        )
+    return current_perf_text, num_tokens
+
+
+def reset_textbox(instruction: str, response: str, perf: str):
+    """
+    Helper function for resetting content of all text fields
+
+    Parameters:
+      instruction (str): Content of user instruction field.
+      response (str): Content of model response field.
+      perf (str): Content of performance info filed
+
+    Returns:
+      empty string for each placeholder
+    """
+    return "", "", ""

llm-question-answering-Copy1 (1).ipynb

@@ -0,0 +1,1249 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f6aa8c52-9bde-41a6-a5f7-4fa93d5c2a6c",
+   "metadata": {},
+   "source": [
+    "# LLM Instruction-following pipeline with OpenVINO \n",
+    "\n",
+    "LLM stands for “Large Language Model,” which refers to a type of artificial intelligence model that is designed to understand and generate human-like text based on the input it receives. LLMs are trained on large datasets of text to learn patterns, grammar, and semantic relationships, allowing them to generate coherent and contextually relevant responses. One core capability of Large Language Models (LLMs) is to follow natural language instructions. Instruction-following models are capable of generating text in response to prompts and are often used for tasks like writing assistance, chatbots, and content generation.\n",
+    "\n",
+    "In this tutorial, we consider how to run an instruction-following text generation pipeline using popular LLMs and OpenVINO. We will use pre-trained models from the [Hugging Face Transformers](https://huggingface.co/docs/transformers/index) library. To simplify the user experience, the [Hugging Face Optimum Intel](https://huggingface.co/docs/optimum/intel/index) library converts the models to OpenVINO™ IR format.\n",
+    "\n",
+    "The tutorial consists of the following steps:\n",
+    "\n",
+    "- Install prerequisites\n",
+    "- Download and convert the model from a public source using the [OpenVINO integration with Hugging Face Optimum](https://huggingface.co/blog/openvino).\n",
+    "- Compress model weights to INT8 and INT4 with [OpenVINO NNCF](https://github.com/openvinotoolkit/nncf)\n",
+    "- Create an instruction-following inference pipeline\n",
+    "- Run instruction-following pipeline\n",
+    "\n",
+    "\n",
+    "#### Table of contents:\n",
+    "\n",
+    "- [Prerequisites](#Prerequisites)\n",
+    "- [Select model for inference](#Select-model-for-inference)\n",
+    "- [Instantiate Model using Optimum Intel](#Instantiate-Model-using-Optimum-Intel)\n",
+    "- [Compress model weights](#Compress-model-weights)\n",
+    "    - [Weights Compression using Optimum Intel](#Weights-Compression-using-Optimum-Intel)\n",
+    "    - [Weights Compression using NNCF](#Weights-Compression-using-NNCF)\n",
+    "- [Select device for inference and model variant](#Select-device-for-inference-and-model-variant)\n",
+    "- [Create an instruction-following inference pipeline](#Create-an-instruction-following-inference-pipeline)\n",
+    "    - [Setup imports](#Setup-imports)\n",
+    "    - [Prepare template for user prompt](#Prepare-template-for-user-prompt)\n",
+    "    - [Main generation function](#Main-generation-function)\n",
+    "    - [Helpers for application](#Helpers-for-application)\n",
+    "- [Run instruction-following pipeline](#Run-instruction-following-pipeline)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "027108c2-1fbe-4be5-9e23-3fc359185a42",
+   "metadata": {},
+   "source": [
+    "## Prerequisites\n",
+    "[back to top ⬆️](#Table-of-contents:)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5d0473c6-3734-422d-a370-2e39d576be0e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "WARNING: Ignoring invalid distribution -ptimum (c:\\users\\kiit\\appdata\\local\\programs\\python\\python310\\lib\\site-packages)\n",
+      "WARNING: Ignoring invalid distribution -ptimum (c:\\users\\kiit\\appdata\\local\\programs\\python\\python310\\lib\\site-packages)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "WARNING: Ignoring invalid distribution -ptimum (c:\\users\\kiit\\appdata\\local\\programs\\python\\python310\\lib\\site-packages)\n",
+      "WARNING: Ignoring invalid distribution -ptimum (c:\\users\\kiit\\appdata\\local\\programs\\python\\python310\\lib\\site-packages)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "WARNING: Ignoring invalid distribution -ptimum (c:\\users\\kiit\\appdata\\local\\programs\\python\\python310\\lib\\site-packages)\n",
+      "WARNING: Ignoring invalid distribution -ptimum (c:\\users\\kiit\\appdata\\local\\programs\\python\\python310\\lib\\site-packages)\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install -Uq pip\n",
+    "%pip uninstall -q -y optimum optimum-intel\n",
+    "%pip install --pre -Uq openvino openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n",
+    "%pip install -q \"torch>=2.1\" \"nncf>=2.7\" \"transformers>=4.36.0\" onnx \"optimum>=1.16.1\" \"accelerate\" \"datasets>=2.14.6\" \"gradio>=4.19\" \"git+https://github.com/huggingface/optimum-intel.git\" --extra-index-url https://download.pytorch.org/whl/cpu"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "611cc777-d5bc-4c7b-92e4-a4befa13b2ce",
+   "metadata": {},
+   "source": [
+    "## Select model for inference\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "The tutorial supports different models, you can select one from the provided options to compare the quality of open source LLM solutions.\n",
+    ">**Note**: conversion of some models can require additional actions from user side and at least 64GB RAM for conversion.\n",
+    "\n",
+    "The available options are:\n",
+    "\n",
+    "* **tiny-llama-1b-chat** - This is the chat model finetuned on top of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T). The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens with the adoption of the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint. More details about model can be found in [model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)\n",
+    "* **phi-2** - Phi-2 is a Transformer with 2.7 billion parameters. It was trained using the same data sources as [Phi-1.5](https://huggingface.co/microsoft/phi-1_5), augmented with a new data source that consists of various NLP synthetic texts and filtered websites (for safety and educational value). When assessed against benchmarks testing common sense, language understanding, and logical reasoning, Phi-2 showcased a nearly state-of-the-art performance among models with less than 13 billion parameters. More details about model can be found in [model card](https://huggingface.co/microsoft/phi-2#limitations-of-phi-2).\n",
+    "* **dolly-v2-3b** - Dolly 2.0 is an instruction-following large language model trained on the Databricks machine-learning platform that is licensed for commercial use. It is based on [Pythia](https://github.com/EleutherAI/pythia) and is trained on ~15k instruction/response fine-tuning records generated by Databricks employees in various capability domains, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization. Dolly 2.0 works by processing natural language instructions and generating responses that follow the given instructions. It can be used for a wide range of applications, including closed question-answering, summarization, and generation. More details about model can be found in [model card](https://huggingface.co/databricks/dolly-v2-3b).\n",
+    "* **red-pajama-3b-instruct** -  A 2.8B parameter pre-trained language model based on GPT-NEOX architecture. The model was fine-tuned for few-shot applications on the data of [GPT-JT](https://huggingface.co/togethercomputer/GPT-JT-6B-v1), with exclusion of tasks that overlap with the HELM core scenarios.More details about model can be found in [model card](https://huggingface.co/togethercomputer/RedPajama-INCITE-Instruct-3B-v1).\n",
+    "* **mistral-7b** - The Mistral-7B-v0.2 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. You can find more details about model in the [model card](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2), [paper](https://arxiv.org/abs/2310.06825) and [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n",
+    "* **llama-3-8b-instruct** - Llama 3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. The Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks. More details about model can be found in [Meta blog post](https://ai.meta.com/blog/meta-llama-3/), [model website](https://llama.meta.com/llama3) and [model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct).\n",
+    ">**Note**: run model with demo, you will need to accept license agreement. \n",
+    ">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), carefully read terms of usage and click accept button.  You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
+    ">You can login on Hugging Face Hub in notebook environment, using following code:\n",
+    " \n",
+    "```python\n",
+    "    ## login to huggingfacehub to get access to pretrained model \n",
+    "\n",
+    "    from huggingface_hub import notebook_login, whoami\n",
+    "\n",
+    "    try:\n",
+    "        whoami()\n",
+    "        print('Authorization token already provided')\n",
+    "    except OSError:\n",
+    "        notebook_login()\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "51920510-effc-49ef-8c6f-81c951d96a9b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import requests\n",
+    "\n",
+    "# Fetch `notebook_utils` module\n",
+    "r = requests.get(\n",
+    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
+    ")\n",
+    "open(\"notebook_utils.py\", \"w\").write(r.text)\n",
+    "from notebook_utils import download_file\n",
+    "\n",
+    "if not Path(\"./config.py\").exists():\n",
+    "    download_file(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/llm-question-answering/config.py\")\n",
+    "from config import SUPPORTED_LLM_MODELS\n",
+    "import ipywidgets as widgets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "27b42290-a9b5-4453-9a4c-ffa44bbd966d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ac9109fc5e134507839e2b6da803cc48",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Model:', index=1, options=('tiny-llama-1b', 'phi-2', 'dolly-v2-3b', 'red-pajama-instruct…"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model_ids = list(SUPPORTED_LLM_MODELS)\n",
+    "\n",
+    "model_id = widgets.Dropdown(\n",
+    "    options=model_ids,\n",
+    "    value=model_ids[1],\n",
+    "    description=\"Model:\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "model_id"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "37e9634f-4fc7-4d9c-9ade-b3e8684a0828",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Selected model phi-2\n"
+     ]
+    }
+   ],
+   "source": [
+    "model_configuration = SUPPORTED_LLM_MODELS[model_id.value]\n",
+    "print(f\"Selected model {model_id.value}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "7937959a-d2a1-49bd-bf12-35554fa901d1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'model_id': 'susnato/phi-2',\n",
+       " 'prompt_template': 'Instruct:{instruction}\\nOutput:'}"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model_configuration = SUPPORTED_LLM_MODELS[model_id.value]\n",
+    "model_configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e4fd394-b4fb-4eef-8bdc-d116572aa8f0",
+   "metadata": {},
+   "source": [
+    "## Instantiate Model using Optimum Intel\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "Optimum Intel can be used to load optimized models from the [Hugging Face Hub](https://huggingface.co/docs/optimum/intel/hf.co/models) and create pipelines to run an inference with OpenVINO Runtime using Hugging Face APIs. The Optimum Inference models are API compatible with Hugging Face Transformers models.  This means we just need to replace `AutoModelForXxx` class with the corresponding `OVModelForXxx` class.\n",
+    "\n",
+    "Below is an example of the RedPajama model\n",
+    "\n",
+    "```diff\n",
+    "-from transformers import AutoModelForCausalLM\n",
+    "+from optimum.intel.openvino import OVModelForCausalLM\n",
+    "from transformers import AutoTokenizer, pipeline\n",
+    "\n",
+    "model_id = \"togethercomputer/RedPajama-INCITE-Chat-3B-v1\"\n",
+    "-model = AutoModelForCausalLM.from_pretrained(model_id)\n",
+    "+model = OVModelForCausalLM.from_pretrained(model_id, export=True)\n",
+    "```\n",
+    "\n",
+    "Model class initialization starts with calling `from_pretrained` method. When downloading and converting the Transformers model, the parameter `export=True` should be added. We can save the converted model for the next usage with the `save_pretrained` method.\n",
+    "Tokenizer class and pipelines API are compatible with Optimum models.\n",
+    "\n",
+    "To optimize the generation process and use memory more efficiently, the `use_cache=True` option is enabled. Since the output side is auto-regressive, an output token hidden state remains the same once computed for every further generation step. Therefore, recomputing it every time you want to generate a new token seems wasteful. With the cache, the model saves the hidden state once it has been computed. The model only computes the one for the most recently generated output token at each time step, re-using the saved ones for hidden tokens. This reduces the generation complexity from $O(n^3)$ to $O(n^2)$ for a transformer model. More details about how it works can be found in this [article](https://scale.com/blog/pytorch-improvements#Text%20Translation). With this option, the model gets the previous step's hidden states (cached attention keys and values) as input and additionally provides hidden states for the current step as output. It means for all next iterations, it is enough to provide only a new token obtained from the previous step and cached key values to get the next token prediction. \n",
+    "\n",
+    "## Compress model weights\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "The Weights Compression algorithm is aimed at compressing the weights of the models and can be used to optimize the model footprint and performance of large models where the size of weights is relatively larger than the size of activations, for example, Large Language Models (LLM). Compared to INT8 compression, INT4 compression improves performance even more but introduces a minor drop in prediction quality.\n",
+    "\n",
+    "\n",
+    "### Weights Compression using Optimum Intel\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "Optimum Intel supports weight compression via NNCF out of the box. For 8-bit compression we pass `load_in_8bit=True` to `from_pretrained()` method of `OVModelForCausalLM`. For 4 bit compression we provide `quantization_config=OVWeightQuantizationConfig(bits=4, ...)` argument containing number of bits and other compression parameters. An example of this approach usage you can find in [llm-chatbot notebook](../llm-chatbot)\n",
+    "\n",
+    "### Weights Compression using NNCF\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "You also can perform weights compression for OpenVINO models using NNCF directly. `nncf.compress_weights` function accepts the OpenVINO model instance and compresses its weights for Linear and Embedding layers. We will consider this variant in this notebook for both int4 and int8 compression.\n",
+    "\n",
+    "\n",
+    ">**Note**: This tutorial involves conversion model for FP16 and INT4/INT8 weights compression scenarios. It may be memory and time-consuming in the first run. You can manually control the compression precision below.\n",
+    ">**Note**: There may be no speedup for INT4/INT8 compressed models on dGPU"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "f81602ca-4674-4b61-b2c8-ca11631428b1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "03c417061a1845e9bceb4e40078ffc67",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Checkbox(value=True, description='Prepare INT4 model')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "6dac25b3d6824cfebae7f076cefaac98",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Checkbox(value=False, description='Prepare INT8 model')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "4fac7143c54545c4944f3dc4d83a8a71",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Checkbox(value=False, description='Prepare FP16 model')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from IPython.display import display\n",
+    "\n",
+    "prepare_int4_model = widgets.Checkbox(\n",
+    "    value=True,\n",
+    "    description=\"Prepare INT4 model\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "prepare_int8_model = widgets.Checkbox(\n",
+    "    value=False,\n",
+    "    description=\"Prepare INT8 model\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "prepare_fp16_model = widgets.Checkbox(\n",
+    "    value=False,\n",
+    "    description=\"Prepare FP16 model\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "display(prepare_int4_model)\n",
+    "display(prepare_int8_model)\n",
+    "display(prepare_fp16_model)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "0066fbec-b89b-4caa-94fb-9ea8598c22e0",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Framework not specified. Using pt to export the model.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ac0d64ad764b4edf96d653f7634b3be6",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
+      "Using framework PyTorch: 2.3.1+cpu\n",
+      "Overriding 1 configuration item(s)\n",
+      "\t- use_cache -> True\n",
+      "C:\\Users\\KIIT\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\transformers\\modeling_attn_mask_utils.py:114: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
+      "  if (input_shape[-1] > 1 or self.sliding_window is not None) and self.is_causal:\n",
+      "C:\\Users\\KIIT\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\optimum\\exporters\\onnx\\model_patcher.py:303: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
+      "  if past_key_values_length > 0:\n",
+      "C:\\Users\\KIIT\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\transformers\\models\\phi\\modeling_phi.py:107: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
+      "  if seq_len > self.max_seq_len_cached:\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "057d63d7db3943f7b0549182c24e97fe",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Output()"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
+      ],
+      "text/plain": []
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
+       "</pre>\n"
+      ],
+      "text/plain": [
+       "\n"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Configuration saved in phi-2\\INT8_compressed_weights\\openvino_config.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "import logging\n",
+    "import openvino as ov\n",
+    "import nncf\n",
+    "from optimum.intel.openvino import OVModelForCausalLM, OVWeightQuantizationConfig\n",
+    "import gc\n",
+    "\n",
+    "\n",
+    "nncf.set_log_level(logging.ERROR)\n",
+    "\n",
+    "pt_model_id = model_configuration[\"model_id\"]\n",
+    "fp16_model_dir = Path(model_id.value) / \"FP16\"\n",
+    "int8_model_dir = Path(model_id.value) / \"INT8_compressed_weights\"\n",
+    "int4_model_dir = Path(model_id.value) / \"INT4_compressed_weights\"\n",
+    "\n",
+    "core = ov.Core()\n",
+    "\n",
+    "\n",
+    "def convert_to_fp16():\n",
+    "    if (fp16_model_dir / \"openvino_model.xml\").exists():\n",
+    "        return\n",
+    "    ov_model = OVModelForCausalLM.from_pretrained(pt_model_id, export=True, compile=False, load_in_8bit=False)\n",
+    "    ov_model.half()\n",
+    "    ov_model.save_pretrained(fp16_model_dir)\n",
+    "    del ov_model\n",
+    "    gc.collect()\n",
+    "\n",
+    "\n",
+    "def convert_to_int8():\n",
+    "    if (int8_model_dir / \"openvino_model.xml\").exists():\n",
+    "        return\n",
+    "    ov_model = OVModelForCausalLM.from_pretrained(pt_model_id, export=True, compile=False, load_in_8bit=True)\n",
+    "    ov_model.save_pretrained(int8_model_dir)\n",
+    "    del ov_model\n",
+    "    gc.collect()\n",
+    "\n",
+    "\n",
+    "def convert_to_int4():\n",
+    "    compression_configs = {\n",
+    "        \"mistral-7b\": {\n",
+    "            \"sym\": True,\n",
+    "            \"group_size\": 64,\n",
+    "            \"ratio\": 0.6,\n",
+    "        },\n",
+    "        \"red-pajama-3b-instruct\": {\n",
+    "            \"sym\": False,\n",
+    "            \"group_size\": 128,\n",
+    "            \"ratio\": 0.5,\n",
+    "        },\n",
+    "        \"dolly-v2-3b\": {\"sym\": False, \"group_size\": 32, \"ratio\": 0.5},\n",
+    "        \"llama-3-8b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
+    "        \"default\": {\n",
+    "            \"sym\": False,\n",
+    "            \"group_size\": 128,\n",
+    "            \"ratio\": 0.8,\n",
+    "        },\n",
+    "    }\n",
+    "\n",
+    "    model_compression_params = compression_configs.get(model_id.value, compression_configs[\"default\"])\n",
+    "    if (int4_model_dir / \"openvino_model.xml\").exists():\n",
+    "        return\n",
+    "    ov_model = OVModelForCausalLM.from_pretrained(\n",
+    "        pt_model_id,\n",
+    "        export=True,\n",
+    "        compile=False,\n",
+    "        quantization_config=OVWeightQuantizationConfig(bits=4, **model_compression_params),\n",
+    "    )\n",
+    "    ov_model.save_pretrained(int4_model_dir)\n",
+    "    del ov_model\n",
+    "    gc.collect()\n",
+    "\n",
+    "\n",
+    "if prepare_fp16_model.value:\n",
+    "    convert_to_fp16()\n",
+    "if prepare_int8_model.value:\n",
+    "    convert_to_int8()\n",
+    "if prepare_int4_model.value:\n",
+    "    convert_to_int4()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b5f31838-c7d6-4f52-aa2d-1f29c6f3b397",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "60355f86-2250-4ebe-82ac-950f2d4fb01b",
+   "metadata": {},
+   "source": [
+    "Let's compare model size for different compression types"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "42c7b254-1ce4-4f23-813a-9bdc23aed327",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Size of model with INT8 compressed weights is 2656.55 MB\n",
+      "Size of model with INT4 compressed weights is 1739.13 MB\n"
+     ]
+    }
+   ],
+   "source": [
+    "fp16_weights = fp16_model_dir / \"openvino_model.bin\"\n",
+    "int8_weights = int8_model_dir / \"openvino_model.bin\"\n",
+    "int4_weights = int4_model_dir / \"openvino_model.bin\"\n",
+    "\n",
+    "if fp16_weights.exists():\n",
+    "    print(f\"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB\")\n",
+    "for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):\n",
+    "    if compressed_weights.exists():\n",
+    "        print(f\"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB\")\n",
+    "    if compressed_weights.exists() and fp16_weights.exists():\n",
+    "        print(f\"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3df73379-bccc-41b1-9c94-c3040819805b",
+   "metadata": {},
+   "source": [
+    "## Select device for inference and model variant\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    ">**Note**: There may be no speedup for INT4/INT8 compressed models on dGPU."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "d2d7bf5b-8a05-4c3b-a36b-631af5c197e9",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "49fc667dcf434c9f80db2d130b45212b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "core = ov.Core()\n",
+    "\n",
+    "support_devices = core.available_devices\n",
+    "if \"NPU\" in support_devices:\n",
+    "    support_devices.remove(\"NPU\")\n",
+    "\n",
+    "device = widgets.Dropdown(\n",
+    "    options=support_devices + [\"AUTO\"],\n",
+    "    value=\"CPU\",\n",
+    "    description=\"Device:\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "device"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "01673d97-6645-4d2a-8306-293b8064b317",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "49fc667dcf434c9f80db2d130b45212b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "device"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "24532480-80a5-4953-9cd6-78ac51a1cd8f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "58cde7d954a748baa50af1f82626c883",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Dropdown(description='Model to run:', options=('INT4', 'INT8'), value='INT4')"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "available_models = []\n",
+    "if int4_model_dir.exists():\n",
+    "    available_models.append(\"INT4\")\n",
+    "if int8_model_dir.exists():\n",
+    "    available_models.append(\"INT8\")\n",
+    "if fp16_model_dir.exists():\n",
+    "    available_models.append(\"FP16\")\n",
+    "\n",
+    "model_to_run = widgets.Dropdown(\n",
+    "    options=available_models,\n",
+    "    value=available_models[0],\n",
+    "    description=\"Model to run:\",\n",
+    "    disabled=False,\n",
+    ")\n",
+    "\n",
+    "model_to_run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "5259c1c5-4128-4210-9ad2-faf33ee40e86",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading model from phi-2\\INT8_compressed_weights\n",
+      "susnato/phi-2\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
+      "Compiling the model to CPU ...\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "if model_to_run.value == \"INT4\":\n",
+    "    model_dir = int4_model_dir\n",
+    "elif model_to_run.value == \"INT8\":\n",
+    "    model_dir = int8_model_dir\n",
+    "else:\n",
+    "    model_dir = fp16_model_dir\n",
+    "print(f\"Loading model from {model_dir}\")\n",
+    "\n",
+    "model_name = model_configuration[\"model_id\"]\n",
+    "print(model_name)\n",
+    "ov_config = {\"PERFORMANCE_HINT\": \"LATENCY\", \"NUM_STREAMS\": \"1\", \"CACHE_DIR\": \"\"}\n",
+    "\n",
+    "tok = AutoTokenizer.from_pretrained(model_name)\n",
+    "\n",
+    "ov_model = OVModelForCausalLM.from_pretrained(\n",
+    "    model_dir,\n",
+    "    device=device.value,\n",
+    "    ov_config=ov_config,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bf13f6c3-6671-408e-ae0e-aaa3d8a6eaac",
+   "metadata": {},
+   "source": [
+    "## Create an instruction-following inference pipeline\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    " \n",
+    " The `run_generation` function accepts user-provided text input, tokenizes it, and runs the generation process. Text generation is an iterative process, where each next token depends on previously generated until a maximum number of tokens or stop generation condition is not reached. To obtain intermediate generation results without waiting until when generation is finished, we will use [`TextIteratorStreamer`](https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.TextIteratorStreamer), provided as part of HuggingFace [Streaming API](https://huggingface.co/docs/transformers/main/en/generation_strategies#streaming).\n",
+    " \n",
+    "The diagram below illustrates how the instruction-following pipeline works\n",
+    "\n",
+    "![generation pipeline)](https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/e881f4a4-fcc8-427a-afe1-7dd80aebd66e)\n",
+    "\n",
+    "As can be seen, on the first iteration, the user provided instructions converted to token ids using a tokenizer, then prepared input provided to the model. The model generates probabilities for all tokens in logits format  The way the next token will be selected over predicted probabilities is driven by the selected decoding methodology. You can find more information about the most popular decoding methods in this [blog](https://huggingface.co/blog/how-to-generate).\n",
+    "\n",
+    "There are several parameters that can control text generation quality:\n",
+    "\n",
+    "  * `Temperature` is a parameter used to control the level of creativity in AI-generated text. By adjusting the `temperature`, you can influence the AI model's probability distribution, making the text more focused or diverse.  \n",
+    "  Consider the following example: The AI model has to complete the sentence \"The cat is ____.\" with the following token probabilities:  \n",
+    "\n",
+    "    playing: 0.5  \n",
+    "    sleeping: 0.25  \n",
+    "    eating: 0.15  \n",
+    "    driving: 0.05  \n",
+    "    flying: 0.05  \n",
+    "\n",
+    "    - **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as \"playing.\"  \n",
+    "    - **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as \"playing,\" \"sleeping,\" or \"eating.\"  \n",
+    "    - **High temperature** (e.g., 2.0): The AI model becomes more adventurous, increasing the chances of selecting less likely tokens, such as \"driving\" and \"flying.\"\n",
+    "  * `Top-p`, also known as nucleus sampling, is a parameter used to control the range of tokens considered by the AI model based on their cumulative probability. By adjusting the `top-p` value, you can influence the AI model's token selection, making it more focused or diverse.\n",
+    "  Using the same example with the cat, consider the following top_p settings:  \n",
+    "    - **Low top_p** (e.g., 0.5): The AI model considers only tokens with the highest cumulative probability, such as \"playing.\"  \n",
+    "    - **Medium top_p** (e.g., 0.8): The AI model considers tokens with a higher cumulative probability, such as \"playing,\" \"sleeping,\" and \"eating.\"  \n",
+    "    - **High top_p** (e.g., 1.0): The AI model considers all tokens, including those with lower probabilities, such as \"driving\" and \"flying.\" \n",
+    "  * `Top-k` is another popular sampling strategy. In comparison with Top-P, which chooses from the smallest possible set of words whose cumulative probability exceeds the probability P, in Top-K sampling K most likely next words are filtered and the probability mass is redistributed among only those K next words. In our example with cat, if k=3, then only \"playing\", \"sleeping\" and \"eating\" will be taken into account as possible next word.\n",
+    "\n",
+    "To optimize the generation process and use memory more efficiently, the `use_cache=True` option is enabled. Since the output side is auto-regressive, an output token hidden state remains the same once computed for every further generation step. Therefore, recomputing it every time you want to generate a new token seems wasteful. With the cache, the model saves the hidden state once it has been computed. The model only computes the one for the most recently generated output token at each time step, re-using the saved ones for hidden tokens. This reduces the generation complexity from O(n^3) to O(n^2) for a transformer model. More details about how it works can be found in this [article](https://scale.com/blog/pytorch-improvements#Text%20Translation). With this option, the model gets the previous step's hidden states (cached attention keys and values) as input and additionally provides hidden states for the current step as output. It means for all next iterations, it is enough to provide only a new token obtained from the previous step and cached key values to get the next token prediction. \n",
+    "\n",
+    "The generation cycle repeats until the end of the sequence token is reached or it also can be interrupted when maximum tokens will be generated. As already mentioned before, we can enable printing current generated tokens without waiting until when the whole generation is finished using Streaming API, it adds a new token to the output queue and then prints them when they are ready."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eb7af692-0a15-4493-86ef-a80cda21551c",
+   "metadata": {},
+   "source": [
+    "### Setup imports\n",
+    "[back to top ⬆️](#Table-of-contents:)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "19d23a99-3284-42db-b7dc-5805d219f70d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from threading import Thread\n",
+    "from time import perf_counter\n",
+    "from typing import List\n",
+    "import gradio as gr\n",
+    "from transformers import AutoTokenizer, TextIteratorStreamer\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b341a9e6-290d-4780-90da-1ee64cee436d",
+   "metadata": {},
+   "source": [
+    "### Prepare template for user prompt\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "For effective generation, model expects to have input in specific format. The code below prepare template for passing user instruction into model with providing additional context."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "e2638c5b-47ad-4213-80da-8cfc2659b3aa",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
+     ]
+    }
+   ],
+   "source": [
+    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
+    "tokenizer_kwargs = model_configuration.get(\"toeknizer_kwargs\", {})\n",
+    "\n",
+    "\n",
+    "def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:\n",
+    "    \"\"\"\n",
+    "    Gets the token ID for a given string that has been added to the tokenizer as a special token.\n",
+    "\n",
+    "    Args:\n",
+    "        tokenizer (PreTrainedTokenizer): the tokenizer\n",
+    "        key (str): the key to convert to a single token\n",
+    "\n",
+    "    Raises:\n",
+    "        RuntimeError: if more than one ID was generated\n",
+    "\n",
+    "    Returns:\n",
+    "        int: the token ID for the given key\n",
+    "    \"\"\"\n",
+    "    token_ids = tokenizer.encode(key)\n",
+    "    if len(token_ids) > 1:\n",
+    "        raise ValueError(f\"Expected only a single token for '{key}' but found {token_ids}\")\n",
+    "    return token_ids[0]\n",
+    "\n",
+    "\n",
+    "response_key = model_configuration.get(\"response_key\")\n",
+    "tokenizer_response_key = None\n",
+    "\n",
+    "if response_key is not None:\n",
+    "    tokenizer_response_key = next(\n",
+    "        (token for token in tokenizer.additional_special_tokens if token.startswith(response_key)),\n",
+    "        None,\n",
+    "    )\n",
+    "\n",
+    "end_key_token_id = None\n",
+    "if tokenizer_response_key:\n",
+    "    try:\n",
+    "        end_key = model_configuration.get(\"end_key\")\n",
+    "        if end_key:\n",
+    "            end_key_token_id = get_special_token_id(tokenizer, end_key)\n",
+    "        # Ensure generation stops once it generates \"### End\"\n",
+    "    except ValueError:\n",
+    "        pass\n",
+    "\n",
+    "prompt_template = model_configuration.get(\"prompt_template\", \"{instruction}\")\n",
+    "end_key_token_id = end_key_token_id or tokenizer.eos_token_id\n",
+    "pad_token_id = end_key_token_id or tokenizer.pad_token_id"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55dbc4ae-da28-4be8-b928-3dd68c197937",
+   "metadata": {},
+   "source": [
+    "### Main generation function\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "As it was discussed above, `run_generation` function is the entry point for starting generation. It gets provided input instruction as parameter and returns model response."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "27802e81-9d42-4d71-99ea-5f76db5237f1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def run_generation(\n",
+    "    user_text: str,\n",
+    "    top_p: float,\n",
+    "    temperature: float,\n",
+    "    top_k: int,\n",
+    "    max_new_tokens: int,\n",
+    "    perf_text: str,\n",
+    "):\n",
+    "    \"\"\"\n",
+    "    Text generation function\n",
+    "\n",
+    "    Parameters:\n",
+    "      user_text (str): User-provided instruction for a generation.\n",
+    "      top_p (float):  Nucleus sampling. If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.\n",
+    "      temperature (float): The value used to module the logits distribution.\n",
+    "      top_k (int): The number of highest probability vocabulary tokens to keep for top-k-filtering.\n",
+    "      max_new_tokens (int): Maximum length of generated sequence.\n",
+    "      perf_text (str): Content of text field for printing performance results.\n",
+    "    Returns:\n",
+    "      model_output (str) - model-generated text\n",
+    "      perf_text (str) - updated perf text filed content\n",
+    "    \"\"\"\n",
+    "\n",
+    "    # Prepare input prompt according to model expected template\n",
+    "    prompt_text = prompt_template.format(instruction=user_text)\n",
+    "\n",
+    "    # Tokenize the user text.\n",
+    "    model_inputs = tokenizer(prompt_text, return_tensors=\"pt\", **tokenizer_kwargs)\n",
+    "\n",
+    "    # Start generation on a separate thread, so that we don't block the UI. The text is pulled from the streamer\n",
+    "    # in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.\n",
+    "    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)\n",
+    "    generate_kwargs = dict(\n",
+    "        model_inputs,\n",
+    "        streamer=streamer,\n",
+    "        max_new_tokens=max_new_tokens,\n",
+    "        do_sample=True,\n",
+    "        top_p=top_p,\n",
+    "        temperature=float(temperature),\n",
+    "        top_k=top_k,\n",
+    "        eos_token_id=end_key_token_id,\n",
+    "        pad_token_id=pad_token_id,\n",
+    "    )\n",
+    "    t = Thread(target=ov_model.generate, kwargs=generate_kwargs)\n",
+    "    t.start()\n",
+    "\n",
+    "    # Pull the generated text from the streamer, and update the model output.\n",
+    "    model_output = \"\"\n",
+    "    per_token_time = []\n",
+    "    num_tokens = 0\n",
+    "    start = perf_counter()\n",
+    "    for new_text in streamer:\n",
+    "        current_time = perf_counter() - start\n",
+    "        model_output += new_text\n",
+    "        perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)\n",
+    "        yield model_output, perf_text\n",
+    "        start = perf_counter()\n",
+    "    return model_output, perf_text"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6d7b7182-1d9b-485b-81f2-1a9fab6d5cd4",
+   "metadata": {},
+   "source": [
+    "### Helpers for application\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "For making interactive user interface we will use Gradio library. The code bellow provides useful functions used for communication with UI elements."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "c9b2a2b7-5a91-470f-98d7-7355664bc1be",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def estimate_latency(\n",
+    "    current_time: float,\n",
+    "    current_perf_text: str,\n",
+    "    new_gen_text: str,\n",
+    "    per_token_time: List[float],\n",
+    "    num_tokens: int,\n",
+    "):\n",
+    "    \"\"\"\n",
+    "    Helper function for performance estimation\n",
+    "\n",
+    "    Parameters:\n",
+    "      current_time (float): This step time in seconds.\n",
+    "      current_perf_text (str): Current content of performance UI field.\n",
+    "      new_gen_text (str): New generated text.\n",
+    "      per_token_time (List[float]): history of performance from previous steps.\n",
+    "      num_tokens (int): Total number of generated tokens.\n",
+    "\n",
+    "    Returns:\n",
+    "      update for performance text field\n",
+    "      update for a total number of tokens\n",
+    "    \"\"\"\n",
+    "    num_current_toks = len(tokenizer.encode(new_gen_text))\n",
+    "    num_tokens += num_current_toks\n",
+    "    per_token_time.append(num_current_toks / current_time)\n",
+    "    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:\n",
+    "        current_bucket = per_token_time[:-10]\n",
+    "        return (\n",
+    "            f\"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}\",\n",
+    "            num_tokens,\n",
+    "        )\n",
+    "    return current_perf_text, num_tokens\n",
+    "\n",
+    "\n",
+    "def reset_textbox(instruction: str, response: str, perf: str):\n",
+    "    \"\"\"\n",
+    "    Helper function for resetting content of all text fields\n",
+    "\n",
+    "    Parameters:\n",
+    "      instruction (str): Content of user instruction field.\n",
+    "      response (str): Content of model response field.\n",
+    "      perf (str): Content of performance info filed\n",
+    "\n",
+    "    Returns:\n",
+    "      empty string for each placeholder\n",
+    "    \"\"\"\n",
+    "    return \"\", \"\", \"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "31ebb167-0e55-4271-aedd-13814c2356d2",
+   "metadata": {},
+   "source": [
+    "## Run instruction-following pipeline\n",
+    "[back to top ⬆️](#Table-of-contents:)\n",
+    "\n",
+    "Now, we are ready to explore model capabilities. This demo provides a simple interface that allows communication with a model using text instruction. Type your instruction into the `User instruction` field or select one from predefined examples and click on the `Submit` button to start generation. Additionally, you can modify advanced generation parameters:\n",
+    "\n",
+    "* `Device` - allows switching inference device. Please note, every time when new device is selected, model will be recompiled and this takes some time.\n",
+    "* `Max New Tokens` - maximum size of generated text.\n",
+    "* `Top-p (nucleus sampling)` -  if set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.\n",
+    "* `Top-k` - the number of highest probability vocabulary tokens to keep for top-k-filtering.\n",
+    "* `Temperature` - the value used to module the logits distribution."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "9f222d02-847a-490f-8d66-02608a53259b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running on local URL:  http://127.0.0.1:7861\n",
+      "\n",
+      "To create a public link, set `share=True` in `launch()`.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div><iframe src=\"http://127.0.0.1:7861/\" width=\"100%\" height=\"800\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "examples = [\n",
+    "    \"Give me a recipe for pizza with pineapple\",\n",
+    "    \"Write me a tweet about the new OpenVINO release\",\n",
+    "    \"Explain the difference between CPU and GPU\",\n",
+    "    \"Give five ideas for a great weekend with family\",\n",
+    "    \"Do Androids dream of Electric sheep?\",\n",
+    "    \"Who is Dolly?\",\n",
+    "    \"Please give me advice on how to write resume?\",\n",
+    "    \"Name 3 advantages to being a cat\",\n",
+    "    \"Write instructions on how to become a good AI engineer\",\n",
+    "    \"Write a love letter to my best friend\",\n",
+    "]\n",
+    "\n",
+    "\n",
+    "with gr.Blocks() as demo:\n",
+    "    gr.Markdown(\n",
+    "        \"# Question Answering with \" + model_id.value + \" and OpenVINO.\\n\"\n",
+    "        \"Provide instruction which describes a task below or select among predefined examples and model writes response that performs requested task.\"\n",
+    "    )\n",
+    "\n",
+    "    with gr.Row():\n",
+    "        with gr.Column(scale=4):\n",
+    "            user_text = gr.Textbox(\n",
+    "                placeholder=\"Write an email about an alpaca that likes flan\",\n",
+    "                label=\"User instruction\",\n",
+    "            )\n",
+    "            model_output = gr.Textbox(label=\"Model response\", interactive=False)\n",
+    "            performance = gr.Textbox(label=\"Performance\", lines=1, interactive=False)\n",
+    "            with gr.Column(scale=1):\n",
+    "                button_clear = gr.Button(value=\"Clear\")\n",
+    "                button_submit = gr.Button(value=\"Submit\")\n",
+    "            gr.Examples(examples, user_text)\n",
+    "        with gr.Column(scale=1):\n",
+    "            max_new_tokens = gr.Slider(\n",
+    "                minimum=1,\n",
+    "                maximum=1000,\n",
+    "                value=256,\n",
+    "                step=1,\n",
+    "                interactive=True,\n",
+    "                label=\"Max New Tokens\",\n",
+    "            )\n",
+    "            top_p = gr.Slider(\n",
+    "                minimum=0.05,\n",
+    "                maximum=1.0,\n",
+    "                value=0.92,\n",
+    "                step=0.05,\n",
+    "                interactive=True,\n",
+    "                label=\"Top-p (nucleus sampling)\",\n",
+    "            )\n",
+    "            top_k = gr.Slider(\n",
+    "                minimum=0,\n",
+    "                maximum=50,\n",
+    "                value=0,\n",
+    "                step=1,\n",
+    "                interactive=True,\n",
+    "                label=\"Top-k\",\n",
+    "            )\n",
+    "            temperature = gr.Slider(\n",
+    "                minimum=0.1,\n",
+    "                maximum=5.0,\n",
+    "                value=0.8,\n",
+    "                step=0.1,\n",
+    "                interactive=True,\n",
+    "                label=\"Temperature\",\n",
+    "            )\n",
+    "\n",
+    "    user_text.submit(\n",
+    "        run_generation,\n",
+    "        [user_text, top_p, temperature, top_k, max_new_tokens, performance],\n",
+    "        [model_output, performance],\n",
+    "    )\n",
+    "    button_submit.click(\n",
+    "        run_generation,\n",
+    "        [user_text, top_p, temperature, top_k, max_new_tokens, performance],\n",
+    "        [model_output, performance],\n",
+    "    )\n",
+    "    button_clear.click(\n",
+    "        reset_textbox,\n",
+    "        [user_text, model_output, performance],\n",
+    "        [user_text, model_output, performance],\n",
+    "    )\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    demo.queue()\n",
+    "    try:\n",
+    "        demo.launch(height=800)\n",
+    "    except Exception:\n",
+    "        demo.launch(share=True, height=800)\n",
+    "\n",
+    "# If you are launching remotely, specify server_name and server_port\n",
+    "# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
+    "# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "440417f8-d2e2-4d8d-8311-882d133bb572",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dd9f90e4-0ec6-4164-8a31-044b1079d3e7",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7297ddd6-2c3d-4540-aa9b-f1c48c274a86",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openvino_env",
+   "language": "python",
+   "name": "openvino_env"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  },
+  "openvino_notebooks": {
+   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/daafd702-5a42-4f54-ae72-2e4480d73501",
+   "tags": {
+    "categories": [
+     "Model Demos",
+     "AI Trends"
+    ],
+    "libraries": [],
+    "other": [
+     "LLM"
+    ],
+    "tasks": [
+     "Text Generation"
+    ]
+   }
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {},
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebook_utils.py

@@ -0,0 +1,660 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+# In[ ]:
+
+
+import os
+import threading
+import time
+import urllib.parse
+from os import PathLike
+from pathlib import Path
+from typing import List, NamedTuple, Optional, Tuple
+
+import numpy as np
+from openvino.runtime import Core, Type, get_version
+from IPython.display import HTML, Image, display
+
+import openvino as ov
+from openvino.runtime.passes import Manager, MatcherPass, WrapType, Matcher
+from openvino.runtime import opset10 as ops
+
+
+# ## Files
+#
+# Load an image, download a file, download an IR model, and create a progress bar to show download progress.
+
+# In[ ]:
+
+
+def load_image(path: str) -> np.ndarray:
+    """
+    Loads an image from `path` and returns it as BGR numpy array. `path`
+    should point to an image file, either a local filename or a url. The image is
+    not stored to the filesystem. Use the `download_file` function to download and
+    store an image.
+
+    :param path: Local path name or URL to image.
+    :return: image as BGR numpy array
+    """
+    import cv2
+    import requests
+
+    if path.startswith("http"):
+        # Set User-Agent to Mozilla because some websites block
+        # requests with User-Agent Python
+        response = requests.get(path, headers={"User-Agent": "Mozilla/5.0"})
+        array = np.asarray(bytearray(response.content), dtype="uint8")
+        image = cv2.imdecode(array, -1)  # Loads the image as BGR
+    else:
+        image = cv2.imread(path)
+    return image
+
+
+def download_file(
+    url: PathLike,
+    filename: PathLike = None,
+    directory: PathLike = None,
+    show_progress: bool = True,
+    silent: bool = False,
+    timeout: int = 10,
+) -> PathLike:
+    """
+    Download a file from a url and save it to the local filesystem. The file is saved to the
+    current directory by default, or to `directory` if specified. If a filename is not given,
+    the filename of the URL will be used.
+
+    :param url: URL that points to the file to download
+    :param filename: Name of the local file to save. Should point to the name of the file only,
+                     not the full path. If None the filename from the url will be used
+    :param directory: Directory to save the file to. Will be created if it doesn't exist
+                      If None the file will be saved to the current working directory
+    :param show_progress: If True, show an TQDM ProgressBar
+    :param silent: If True, do not print a message if the file already exists
+    :param timeout: Number of seconds before cancelling the connection attempt
+    :return: path to downloaded file
+    """
+    from tqdm.notebook import tqdm_notebook
+    import requests
+
+    filename = filename or Path(urllib.parse.urlparse(url).path).name
+    chunk_size = 16384  # make chunks bigger so that not too many updates are triggered for Jupyter front-end
+
+    filename = Path(filename)
+    if len(filename.parts) > 1:
+        raise ValueError(
+            "`filename` should refer to the name of the file, excluding the directory. "
+            "Use the `directory` parameter to specify a target directory for the downloaded file."
+        )
+
+    # create the directory if it does not exist, and add the directory to the filename
+    if directory is not None:
+        directory = Path(directory)
+        directory.mkdir(parents=True, exist_ok=True)
+        filename = directory / Path(filename)
+
+    try:
+        response = requests.get(url=url, headers={"User-agent": "Mozilla/5.0"}, stream=True)
+        response.raise_for_status()
+    except (
+        requests.exceptions.HTTPError
+    ) as error:  # For error associated with not-200 codes. Will output something like: "404 Client Error: Not Found for url: {url}"
+        raise Exception(error) from None
+    except requests.exceptions.Timeout:
+        raise Exception(
+            "Connection timed out. If you access the internet through a proxy server, please "
+            "make sure the proxy is set in the shell from where you launched Jupyter."
+        ) from None
+    except requests.exceptions.RequestException as error:
+        raise Exception(f"File downloading failed with error: {error}") from None
+
+    # download the file if it does not exist, or if it exists with an incorrect file size
+    filesize = int(response.headers.get("Content-length", 0))
+    if not filename.exists() or (os.stat(filename).st_size != filesize):
+        with tqdm_notebook(
+            total=filesize,
+            unit="B",
+            unit_scale=True,
+            unit_divisor=1024,
+            desc=str(filename),
+            disable=not show_progress,
+        ) as progress_bar:
+            with open(filename, "wb") as file_object:
+                for chunk in response.iter_content(chunk_size):
+                    file_object.write(chunk)
+                    progress_bar.update(len(chunk))
+                    progress_bar.refresh()
+    else:
+        if not silent:
+            print(f"'{filename}' already exists.")
+
+    response.close()
+
+    return filename.resolve()
+
+
+def download_ir_model(model_xml_url: str, destination_folder: PathLike = None) -> PathLike:
+    """
+    Download IR model from `model_xml_url`. Downloads model xml and bin file; the weights file is
+    assumed to exist at the same location and name as model_xml_url with a ".bin" extension.
+
+    :param model_xml_url: URL to model xml file to download
+    :param destination_folder: Directory where downloaded model xml and bin are saved. If None, model
+                               files are saved to the current directory
+    :return: path to downloaded xml model file
+    """
+    model_bin_url = model_xml_url[:-4] + ".bin"
+    model_xml_path = download_file(model_xml_url, directory=destination_folder, show_progress=False)
+    download_file(model_bin_url, directory=destination_folder)
+    return model_xml_path
+
+
+# ## Images
+
+# ### Convert Pixel Data
+#
+# Normalize image pixel values between 0 and 1, and convert images to RGB and BGR.
+
+# In[ ]:
+
+
+def normalize_minmax(data):
+    """
+    Normalizes the values in `data` between 0 and 1
+    """
+    if data.max() == data.min():
+        raise ValueError("Normalization is not possible because all elements of" f"`data` have the same value: {data.max()}.")
+    return (data - data.min()) / (data.max() - data.min())
+
+
+def to_rgb(image_data: np.ndarray) -> np.ndarray:
+    """
+    Convert image_data from BGR to RGB
+    """
+    import cv2
+
+    return cv2.cvtColor(image_data, cv2.COLOR_BGR2RGB)
+
+
+def to_bgr(image_data: np.ndarray) -> np.ndarray:
+    """
+    Convert image_data from RGB to BGR
+    """
+    import cv2
+
+    return cv2.cvtColor(image_data, cv2.COLOR_RGB2BGR)
+
+
+# ## Videos
+
+# ### Video Player
+#
+# Custom video player to fulfill FPS requirements. You can set target FPS and output size, flip the video horizontally or skip first N frames.
+
+# In[ ]:
+
+
+class VideoPlayer:
+    """
+    Custom video player to fulfill FPS requirements. You can set target FPS and output size,
+    flip the video horizontally or skip first N frames.
+
+    :param source: Video source. It could be either camera device or video file.
+    :param size: Output frame size.
+    :param flip: Flip source horizontally.
+    :param fps: Target FPS.
+    :param skip_first_frames: Skip first N frames.
+    """
+
+    def __init__(self, source, size=None, flip=False, fps=None, skip_first_frames=0):
+        import cv2
+
+        self.cv2 = cv2  # This is done to access the package in class methods
+        self.__cap = cv2.VideoCapture(source)
+        if not self.__cap.isOpened():
+            raise RuntimeError(f"Cannot open {'camera' if isinstance(source, int) else ''} {source}")
+        # skip first N frames
+        self.__cap.set(cv2.CAP_PROP_POS_FRAMES, skip_first_frames)
+        # fps of input file
+        self.__input_fps = self.__cap.get(cv2.CAP_PROP_FPS)
+        if self.__input_fps <= 0:
+            self.__input_fps = 60
+        # target fps given by user
+        self.__output_fps = fps if fps is not None else self.__input_fps
+        self.__flip = flip
+        self.__size = None
+        self.__interpolation = None
+        if size is not None:
+            self.__size = size
+            # AREA better for shrinking, LINEAR better for enlarging
+            self.__interpolation = cv2.INTER_AREA if size[0] < self.__cap.get(cv2.CAP_PROP_FRAME_WIDTH) else cv2.INTER_LINEAR
+        # first frame
+        _, self.__frame = self.__cap.read()
+        self.__lock = threading.Lock()
+        self.__thread = None
+        self.__stop = False
+
+    """
+    Start playing.
+    """
+
+    def start(self):
+        self.__stop = False
+        self.__thread = threading.Thread(target=self.__run, daemon=True)
+        self.__thread.start()
+
+    """
+    Stop playing and release resources.
+    """
+
+    def stop(self):
+        self.__stop = True
+        if self.__thread is not None:
+            self.__thread.join()
+        self.__cap.release()
+
+    def __run(self):
+        prev_time = 0
+        while not self.__stop:
+            t1 = time.time()
+            ret, frame = self.__cap.read()
+            if not ret:
+                break
+
+            # fulfill target fps
+            if 1 / self.__output_fps < time.time() - prev_time:
+                prev_time = time.time()
+                # replace by current frame
+                with self.__lock:
+                    self.__frame = frame
+
+            t2 = time.time()
+            # time to wait [s] to fulfill input fps
+            wait_time = 1 / self.__input_fps - (t2 - t1)
+            # wait until
+            time.sleep(max(0, wait_time))
+
+        self.__frame = None
+
+    """
+    Get current frame.
+    """
+
+    def next(self):
+        import cv2
+
+        with self.__lock:
+            if self.__frame is None:
+                return None
+            # need to copy frame, because can be cached and reused if fps is low
+            frame = self.__frame.copy()
+        if self.__size is not None:
+            frame = self.cv2.resize(frame, self.__size, interpolation=self.__interpolation)
+        if self.__flip:
+            frame = self.cv2.flip(frame, 1)
+        return frame
+
+
+# ## Visualization
+
+# ### Segmentation
+#
+# Define a SegmentationMap NamedTuple that keeps the labels and colormap for a segmentation project/dataset. Create CityScapesSegmentation and BinarySegmentation SegmentationMaps. Create a function to convert a segmentation map to an RGB image with a colormap, and to show the segmentation result as an overlay over the original image.
+
+# In[ ]:
+
+
+class Label(NamedTuple):
+    index: int
+    color: Tuple
+    name: Optional[str] = None
+
+
+# In[ ]:
+
+
+class SegmentationMap(NamedTuple):
+    labels: List
+
+    def get_colormap(self):
+        return np.array([label.color for label in self.labels])
+
+    def get_labels(self):
+        labelnames = [label.name for label in self.labels]
+        if any(labelnames):
+            return labelnames
+        else:
+            return None
+
+
+# In[ ]:
+
+
+cityscape_labels = [
+    Label(index=0, color=(128, 64, 128), name="road"),
+    Label(index=1, color=(244, 35, 232), name="sidewalk"),
+    Label(index=2, color=(70, 70, 70), name="building"),
+    Label(index=3, color=(102, 102, 156), name="wall"),
+    Label(index=4, color=(190, 153, 153), name="fence"),
+    Label(index=5, color=(153, 153, 153), name="pole"),
+    Label(index=6, color=(250, 170, 30), name="traffic light"),
+    Label(index=7, color=(220, 220, 0), name="traffic sign"),
+    Label(index=8, color=(107, 142, 35), name="vegetation"),
+    Label(index=9, color=(152, 251, 152), name="terrain"),
+    Label(index=10, color=(70, 130, 180), name="sky"),
+    Label(index=11, color=(220, 20, 60), name="person"),
+    Label(index=12, color=(255, 0, 0), name="rider"),
+    Label(index=13, color=(0, 0, 142), name="car"),
+    Label(index=14, color=(0, 0, 70), name="truck"),
+    Label(index=15, color=(0, 60, 100), name="bus"),
+    Label(index=16, color=(0, 80, 100), name="train"),
+    Label(index=17, color=(0, 0, 230), name="motorcycle"),
+    Label(index=18, color=(119, 11, 32), name="bicycle"),
+    Label(index=19, color=(255, 255, 255), name="background"),
+]
+
+CityScapesSegmentation = SegmentationMap(cityscape_labels)
+
+binary_labels = [
+    Label(index=0, color=(255, 255, 255), name="background"),
+    Label(index=1, color=(0, 0, 0), name="foreground"),
+]
+
+BinarySegmentation = SegmentationMap(binary_labels)
+
+
+# In[ ]:
+
+
+def segmentation_map_to_image(result: np.ndarray, colormap: np.ndarray, remove_holes: bool = False) -> np.ndarray:
+    """
+    Convert network result of floating point numbers to an RGB image with
+    integer values from 0-255 by applying a colormap.
+
+    :param result: A single network result after converting to pixel values in H,W or 1,H,W shape.
+    :param colormap: A numpy array of shape (num_classes, 3) with an RGB value per class.
+    :param remove_holes: If True, remove holes in the segmentation result.
+    :return: An RGB image where each pixel is an int8 value according to colormap.
+    """
+    import cv2
+
+    if len(result.shape) != 2 and result.shape[0] != 1:
+        raise ValueError(f"Expected result with shape (H,W) or (1,H,W), got result with shape {result.shape}")
+
+    if len(np.unique(result)) > colormap.shape[0]:
+        raise ValueError(
+            f"Expected max {colormap[0]} classes in result, got {len(np.unique(result))} "
+            "different output values. Please make sure to convert the network output to "
+            "pixel values before calling this function."
+        )
+    elif result.shape[0] == 1:
+        result = result.squeeze(0)
+
+    result = result.astype(np.uint8)
+
+    contour_mode = cv2.RETR_EXTERNAL if remove_holes else cv2.RETR_TREE
+    mask = np.zeros((result.shape[0], result.shape[1], 3), dtype=np.uint8)
+    for label_index, color in enumerate(colormap):
+        label_index_map = result == label_index
+        label_index_map = label_index_map.astype(np.uint8) * 255
+        contours, hierarchies = cv2.findContours(label_index_map, contour_mode, cv2.CHAIN_APPROX_SIMPLE)
+        cv2.drawContours(
+            mask,
+            contours,
+            contourIdx=-1,
+            color=color.tolist(),
+            thickness=cv2.FILLED,
+        )
+
+    return mask
+
+
+def segmentation_map_to_overlay(image, result, alpha, colormap, remove_holes=False) -> np.ndarray:
+    """
+    Returns a new image where a segmentation mask (created with colormap) is overlayed on
+    the source image.
+
+    :param image: Source image.
+    :param result: A single network result after converting to pixel values in H,W or 1,H,W shape.
+    :param alpha: Alpha transparency value for the overlay image.
+    :param colormap: A numpy array of shape (num_classes, 3) with an RGB value per class.
+    :param remove_holes: If True, remove holes in the segmentation result.
+    :return: An RGP image with segmentation mask overlayed on the source image.
+    """
+    import cv2
+
+    if len(image.shape) == 2:
+        image = np.repeat(np.expand_dims(image, -1), 3, 2)
+    mask = segmentation_map_to_image(result, colormap, remove_holes)
+    image_height, image_width = image.shape[:2]
+    mask = cv2.resize(src=mask, dsize=(image_width, image_height))
+    return cv2.addWeighted(mask, alpha, image, 1 - alpha, 0)
+
+
+# ### Network Results
+#
+# Show network result image, optionally together with the source image and a legend with labels.
+
+# In[ ]:
+
+
+def viz_result_image(
+    result_image: np.ndarray,
+    source_image: np.ndarray = None,
+    source_title: str = None,
+    result_title: str = None,
+    labels: List[Label] = None,
+    resize: bool = False,
+    bgr_to_rgb: bool = False,
+    hide_axes: bool = False,
+):
+    """
+    Show result image, optionally together with source images, and a legend with labels.
+
+    :param result_image: Numpy array of RGB result image.
+    :param source_image: Numpy array of source image. If provided this image will be shown
+                         next to the result image. source_image is expected to be in RGB format.
+                         Set bgr_to_rgb to True if source_image is in BGR format.
+    :param source_title: Title to display for the source image.
+    :param result_title: Title to display for the result image.
+    :param labels: List of labels. If provided, a legend will be shown with the given labels.
+    :param resize: If true, resize the result image to the same shape as the source image.
+    :param bgr_to_rgb: If true, convert the source image from BGR to RGB. Use this option if
+                       source_image is a BGR image.
+    :param hide_axes: If true, do not show matplotlib axes.
+    :return: Matplotlib figure with result image
+    """
+    import cv2
+    import matplotlib.pyplot as plt
+    from matplotlib.lines import Line2D
+
+    if bgr_to_rgb:
+        source_image = to_rgb(source_image)
+    if resize:
+        result_image = cv2.resize(result_image, (source_image.shape[1], source_image.shape[0]))
+
+    num_images = 1 if source_image is None else 2
+
+    fig, ax = plt.subplots(1, num_images, figsize=(16, 8), squeeze=False)
+    if source_image is not None:
+        ax[0, 0].imshow(source_image)
+        ax[0, 0].set_title(source_title)
+
+    ax[0, num_images - 1].imshow(result_image)
+    ax[0, num_images - 1].set_title(result_title)
+
+    if hide_axes:
+        for a in ax.ravel():
+            a.axis("off")
+    if labels:
+        colors = labels.get_colormap()
+        lines = [
+            Line2D(
+                [0],
+                [0],
+                color=[item / 255 for item in c.tolist()],
+                linewidth=3,
+                linestyle="-",
+            )
+            for c in colors
+        ]
+        plt.legend(
+            lines,
+            labels.get_labels(),
+            bbox_to_anchor=(1, 1),
+            loc="upper left",
+            prop={"size": 12},
+        )
+    plt.close(fig)
+    return fig
+
+
+# ### Live Inference
+
+# In[ ]:
+
+
+def show_array(frame: np.ndarray, display_handle=None):
+    """
+    Display array `frame`. Replace information at `display_handle` with `frame`
+    encoded as jpeg image. `frame` is expected to have data in BGR order.
+
+    Create a display_handle with: `display_handle = display(display_id=True)`
+    """
+    import cv2
+
+    _, frame = cv2.imencode(ext=".jpeg", img=frame)
+    if display_handle is None:
+        display_handle = display(Image(data=frame.tobytes()), display_id=True)
+    else:
+        display_handle.update(Image(data=frame.tobytes()))
+    return display_handle
+
+
+# ## Checks and Alerts
+#
+# Create an alert class to show stylized info/error/warning messages and a `check_device` function that checks whether a given device is available.
+
+# In[ ]:
+
+
+class NotebookAlert(Exception):
+    def __init__(self, message: str, alert_class: str):
+        """
+        Show an alert box with the given message.
+
+        :param message: The message to display.
+        :param alert_class: The class for styling the message. Options: info, warning, success, danger.
+        """
+        self.message = message
+        self.alert_class = alert_class
+        self.show_message()
+
+    def show_message(self):
+        display(HTML(f"""<div class="alert alert-{self.alert_class}">{self.message}"""))
+
+
+class DeviceNotFoundAlert(NotebookAlert):
+    def __init__(self, device: str):
+        """
+        Show a warning message about an unavailable device. This class does not check whether or
+        not the device is available, use the `check_device` function to check this. `check_device`
+        also shows the warning if the device is not found.
+
+        :param device: The unavailable device.
+        :return: A formatted alert box with the message that `device` is not available, and a list
+                 of devices that are available.
+        """
+        ie = Core()
+        supported_devices = ie.available_devices
+        self.message = f"Running this cell requires a {device} device, " "which is not available on this system. "
+        self.alert_class = "warning"
+        if len(supported_devices) == 1:
+            self.message += f"The following device is available: {ie.available_devices[0]}"
+        else:
+            self.message += "The following devices are available: " f"{', '.join(ie.available_devices)}"
+        super().__init__(self.message, self.alert_class)
+
+
+def check_device(device: str) -> bool:
+    """
+    Check if the specified device is available on the system.
+
+    :param device: Device to check. e.g. CPU, GPU
+    :return: True if the device is available, False if not. If the device is not available,
+             a DeviceNotFoundAlert will be shown.
+    """
+    ie = Core()
+    if device not in ie.available_devices:
+        DeviceNotFoundAlert(device)
+        return False
+    else:
+        return True
+
+
+def check_openvino_version(version: str) -> bool:
+    """
+    Check if the specified OpenVINO version is installed.
+
+    :param version: the OpenVINO version to check. Example: 2021.4
+    :return: True if the version is installed, False if not. If the version is not installed,
+             an alert message will be shown.
+    """
+    installed_version = get_version()
+    if version not in installed_version:
+        NotebookAlert(
+            f"This notebook requires OpenVINO {version}. "
+            f"The version on your system is: <i>{installed_version}</i>.<br>"
+            "Please run <span style='font-family:monospace'>pip install --upgrade -r requirements.txt</span> "
+            "in the openvino_env environment to install this version. "
+            "See the <a href='https://github.com/openvinotoolkit/openvino_notebooks'>"
+            "OpenVINO Notebooks README</a> for detailed instructions",
+            alert_class="danger",
+        )
+        return False
+    else:
+        return True
+
+
+packed_layername_tensor_dict_list = [{"name": "aten::mul/Multiply"}]
+
+
+class ReplaceTensor(MatcherPass):
+    def __init__(self, packed_layername_tensor_dict_list):
+        MatcherPass.__init__(self)
+        self.model_changed = False
+
+        param = WrapType("opset10.Multiply")
+
+        def callback(matcher: Matcher) -> bool:
+            root = matcher.get_match_root()
+            if root is None:
+                return False
+            for y in packed_layername_tensor_dict_list:
+                root_name = root.get_friendly_name()
+                if root_name.find(y["name"]) != -1:
+                    max_fp16 = np.array([[[[-np.finfo(np.float16).max]]]]).astype(np.float32)
+                    new_tenser = ops.constant(max_fp16, Type.f32, name="Constant_4431")
+                    root.set_arguments([root.input_value(0).node, new_tenser])
+                    packed_layername_tensor_dict_list.remove(y)
+
+            return True
+
+        self.register_matcher(Matcher(param, "ReplaceTensor"), callback)
+
+
+def optimize_bge_embedding(model_path, output_model_path):
+    """
+    optimize_bge_embedding used to optimize BGE model for NPU device
+
+    Arguments:
+        model_path {str} -- original BGE IR model path
+        output_model_path {str} -- Converted BGE IR model path
+    """
+    core = Core()
+    ov_model = core.read_model(model_path)
+    manager = Manager()
+    manager.register_pass(ReplaceTensor(packed_layername_tensor_dict_list))
+    manager.run_passes(ov_model)
+    ov.save_model(ov_model, output_model_path, compress_to_fp16=False)

phi-2/INT4_compressed_weights/config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "susnato/phi-2",
+  "architectures": [
+    "PhiForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "embd_pdrop": 0.0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu_new",
+  "hidden_size": 2560,
+  "initializer_range": 0.02,
+  "intermediate_size": 10240,
+  "is_decoder": true,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "phi",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "partial_rotary_factor": 0.4,
+  "pretraining_tp": 1,
+  "qk_layernorm": false,
+  "resid_pdrop": 0.1,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "vocab_size": 51200
+}

phi-2/INT4_compressed_weights/generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.41.2"
+}

phi-2/INT4_compressed_weights/openvino_config.json

@@ -0,0 +1,22 @@
+{
+  "compression": null,
+  "dtype": "int4",
+  "input_info": null,
+  "optimum_version": "1.21.0",
+  "quantization_config": {
+    "all_layers": null,
+    "bits": 4,
+    "dataset": null,
+    "group_size": 128,
+    "ignored_scope": null,
+    "num_samples": null,
+    "quant_method": "default",
+    "ratio": 0.8,
+    "scale_estimation": null,
+    "sensitivity_metric": null,
+    "sym": false,
+    "tokenizer": null
+  },
+  "save_onnx_model": false,
+  "transformers_version": "4.41.2"
+}

phi-2/INT4_compressed_weights/openvino_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:203aaad089bbdfd1590bcc913733ac8f255a399ca308b7d43b268eaf7ca922dc
+size 1823610588

phi-2/INT8_compressed_weights/config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "susnato/phi-2",
+  "architectures": [
+    "PhiForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "embd_pdrop": 0.0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu_new",
+  "hidden_size": 2560,
+  "initializer_range": 0.02,
+  "intermediate_size": 10240,
+  "is_decoder": true,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "phi",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "partial_rotary_factor": 0.4,
+  "pretraining_tp": 1,
+  "qk_layernorm": false,
+  "resid_pdrop": 0.1,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "vocab_size": 51200
+}

phi-2/INT8_compressed_weights/generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.41.2"
+}

phi-2/INT8_compressed_weights/openvino_config.json

@@ -0,0 +1,22 @@
+{
+  "compression": null,
+  "dtype": "int8",
+  "input_info": null,
+  "optimum_version": "1.21.0",
+  "quantization_config": {
+    "all_layers": null,
+    "bits": 8,
+    "dataset": null,
+    "group_size": -1,
+    "ignored_scope": null,
+    "num_samples": null,
+    "quant_method": "default",
+    "ratio": 1.0,
+    "scale_estimation": null,
+    "sensitivity_metric": null,
+    "sym": false,
+    "tokenizer": null
+  },
+  "save_onnx_model": false,
+  "transformers_version": "4.41.2"
+}

phi-2/INT8_compressed_weights/openvino_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81274a4f75f1197be9f86cf61f84e9482106ff269fad755ab559291549cd1b53
+size 2785599684

templates/dashboard.html

@@ -0,0 +1,64 @@
+<!doctype html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Dashboard</title>
+    <style>
+        body {
+            font-family: Arial, sans-serif;
+            background-color: #f4f4f4;
+            margin: 0;
+            padding: 0;
+        }
+        .container {
+            max-width: 1200px;
+            margin: 0 auto;
+            padding: 20px;
+        }
+        h1 {
+            color: #333;
+        }
+        .gradio-container {
+            margin-top: 20px;
+        }
+        iframe {
+            border: none;
+            width: 100%;
+            height: 800px;
+        }
+        .logout {
+            margin-top: 20px;
+            text-align: right;
+        }
+        .start-button {
+            margin-top: 20px;
+            background-color: #4CAF50;
+            color: white;
+            border: none;
+            padding: 10px 20px;
+            text-align: center;
+            text-decoration: none;
+            display: inline-block;
+            font-size: 16px;
+            cursor: pointer;
+            border-radius: 4px;
+        }
+    </style>
+    <script>
+        function openGradioInNewWindow() {
+            window.open("{{ url_for('gradio') }}");
+        }
+    </script>
+</head>
+<body>
+    <div class="container">
+        <h1>Welcome, {{ name }}!</h1>
+        <button class="start-button" onclick="openGradioInNewWindow()">Start the Chatbot</button>
+       
+        <div class="logout">
+            <a href="{{ url_for('logout') }}">Logout</a>
+        </div>
+    </div>
+</body>
+</html>

templates/login.html

@@ -0,0 +1,9 @@
+<!doctype html>
+<title>Login</title>
+<h1>Login</h1>
+<form method="post">
+    <input type="text" name="username" placeholder="Username" required>
+    <input type="password" name="password" placeholder="Password" required>
+    <button type="submit">Login</button>
+</form>
+<a href="{{ url_for('signup') }}">Don't have an account? Sign Up</a>

templates/signup.html

@@ -0,0 +1,9 @@
+<!doctype html>
+<title>Sign Up</title>
+<h1>Sign Up</h1>
+<form method="post">
+    <input type="text" name="username" placeholder="Username" required>
+    <input type="password" name="password" placeholder="Password" required>
+    <button type="submit">Sign Up</button>
+</form>
+<a href="{{ url_for('login') }}">Already have an account? Login</a>