{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "PIbogPXyM0wr" }, "source": [ "# Project IO Achievement - UI Design (Oral Exam)" ] }, { "cell_type": "markdown", "metadata": { "id": "x_Vp8SiKM4p1" }, "source": [ "## Problem Definition\n", "\n", "The v1 functionality for the Oral Exam module requires the following:\n", "\n", "1. Upload or generation of questions: either the user should upload a set of questions or we should allow the model to generate the questions. The user should pick or it should be inherent if there is no upload of questions. Note that we must also allow for context to be uploaded (vector store, vector store link, specific documents)\n", "2. The model should prompt the user with a question and pause.\n", "The user should respond by audio.\n", "3. This should continue on until some final point where the exam is over.\n", "\n", "Then:\n", "\n", "1. We should use Whisper to do the transcription, and\n", "2. Send the transcription, questions, and context for GPT4 for evaluation\n", "Return the evaluation.\n", "3. This will primarily be work on a user interface." ] }, { "cell_type": "markdown", "metadata": { "id": "o_60X8H3NEne" }, "source": [ "## Libraries\n", "\n", "This section will install and import some important libraries such as Langchain, openai, Gradio, and so on" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "pxcqXgg2aAN7" }, "outputs": [], "source": [ "# install libraries here\n", "# -q flag for \"quiet\" install\n", "%%capture\n", "!pip install -q langchain\n", "!pip install -q openai\n", "!pip install -q gradio\n", "# !pip install -q datasets\n", "!pip install -q torchaudio\n", "!pip install -q git+https://github.com/openai/whisper.git\n", "!pip install -q docx\n", "!pip install -q PyPDF2\n", "!pip install -q python-docx" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "pEjM1tLsMZBq" }, "outputs": [], "source": [ "# import libraries here\n", "from langchain.llms import OpenAI\n", "from langchain.prompts import PromptTemplate\n", "from langchain.document_loaders import TextLoader\n", "from langchain.indexes import VectorstoreIndexCreator\n", "from langchain import ConversationChain, LLMChain, PromptTemplate\n", "from langchain.chat_models import ChatOpenAI\n", "from langchain.memory import ConversationBufferWindowMemory\n", "from langchain.prompts import ChatPromptTemplate\n", "from langchain.text_splitter import CharacterTextSplitter\n", "from langchain.embeddings import OpenAIEmbeddings\n", "import openai\n", "import os\n", "from getpass import getpass\n", "# from IPython.display import display, Javascript, HTML\n", "# from google.colab.output import eval_js\n", "# from base64 import b64decode\n", "# import ipywidgets as widgets\n", "# from IPython.display import clear_output\n", "import time\n", "import requests\n", "# from datasets import load_dataset\n", "# from torchaudio.transforms import Resample\n", "import whisper\n", "import numpy as np\n", "import torch\n", "import librosa\n", "# from datasets import load_dataset\n", "#from jiwer import wer\n", "import pandas as pd\n", "import gradio as gr\n", "from docx import Document\n", "import PyPDF2\n", "from pydub import AudioSegment\n", "import tempfile" ] }, { "cell_type": "markdown", "metadata": { "id": "03KLZGI_a5W5" }, "source": [ "## API Keys\n", "\n", "Use these cells to load the API keys required for this notebook. The below code cell uses the `getpass` library." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5smcWj4DbFgy", "outputId": "6bc91507-cd3c-4808-8976-811d7fc7cb29" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "··········\n" ] } ], "source": [ "openai_api_key = getpass()\n", "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n", "openai.api_key = openai_api_key" ] }, { "cell_type": "markdown", "metadata": { "id": "pMo9x8u4AEV1" }, "source": [ "## Prompt Design" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UgnCZRMhADvo", "outputId": "462e62c7-a618-4549-e651-858514757235" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "ChatOpenAI(cache=None, verbose=False, callbacks=None, callback_manager=None, tags=None, metadata=None, client=, model_name='gpt-4', temperature=0.0, model_kwargs={}, openai_api_key='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', openai_api_base='', openai_organization='', openai_proxy='', request_timeout=None, max_retries=6, streaming=False, n=1, max_tokens=None, tiktoken_model_name=None)" ] }, "metadata": {}, "execution_count": 4 } ], "source": [ "chat = ChatOpenAI(temperature=0.0, model_name='gpt-4')\n", "chat" ] }, { "cell_type": "markdown", "source": [ "### Chatbot Prompts" ], "metadata": { "id": "2tTNiyU-ZcDU" } }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "r-VmK_7vHrmw", "outputId": "8c314a8f-dad5-47b9-ddba-f6009a73b80d" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['history', 'input', 'instruction', 'questions']" ] }, "metadata": {}, "execution_count": 15 } ], "source": [ "template_string3 = \"\"\"\n", "Please ask me the following questions in sequence, and after I provide the answer, \\\n", "please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. \\\n", "Just ask me the question, and please do not show any other text (no need for greetings for example) \\\n", "Here are the questions that you can will me: {questions}. \\\n", "Here are the chat history: {history}. \\\n", "{input}\n", "\n", "Once all questions are answered, thank the user and give overall feedback for the question answering part.\n", "\"\"\"\n", "prompt_template3 = ChatPromptTemplate.from_template(template_string3)\n", "prompt_template3.messages[0].prompt.input_variables" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 72 }, "id": "XaK7D5B4bMYv", "outputId": "55f495b5-d8d0-4a80-b5c4-128dba34eebe" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'\\nPlease ask me the following questions in sequence, and after I provide the answer, please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. Just ask me the question, and please do not show any other text (no need for greetings for example) Here are the questions that you can will me: {questions}. Here are the chat history: {history}. {input}\\n\\nOnce all questions are answered, thank the user and give overall feedback for the question answering part.\\n'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 16 } ], "source": [ "prompt_template3.messages[0].prompt.template" ] }, { "cell_type": "markdown", "metadata": { "id": "l4o8R5eUE1n8" }, "source": [ "### Functions" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "ABN0X9xQHeii" }, "outputs": [], "source": [ "def embed_key(openai_api_key):\n", " os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n", "\n", "def transcribe(audio_file_path):\n", " try:\n", " with open(audio_file_path, \"rb\") as audio_file:\n", " # Call OpenAI's Whisper model for transcription\n", " transcript = openai.Audio.transcribe(\"whisper-1\", audio_file)\n", " transcribed_text = transcript[\"text\"]\n", " return transcribed_text\n", " except:\n", " return \"Your answer will be transcribed here\"\n", "\n", "def process_file(files):\n", " for file in files:\n", " try:\n", " extension = file.name.split('.')[-1].lower()\n", " if extension == 'docx':\n", " doc = Document(file.name)\n", " full_text = []\n", " for paragraph in doc.paragraphs:\n", " full_text.append(paragraph.text)\n", " return '\\n'.join(full_text)\n", "\n", " elif extension == 'pdf':\n", " pdf_file = open(file.name, 'rb')\n", " reader = PyPDF2.PdfReader(pdf_file)\n", " num_pages = len(reader.pages)\n", " full_text = []\n", " for page in range(num_pages):\n", " page_obj = reader.pages[page]\n", " full_text.append(page_obj.extract_text())\n", " pdf_file.close()\n", " return '\\n'.join(full_text)\n", "\n", " elif extension == 'txt':\n", " with open(file.name, 'r') as txt_file:\n", " full_text = txt_file.read()\n", " return full_text\n", "\n", " else:\n", " return \"Unsupported file type\"\n", " except FileNotFoundError:\n", " return \"File not found\"\n", " except PermissionError:\n", " return \"Permission denied\"\n", "\n", "def generate_questions(text, prompt):\n", " test_input1 = question_template.format_messages(\n", " context = text,\n", " pre_prompt = prompt)\n", "\n", " response = chat(test_input1)\n", " return response.content\n", "\n", "\n", "def ai_evaluate(context, audio_transcript, QA, instructions):\n", " test_input1 = evaluate_template.format_messages(\n", " context = context,\n", " transcript = audio_transcript,\n", " QA = QA,\n", " instructions = instructions)\n", "\n", " response = chat(test_input1)\n", " return response.content\n", "\n", "def upload_file(files):\n", " file_paths = [file.name for file in files]\n", " return file_paths\n", "\n", "def use_these_questions(input):\n", " return input\n", "\n", "################################\n", "\n", "def add_text(history, text, prompt = template_string3):\n", " new_history = [(prompt, None)] + history + [(text, None)]\n", " return new_history, gr.update(value=\"\", interactive=False)\n", "\n", "# def add_file(history, file):\n", "# history = history + [((file.name,), None)]\n", "# return history\n", "\n", "\n", "def bot_initialize(input, instruction_feedback, questions_used, history):\n", "\n", " template_string3 = \"\"\"\n", " Please ask me the following questions in sequence, and after I provide the answer, \\\n", " please give me some feedback. Here is the instruction for feedback: {instruction}. If no instruction is provided, please provide feedback based on your judgement. \\\n", " Here are the questions that you can ask me: {questions}. \\\n", " Here are the chat history: {history}. \\\n", " {input} \\\n", "\n", " *** Remember, just ask me the question, give feedbacks, and ask the next questions. Do not forget to ask the next question after feedbacks. \\\n", " \"\"\"\n", " prompt_template3 = ChatPromptTemplate.from_template(template_string3)\n", "\n", " test_input1 = prompt_template3.format_messages(\n", " instruction = instruction_feedback,\n", " history = history,\n", " questions = questions_used,\n", " input = input)\n", "\n", " response = chat(test_input1)\n", " return response.content\n", "\n", "# def initialize(instruction_feedback, questions_used, chat_history, ready):\n", "# test_input1 = prompt_template3.format_messages(\n", "# instruction = instruction_feedback,\n", "# chat_history = chat_history,\n", "# questions = questions_used,\n", "# ready = ready)\n", "# response = chat(test_input1)\n", "# return response.content\n", "\n", "# def bot(history):\n", "# response = \"**That's cool!**\"\n", "# history[-1][1] = \"\"\n", "# for character in response:\n", "# history[-1][1] += character\n", "# time.sleep(0.05)\n", "# yield history\n", "\n", "def message_and_history(input, instruction_feedback, questions_used, history):\n", " history = history or []\n", " s = list(sum(history, ()))\n", " s.append(input)\n", " inp = ' '.join(s)\n", " output = bot_initialize(inp, instruction_feedback, questions_used, history)\n", " history.append((input, output))\n", " return history, history\n", "\n", "def prompt_select(selection, number, length):\n", " if selection == \"Random\":\n", " prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable).\"\n", " elif selection == \"Fill in the Blank\":\n", " prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement.\"\n", " elif selection == \"Short Answer\":\n", " prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted). The questions should be short answer. Expect the correct answers to be {length} sentences long.\"\n", " else:\n", " prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable).\"\n", " return prompt\n", "\n", "# def prompt_select(selection, number, length):\n", "# if selection == \"Random\":\n", "# prompt = f\"Please design a {number} question quiz based on the context provided and the inputted learning objectives (if applicable). The types of questions should be randomized (including multiple choice, short answer, true/false, short answer, etc.). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n", "# elif selection == \"Fill in the Blank\":\n", "# prompt = f\"Create a {number} question fill in the blank quiz refrencing the context provided. The quiz should reflect the learning objectives (if inputted). The 'blank' part of the question should appear as '________'. The answers should reflect what word(s) should go in the blank an accurate statement. An example is the follow: 'The author of the article is ______.' The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n", "# elif selection == \"Short Answer\":\n", "# prompt = f\"Please design a {number} question quiz about which reflects the learning objectives (if inputted). The questions should be short answer. Expect the correct answers to be {length} sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n", "# else:\n", "# prompt = f\"Please design a {number} question {selection.lower()} quiz based on the context provided and the inputted learning objectives (if applicable). Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide 1 question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\"\n", "# return prompt" ] }, { "cell_type": "markdown", "metadata": { "id": "8PzIpcfg4-X0" }, "source": [ "## Integrate Prompts from LO project" ] }, { "cell_type": "markdown", "metadata": { "id": "E5vWxcm25EAC" }, "source": [ "### Creating a Chain for Short Answer Generation" ] }, { "cell_type": "markdown", "metadata": { "id": "y95FExV-5IqI" }, "source": [ "In this example, the context would include the poem \"The Road Not Taken\" by Robert Frost" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "id": "j_qEXDWQ5RSW" }, "outputs": [], "source": [ "# This is what I used to test the function 'generate_questions_v2'\n", "template_string = \"\"\"\n", "You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences.\"\n", "\n", "The following text should be used as the basis for the instructions which follow: {context} \\\n", "\n", "The following is the guideline for generating the questiion: {pre_prompt} \\\n", "\n", "The output should be formatted as following:\n", "\n", "Question 1: ...\n", "Question 2: ...\n", "Question 3: ...\n", "...\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yWOM1XdC5UhQ", "outputId": "69d781f7-fb0c-4dde-9085-ddb9908b82af" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['context', 'pre_prompt']" ] }, "metadata": {}, "execution_count": 9 } ], "source": [ "question_template = ChatPromptTemplate.from_template(template_string)\n", "question_template.messages[0].prompt.input_variables" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "4Mc1ZC3jaydQ" }, "outputs": [], "source": [ "# @title\n", "con = \"\"\" Two roads diverged in a yellow wood,\n", "And sorry I could not travel both\n", "And be one traveler, long I stood\n", "And looked down one as far as I could\n", "To where it bent in the undergrowth;\n", "Then took the other, as just as fair,\n", "And having perhaps the better claim,\n", "Because it was grassy and wanted wear;\n", "Though as for that the passing there\n", "Had worn them really about the same,\n", "And both that morning equally lay\n", "In leaves no step had trodden black.\n", "Oh, I kept the first for another day!\n", "Yet knowing how way leads on to way,\n", "I doubted if I should ever come back.\n", "I shall be telling this with a sigh\n", "Somewhere ages and ages hence:\n", "Two roads diverged in a wood, and I—\n", "I took the one less traveled by,\n", "And that has made all the difference.\n", "—-Robert Frost—-\n", "Education Place: http://www.eduplace.com \"\"\"\n", "\n", "pre = \"Please design a 3 question quiz about which reflects the learning objectives (if inputted). The questions should be short answer. Expect the correct answers to be sentences long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct answer is right.\"\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "KXssPFyEbG3f", "outputId": "8a5389f7-3ec6-4332-d9ae-f45ff5781caf" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'Question 1: What is the main theme of Robert Frost\\'s poem \"The Road Not Taken\"?'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 11 } ], "source": [ "generate_questions(con,pre)" ] }, { "cell_type": "markdown", "metadata": { "id": "DMTybR3PVuoC" }, "source": [ "### Creating a Chain for AI Evaluation" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "Wc-3XAFQVxO_" }, "outputs": [], "source": [ "template_evaluation = \"\"\"\n", "Given\n", "1. The follwing context of the oral exam/presentation: {context} \\\n", "\n", "2. The answer from the student: {transcript} \\\n", "\n", "3. The Questions asked to the student and student answers {QA} \\\n", "\n", "Please evaluate the students performance based on {instructions} \\\n", "\n", "If no instruction is provided, you can evaluate based on your judgement of the students performance.\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "FZXeYNSVVy5g", "outputId": "94a1a4be-03a6-4c20-97b8-4333910991fa" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['QA', 'context', 'instructions', 'transcript']" ] }, "metadata": {}, "execution_count": 13 } ], "source": [ "# @title\n", "evaluate_template = ChatPromptTemplate.from_template(template_evaluation)\n", "evaluate_template.messages[0].prompt.input_variables" ] }, { "cell_type": "markdown", "metadata": { "id": "a3WUL_hFyMkr" }, "source": [ "### Test process_file" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 157 }, "id": "LJX1AKTMyVm8", "outputId": "24f8de7a-f456-47bc-b3b5-0284bbd7076f" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "\"\\ufeffHello,\\n\\n\\nWe are so excited for this semester’s partnership with Data Science Institute and Next Steps at Vanderbilt. Jonathan Wade will be interning Mondays and Wednesdays 2-5 pm and Fridays 12-4 pm starting Monday, January 31st. Jessica will be job coaching on Mondays and Wednesdays from 2-5 pm. It also used to be on Fridays from 2-4 pm but not anymore.\\n\\n\\nBelow is important information and reminders:\\n\\n\\nLocation: 1400 18th Ave S Building, Suite 2000, Nashville, TN 37212\\n\\n\\nDress: Business casual attire (This includes items such as dress pants, khakis, polos and dress shirts)\\n\\n\\nImportant Dates:\\n\\n\\nVanderbilt University is off for Spring Break March 5th - March 13th. No internships will take place these days.\\n\\n\\nAll internships end by Thursday, April 28th.\\n\\n\\nImportant COVID-19 Information: Attached is a document outlining the COVID-19 guidelines all Vanderbilt University students must follow, including Next Steps interns and job coaches while at internship sites. Please note these may change given the evolving nature of the pandemic and any changes will be communicated to internship sites, interns, and job coaches as needed.\\n\\n\\nCareer Development Resource Guide: I am also attaching the Next Steps Career Development Resource guide that outlines student expectations and provides helpful information and resources for site supervisors.\\n\\n\\nInternship Coordinator: Lynda Tricia is the coordinator for this internship and is the main point of contact for any questions or concerns that arise.\\n\\n\\nFinally, below you will find everyone's contact information for your convenience.\\n\\n\\nContacts:\\n\\n\\nIntern: Jonathan Wade, jonathan.j.wade@vanderbilt.edu, 613-472-3867\\n\\n\\nSupervisor: Ruben Miller, ruben.k.miller@vanderbilt.edu, 216-574-3176\\n\\n\\nJob Coach: Jessica Cho, jessica.cho@vanderbilt.edu, 615-999-1134\\n\\n\\nInternship Coordinator: Lynda Tricia, lynda.z.tricia@vanderbilt.edu, 606-415-9999\\n\\n\\nPlease let us know if there are any questions. Thank you!\\n\\n\\n\\n\\nNext Steps at Vanderbilt - Safety Guidelines for Internships\\nMore information on Vanderbilt’s Health and Safety Protocols can be found here: https://www.vanderbilt.edu/coronavirus/community/undergraduate-students/\\nMasks: All Next Steps interns and job coaches must wear masks indoors at internships even if the jobsite does not require masks.\\nInterns and job coaches should have a well-fitted mask that completely covers your nose and mouth, preferably a KN95, KF94 or FFP2 version.\\nLunch Breaks: If an intern or job coach needs a lunch break, they can remove their mask just when they eat or drink. They must be physically distanced from other co-workers when eating. \\nSymptom Monitoring: All interns and job coaches must be free of ANY symptoms related to COVID-19 to go to the internship. If an intern or job coach has symptoms, they should stay home, notify the Next Steps staff and internship supervisor, and get tested at Vanderbilt Student Health or with their medical provider.\\nAccording to the CDC, symptoms may appear 2 to 14 days after exposure to the virus. These include:\\n* Fever or chills\\n* Cough\\n* Shortness of breath or difficulty breathing\\n* Fatigue\\n* Muscle or body aches\\n* Headache\\n* New loss of taste or smell\\n* Sore throat\\n* Congestion or runny nose\\n* Nausea or vomiting\\n* Diarrhea\\n\\n\\nIf an intern or job coach tests positive: If an intern or job coach receives a COVID-19 positive test result, regardless of vaccination status, they should complete the following webform. The webform goes directly to the Vanderbilt Command Center.\\nThe intern or job coach will receive direct communication from the Command Center about their isolation (if they tested positive) or quarantine period (if considered a close contact) and will be instructed to contact Student Health or Occupational Health if they develop symptoms.\\nClose Contact/Quarantine: Interns or job coaches who are a close contact of someone who tests positive should complete the following webform. The webform goes directly to the Command Center.\\n* Close contacts who are unvaccinated will quarantine for 10 days based on CDC guidance.\\no Additional requirements are in place for days 10 to 14 following exposure including:\\n* For days 10 to 14 after last exposure, unvaccinated Vanderbilt community members identified as close contacts must not unmask at any time in public.\\n* Individuals should eat alone or complete any activities alone that require removing a mask in a private space during those four days between day 10-14.\\n* Close contacts who are vaccinated and asymptomatic will not have to quarantine but are recommended to monitor their symptoms and to get a COVID-19 test 5-7 days after last exposure. If asymptomatic, testing can be done at the VU testing center. If individuals develop symptoms, they should test at Student Health, Occupational Health or VUMC or other testing location in the community.\\n* Close contacts who are vaccinated and symptomatic may have to quarantine based on severity of symptoms and specific living situations. This determination will be made by their medical provider in consultation with the Command Center.\\nJob Coaching Supports: If a job coach tests positive and is unable to provide on-site supports, Next Steps staff will follow the procedures outlined below.\\n1. Identify another job coach or Next Steps staff member who can provide job coaching on-site during some of the student’s internship hours.\\n2. Identify another job coach or staff member who can check-in virtually with the student and supervisor during their shift. \\n3. Or, work with the student and supervisor to ensure natural supports are in place if the student must work without support of a job coach during that time period.\"" ] }, "execution_count": 69, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Might need some way to make pdf file to load more readable\n", "# process_file('/content/instrutor_note.docx')\n", "# process_file('/content/Big Data & Economics.pdf')\n", "# process_file('/content/Big Data & Economics.pdf')\n", "process_file('/content/Anonymized Job Coach QA Test Doc (1).txt')" ] }, { "cell_type": "markdown", "metadata": { "id": "M6IzVTjz5cex" }, "source": [ "## UI Design\n" ] }, { "cell_type": "markdown", "metadata": { "id": "u2SY4Akt_t8h" }, "source": [ "### Chatbot V2" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 853 }, "id": "6ENnsKlD_uOC", "outputId": "7b694aa8-e314-4502-a590-3245ebc3e1e0" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Colab notebook detected. This cell will run indefinitely so that you can see errors and logs. To turn off, set debug=False in launch().\n", "Note: opening Chrome Inspector may crash demo inside Colab notebooks.\n", "\n", "To create a public link, set `share=True` in `launch()`.\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "(async (port, path, width, height, cache, element) => {\n", " if (!google.colab.kernel.accessAllowed && !cache) {\n", " return;\n", " }\n", " element.appendChild(document.createTextNode(''));\n", " const url = await google.colab.kernel.proxyPort(port, {cache});\n", "\n", " const external_link = document.createElement('div');\n", " external_link.innerHTML = `\n", "
\n", " Running on \n", " https://localhost:${port}${path}\n", " \n", "
\n", " `;\n", " element.appendChild(external_link);\n", "\n", " const iframe = document.createElement('iframe');\n", " iframe.src = new URL(path, url).toString();\n", " iframe.height = height;\n", " iframe.allow = \"autoplay; camera; microphone; clipboard-read; clipboard-write;\"\n", " iframe.width = width;\n", " iframe.style.border = 0;\n", " element.appendChild(iframe);\n", " })(7860, \"/\", \"100%\", 500, false, window.element)" ] }, "metadata": {} }, { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n", " warnings.warn(warning.format(data.dtype))\n", "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n", " warnings.warn(warning.format(data.dtype))\n", "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n", " warnings.warn(warning.format(data.dtype))\n", "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n", " warnings.warn(warning.format(data.dtype))\n", "/usr/local/lib/python3.10/dist-packages/gradio/processing_utils.py:188: UserWarning: Trying to convert audio automatically from int32 to 16-bit int format.\n", " warnings.warn(warning.format(data.dtype))\n" ] }, { "output_type": "stream", "name": "stdout", "text": [ "Keyboard interruption in main thread... closing server.\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "'\\nWhat Are the Different Types of Machine Learning?\\nHow Do You Handle Missing or Corrupted Data in a Dataset?\\nHow Can You Choose a Classifier Based on a Training Set Data Size?\\nExplain the Confusion Matrix with Respect to Machine Learning Algorithms.\\nWhat Are the Differences Between Machine Learning and Deep Learning\\n'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 17 } ], "source": [ "with gr.Blocks(theme=gr.themes.Monochrome()) as demo:\n", " gr.Markdown(\"# Oral Exam App\")\n", " gr.Markdown(\"## OpenAI API key\")\n", " with gr.Box():\n", " gr.HTML(\"\"\"Embed your OpenAI API key below; if you haven't created one already, visit\n", " platform.openai.com/account/api-keys\n", " to sign up for an account and get your personal API key\"\"\",\n", " elem_classes=\"textbox_label\")\n", " input = gr.Textbox(show_label=False, type=\"password\", container=False,\n", " placeholder=\"●●●●●●●●●●●●●●●●●\")\n", " input.change(fn=embed_key, inputs=input, outputs=None)\n", "\n", " with gr.Blocks():\n", " #########################\n", " #########Context#########\n", " #########################\n", " with gr.Accordion(\"Context section\"):\n", " ### Should also allow vector stores\n", " gr.Markdown(\"## Please upload the context document(s) for Oral exam\")\n", " context_input = gr.File(label=\"Click to upload context file\",\n", " file_count=\"multiple\",\n", " file_types=[\".txt\", \".docx\", \".pdf\"])\n", " outputs_context=gr.Textbox(label=\"Context\")\n", " context_input.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n", " # upload_button = gr.Button(value=\"Show context\")\n", " # upload_button.click(process_file, context_input, outputs_context)\n", "\n", " with gr.Blocks():\n", " gr.Markdown(\"\"\"\n", " ## Generate a Premade Prompt\n", " Select your type and number of desired questions. Click \"Generate Prompt\" to get your premade prompt,\n", " and then \"Insert Prompt into Chat\" to copy the text into the chat interface below. \\\n", " You can also copy the prompt using the icon in the upper right corner and paste directly into the input box when interacting with the model.\n", " \"\"\")\n", " with gr.Row():\n", " with gr.Column():\n", " question_type = gr.Dropdown([\"Multiple Choice\", \"True or False\", \"Short Answer\", \"Fill in the Blank\", \"Random\"], label=\"Question Type\")\n", " number_of_questions = gr.Textbox(label=\"Enter desired number of questions\")\n", " sa_desired_length = gr.Dropdown([\"1-2\", \"3-4\", \"5-6\", \"6 or more\"], label = \"For short answer questions only, choose the desired sentence length for answers. The default value is 1-2 sentences.\")\n", " with gr.Column():\n", " prompt_button = gr.Button(\"Generate Prompt\")\n", " premade_prompt_output = gr.Textbox(label=\"Generated prompt (save or copy)\", show_copy_button=True)\n", " prompt_button.click(prompt_select,\n", " inputs=[question_type, number_of_questions, sa_desired_length],\n", " outputs=premade_prompt_output)\n", "\n", " #########################\n", " #######Main Audio########\n", " #########################\n", " with gr.Accordion(\"Main audio section\"):\n", " gr.Markdown(\"## Upload your audio file or start recording\")\n", "\n", " with gr.Column():\n", " with gr.Row():\n", " file_input = gr.Audio(label=\"Upload Audio\", source=\"upload\", type=\"filepath\")\n", " record_inputs = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n", "\n", " gr.Markdown(\"## Transcribe the audio uploaded or recorded\")\n", " outputs_transcribe=gr.Textbox(label=\"Transcription\")\n", "\n", " file_input.change(fn=transcribe, inputs=file_input, outputs=outputs_transcribe)\n", " record_inputs.change(fn=transcribe, inputs=record_inputs, outputs=outputs_transcribe)\n", "\n", " # #########################\n", " # ###Question Generation###\n", " # #########################\n", " # with gr.Accordion(\"Question section\"):\n", " # gr.Markdown(\"## Questions\")\n", " # with gr.Column():\n", " # outputs_qa=gr.Textbox(label=\"Generate questions or Use your own questions\")\n", " # btn3 = gr.Button(value=\"Generate questions\")\n", " # btn3.click(generate_questions, inputs=context_input, outputs=outputs_qa)\n", "\n", "\n", " ########################\n", " ##Question Generation###\n", " ########################\n", " with gr.Accordion(\"Question section\"):\n", " gr.Markdown(\"## Questions\")\n", " with gr.Row():\n", " with gr.Column():\n", " outputs_qa=gr.Textbox(label=\"Generate questions or Use your own questions\")\n", " btn1 = gr.Button(value=\"Generate questions\")\n", " btn1.click(generate_questions, inputs=[outputs_context, premade_prompt_output], outputs=outputs_qa)\n", "\n", " # with gr.Column():\n", " # submit_question=gr.Textbox(label=\"Use existing questions\")\n", " # btn4 = gr.Button(value=\"Use these questions\")\n", " # btn4.click(use_this_question, inputs=outputs_transcribe, outputs=None)\n", "\n", "\n", " #########################\n", " #######Instruction#######\n", " #########################\n", " instruction_qa_input = gr.File(label=\"Click to upload instruction file\",\n", " file_count=\"multiple\",\n", " file_types=[\".txt\", \".docx\", \".pdf\"])\n", " instruction_qa=gr.Textbox(label=\"Or please enter the instruction for question/answering section\")\n", " instruction_qa.change(fn=process_file, inputs=context_input, outputs=outputs_context)\n", "\n", "\n", " #########################\n", " #########Audio QA########\n", " #########################\n", " with gr.Accordion(\"Audio QA section\"):\n", " gr.Markdown(\"## Question answering\")\n", " gr.Markdown(\"### When you are ready to answer questions, press the 'I am ready' button\")\n", " ##### This may be iterative\n", " chatbot = gr.Chatbot([],\n", " elem_id=\"chatbot\",\n", " height=300)\n", " state = gr.State()\n", " message = gr.Textbox(show_label=False,\n", " placeholder=\"Your answer will be transcribed here\",\n", " container=False)\n", " ready_button = gr.Button(value=\"I am ready\")\n", " ready_button.click(message_and_history, inputs=[message, instruction_qa, outputs_qa, state], outputs=[chatbot, state])\n", "\n", " hidden = gr.Textbox(visible = False)\n", " btn_record = gr.Audio(label=\"Record Audio\", source=\"microphone\", type=\"filepath\")\n", " btn_record.change(fn=transcribe, inputs=btn_record, outputs=message)\n", " btn_record.clear(use_these_questions, inputs = hidden, outputs = message)\n", "\n", " submit = gr.Button(\"Submit\")\n", " submit.click(message_and_history,\n", " inputs=[message, instruction_qa, outputs_qa, state],\n", " outputs=[chatbot, state])\n", "\n", " message_records = gr.Textbox(show_label=False,\n", " container=False)\n", " show_records = gr.Button(\"Show QA history\")\n", " show_records.click(use_these_questions,\n", " inputs=state,\n", " outputs=message_records)\n", "\n", " #########################\n", " #######Evaluation########\n", " #########################\n", " with gr.Accordion(\"Evaluation section\"):\n", " gr.Markdown(\"## Evaluation\")\n", " with gr.Tab(\"General evalution\"):\n", " evalution=gr.Textbox(label=\"AI Evaluation\")\n", " btn5 = gr.Button(value=\"Evaluate\")\n", " btn5.click(ai_evaluate, inputs=[outputs_context, outputs_transcribe, message_records, instruction_qa], outputs=evalution)\n", " with gr.Tab(\"Quantitative evalution\"):\n", " table_output = gr.Dataframe(label = \"Some kind of evaluation metrics?\")\n", " btn6 = gr.Button(value=\"Evaluate\")\n", " # btn6.click(ai_evaluate, inputs=[outputs_context, message_records, outputs_qa], outputs=table_output)\n", "\n", " # demo.launch()\n", " # demo.launch(share=True)\n", " demo.launch(debug=True)\n", "\n", "'''\n", "What Are the Different Types of Machine Learning?\n", "How Do You Handle Missing or Corrupted Data in a Dataset?\n", "How Can You Choose a Classifier Based on a Training Set Data Size?\n", "Explain the Confusion Matrix with Respect to Machine Learning Algorithms.\n", "What Are the Differences Between Machine Learning and Deep Learning\n", "'''" ] }, { "cell_type": "markdown", "metadata": { "id": "g2EVIogW69Fd" }, "source": [ "## What's left\n", "- vector store (link) upload\n", "- how to not show the warning when transcribing\n", "- better prompt for evaluation\n", "- try ChatInterface of Gradio" ] }, { "cell_type": "code", "source": [], "metadata": { "id": "-YwOAtNANrx_" }, "execution_count": null, "outputs": [] } ], "metadata": { "colab": { "provenance": [], "include_colab_link": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }