{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"id": "brzvVeAsYiG2"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WMKrKfx8_3fc"
},
"source": [
"# Instructor Grading and Assessment\n",
"This notebook executes grading of student submissions based on the examples provided in the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions) from Dr. Jesse Blocher. In this iteration, we use the Unstructured File Loader, which cannot proccess .json files (the preferred format). We are working on finding a file loader that allows .json. In this version of the notebook, the model has only been trained on Question 2 from the notebook.\n",
"\n",
"To train the model, we used 2 out of the three student example from each grade brack and inputted into a .pdf with clearly defined levels. Then, we used the excluded answers to test the accuracy of the model's grading."
]
},
{
"cell_type": "markdown",
"source": [
"## Grading based on A, B, and C-level answers from previous students to Question 2 from the [Wiki](https://github.com/vanderbilt-data-science/lo-achievement/wiki/Examples-of-great,-good,-and-poor-answers-to-questions):\n",
"\n",
"**Question 2:** Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses)."
],
"metadata": {
"id": "ZTkNQ-dL5iO5"
}
},
{
"cell_type": "markdown",
"source": [
"### Creating .json file from case examples (Question 2)\n",
"The purpose of this cell is to create a json file based on the previously submitted, graded work of students based on the case file provided by Dr. Blocher in the Wiki"
],
"metadata": {
"id": "TYlGEusr64kA"
}
},
{
"cell_type": "code",
"source": [
"# Context: question\n",
"\n",
"q2 = 'Question 2: Why is machine learning so important for businesses? Answer this question generally (i.e. such that it applies to many or at least most businesses).'"
],
"metadata": {
"id": "-kmMFUaLs_Q1"
},
"execution_count": 1,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# A-level answers\n",
"\n",
"A_answer_1 = 'Machine learning is extremely important tool for businesses. It can be used in a variety of ways, but most importantly, it can be used to identify patterns within their data that might not otherwise be identified by human beings. For example, it can understand customer behaviors, optimize logistics, and expand efficiencies throughout the business. Machine learning does not get tired, meaning it can work as long as you want it to. It can sift through massive amounts of data, that no human being can look through in an efficient manner. Machine learning can be used as a tool to identify anomalies when something needs to be checked to save or gain money. The predictions that companies gain from machine learning are cheap, accurate, and automate. These machine learning algorithms can be brought to larger scales to encompass the whole business and its operations. It is important to note, Machine learning is just predictions. Predictions to understand important patterns that could make or break a company since they understand the patterns of their business more. It is an amazing tool, but should be used wisely and carefully because if not, it can expensive, useless, and straight up wrong.'\n",
"A_answer_2 = 'Machine learning is important for most of the sectors in business. Overall, it gives the company of an overview about what would be the trend for their business industry, and analyze the customer behavior to help business segment their customers groups. Today, many companies have a vast amount of information generated by behavior, computer, events, people, and devices. This massive amount of data is difficult for human to handle, and even if human manages it, it is not profitable as human labor is expensive. Thanks to machine learning, companies can utilize their in-house or even third-party data to make something useful for their business. In medical analysis, for example, with human, it takes a very long time to find patterns in thousands of MRI scans. On the other hand, machines can detect patterns in seconds by entering data as long as the information is correctly labeled or trained properly. Another example would be segmenting customer group. In marketing department, the business could use unsupervised machine learning to cluster their customer segments to generate personalized contents that are relevant for each of individuals.'\n",
"\n",
"# List creation\n",
"\n",
"A_answers_list = [A_answer_1, A_answer_2]\n"
],
"metadata": {
"id": "yQT6aExSr1dP"
},
"execution_count": 2,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# B-level answers\n",
"\n",
"B_answer_1 = 'Companies use ML models to improve different aspects of their business, like manufacturing, hiring, deployment, advertising, etc. The main goal is to improve productive and increase profitability of the company. The ML models are fed with company and externally available data to help the company optimize its departments and in turn become more financially successful/ productive. For example, using purchasing history, the company can predict who to advertise products to, to increase sales.'\n",
"B_answer_2 = 'Machine learning allows business to have automated decision, scale, predictive analysis and performance. Machine learning also helps a business have a data strategy. This is how a firm uses data, data infrastructure, governance, etc. to accomplish its strategic goals and maintain/grow their competitive advantage within their industry.'\n",
"B_answer_3 = 'The short answer is ML can help make decisions for businesses. To be clarified, ML does not make decisions for businesses. I mean it can, but people have not trusted ML enough yet and ML has not been that good to let it directly make business decisions. Business people only use ML to help themselves get intuitions of how decisions should be made and make predictions of results they might get based on their decisions. For example, if a business tries to launch a new product, it will use ML to test whether it will work or not on a small scale before it is introduced to a large scale. People called this step piloting. In this step, people collect data that is generated by using the pilot product and analyze their next move. They could tweak some features based on the feedback. If they think the data in their interest shows the product performs well, they might predict that this product will be successful when it is introduced on a large scale. Then, they will launch it.'\n",
"# List creation\n",
"\n",
"B_answers_list = [B_answer_1, B_answer_2, B_answer_3]"
],
"metadata": {
"id": "KB1CmeRwtRvf"
},
"execution_count": 3,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# C-level answers\n",
"\n",
"C_answer_1 = 'Machine learning powers many of the services we use today, such as the recommendation systems of Spotify and Netflix; search engines such as Google and Bing; social media such as TikTok and Instagram; voices such as Siri and Alexa, the list can go on. All these examples show that machine learning is already starting to play a pivotal role in today\"s data-rich world. Machines can help us sift through useful information that can lead to big breakthroughs, and we have seen the widespread use of this technology in various industries such as finance, healthcare, insurance, manufacturing, transformational change, etc.'\n",
"C_answer_3 = 'As technology advanced, there are tons of new data generated and stored. All industries experienced this surge in data, including business. There is a huge amount of business data stored and waited in the database of each firm and they need solutions to utilize these data. Machine learning is a very promising approach for firms to puts these data in and output a meaning pattern or result that could help the firms with their existing work. This could turn into a working product or provide insights that could enhance the efficiency of the company’s workflow. With machine learning, a firm could either enter a new market with the new product or save time and effort with the meaningful insights. Achieving these possibilities with the data they already owned is a low effort but high reward action. This is the reason machine learning is valued by many businesses recently.'\n",
"\n",
"# List creation\n",
"\n",
"C_answers_list = [C_answer_1, C_answer_3]"
],
"metadata": {
"id": "3diAz43othjc"
},
"execution_count": 4,
"outputs": []
},
{
"cell_type": "code",
"source": [
"Q_and_A = [\"Question:\", q2, \"A-level Answers\", A_answers_list, \"B-level Answers\", B_answers_list, \"C-level Answers\", C_answers_list]"
],
"metadata": {
"id": "oAnrMSU6u9do"
},
"execution_count": 5,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import json\n",
"from google.colab import files\n",
"\n",
"def save_example_answers(examples, filename='wiki_ABC_Q2examples.json'):\n",
" with open(filename, 'w') as file:\n",
" json.dump(examples, file)\n",
" files.download(filename)\n",
"\n",
"save_example_answers(Q_and_A)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 17
},
"id": "B16iYMEnri9s",
"outputId": "3e565b3d-804c-4b5e-acc8-efeb955c6c14"
},
"execution_count": 6,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"application/javascript": [
"\n",
" async function download(id, filename, size) {\n",
" if (!google.colab.kernel.accessAllowed) {\n",
" return;\n",
" }\n",
" const div = document.createElement('div');\n",
" const label = document.createElement('label');\n",
" label.textContent = `Downloading \"${filename}\": `;\n",
" div.appendChild(label);\n",
" const progress = document.createElement('progress');\n",
" progress.max = size;\n",
" div.appendChild(progress);\n",
" document.body.appendChild(div);\n",
"\n",
" const buffers = [];\n",
" let downloaded = 0;\n",
"\n",
" const channel = await google.colab.kernel.comms.open(id);\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
"\n",
" for await (const message of channel.messages) {\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
" if (message.buffers) {\n",
" for (const buffer of message.buffers) {\n",
" buffers.push(buffer);\n",
" downloaded += buffer.byteLength;\n",
" progress.value = downloaded;\n",
" }\n",
" }\n",
" }\n",
" const blob = new Blob(buffers, {type: 'application/binary'});\n",
" const a = document.createElement('a');\n",
" a.href = window.URL.createObjectURL(blob);\n",
" a.download = filename;\n",
" div.appendChild(a);\n",
" a.click();\n",
" div.remove();\n",
" }\n",
" "
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"application/javascript": [
"download(\"download_8ec6f24c-9653-4335-8c1b-766926434399\", \"wiki_ABC_Q2examples.json\", 5931)"
]
},
"metadata": {}
}
]
},
{
"cell_type": "markdown",
"source": [
"## **Start here** to interact with model"
],
"metadata": {
"id": "PpKRr5Vw4_0F"
}
},
{
"cell_type": "code",
"source": [
"! pip install -q langchain=='0.0.229' openai gradio numpy chromadb tiktoken unstructured pdf2image pydantic==\"1.10.8\" jq"
],
"metadata": {
"id": "UJi1Oy0CyPHD"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# import necessary libraries here\n",
"from getpass import getpass\n",
"from langchain.llms import OpenAI as openai\n",
"from langchain.chat_models import ChatOpenAI\n",
"from langchain.prompts import PromptTemplate\n",
"from langchain.document_loaders import TextLoader\n",
"from langchain.indexes import VectorstoreIndexCreator\n",
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.embeddings import OpenAIEmbeddings\n",
"from langchain.schema import SystemMessage, HumanMessage, AIMessage\n",
"import numpy as np\n",
"import os\n",
"from langchain.vectorstores import Chroma\n",
"from langchain.document_loaders.unstructured import UnstructuredFileLoader\n",
"from langchain.document_loaders import UnstructuredFileLoader\n",
"from langchain.chains import VectorDBQA\n",
"from langchain.document_loaders import JSONLoader\n",
"import json\n",
"from pathlib import Path\n",
"from pprint import pprint"
],
"metadata": {
"id": "YHytCUoExrYe"
},
"execution_count": 18,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from google.colab import files\n",
"uploaded = files.upload()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 74
},
"id": "Wpt6qsmEw8WP",
"outputId": "4563fa62-5245-4115-ecb9-326353dba29c"
},
"execution_count": 7,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"text/html": [
"\n",
" \n",
" \n",
" "
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Saving wiki_ABC_Q2examples (2).json to wiki_ABC_Q2examples (2).json\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# setup open AI api key\n",
"openai_api_key = getpass()"
],
"metadata": {
"id": "jVPEFX3ixJnM"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
"openai.api_key = openai_api_key"
],
"metadata": {
"id": "MO7VuGmrxVAr"
},
"execution_count": 19,
"outputs": []
},
{
"cell_type": "code",
"source": [
"mdl_name = 'gpt-3.5-turbo-0301'"
],
"metadata": {
"id": "4Thxj6Gk1zVS"
},
"execution_count": 20,
"outputs": []
},
{
"cell_type": "code",
"source": [
"llm = ChatOpenAI(model='gpt-3.5-turbo-16k')\n",
"messages = [\n",
" SystemMessage(content=\"You are a helpful assistant.\"),\n",
" HumanMessage(content=\"\")\n",
"]"
],
"metadata": {
"id": "Sgq9aVqpxZnK"
},
"execution_count": 36,
"outputs": []
},
{
"cell_type": "code",
"source": [
"file_path='/content/wiki_ABC_Q2examples (2).json'"
],
"metadata": {
"id": "Ak7X_ZRba48F"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"data = json.loads(Path(file_path).read_text())\n",
"data = str(data)"
],
"metadata": {
"id": "PKqQROVMc6BP"
},
"execution_count": 26,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Contruct Vector Store\n",
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
"texts = text_splitter.split_text(data)\n",
"\n",
"embeddings = OpenAIEmbeddings()\n",
"\n",
"db = Chroma.from_texts(texts, embeddings)\n",
"\n",
"qa = VectorDBQA.from_chain_type(llm=ChatOpenAI(model_name = mdl_name), chain_type=\"stuff\", vectorstore=db, k=1)"
],
"metadata": {
"id": "i2NadtLlcxur"
},
"execution_count": 37,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# A level\n",
"example1 = 'The most powerful aspect of machine learning is its ability to automate processes. If the goal is well defined and repeatable, an algorithm can be trained to perform that task far faster than any human and often with more reliability. Because of this, businesses can implement machine learning algorithms to solve problems human labor was previously required for, whether that be mental or physical labor. Although implementing machine learning often has high initial costs, because computers do not require payment outside of maintaining their operation, in the long run companies can save money by either making their workers’ tasks more efficient or by entirely automating tasks. This increases the profit margins for those firms.'\n",
"# B level\n",
"example2 ='ML systems can help improve access to data while managing compliance. It helps businesspeople deal with data in a more efficient way, since there is numerous data generated per second in such industry. If they use some of the simple classification models with high accuracy, it will save them a lot of time on data cleaning and validation, which are kind of repetitive and time-consuming. For example, NLP provides an easy way for personnel to query business information, understand business processes, and discover new relationships between business data, ideas based on intuition and insight often emerge. Using models to do prediction helps people making wiser decision. Since models can handle way more data than human, newly collected data can feed in the model and get some predictive result as a reference to the decision makers. This is significant because in this industry, time is precious, and traders must decide quickly and precisely. A little negligence will lead to a big mistake, lose a lot of money, and even affect the company\"s reputation. Models can see patterns that are not easy for human to spot, which is also valuable for modify the way people doing analysis and interpret.'\n",
"# C level\n",
"example3 = 'The machine learning model (or one a broader view, artificial intelligence) is about prediction. According to the lecture, there are tree main innovations in it. Prediction is cheap, more accurate and automated. As a result, armed with machine learning, businesses could automatically gain much more accurate and powerful capabilities in forecasting, leading to a big savings both in time and money.'\n",
"\n",
"# Randomized list of answers\n",
"training_answers = [example2, example1, example3]"
],
"metadata": {
"id": "CJxUs8lG12kd"
},
"execution_count": 38,
"outputs": []
},
{
"cell_type": "code",
"source": [
"query = f\"\"\" Please grade the following student answers: {training_answers} to the question ({q2}).\n",
"The uploaded pdf should serve as as examples of A, B, and C level answers. In the document, the\n",
"original question is printed, as well as examples of previous student answers that have recieved\n",
"A, B, and C grades (labeled accordingly)\"\"\"\n",
"\n",
"query_prefix = \"\"\" The uploaded pdf should serve as as examples of A, B, and C level answers.\n",
"In the document, the original question is printed, as well as examples of previous student answers that have recieved\n",
"A, B, and C grades (labeled accordingly).\"\"\"\n",
"answer = qa.run(query_prefix + query)\n",
"print(answer)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "naKuTxKa2-U8",
"outputId": "fc10a157-071d-4a09-bf4d-315fec1d53e9"
},
"execution_count": 56,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"The first answer would be a B-level answer. It mentions the efficiency of using machine learning for data cleaning and validation, as well as the ability to handle large amounts of data and make predictions. However, it could be more specific in terms of how machine learning can benefit businesses in various industries.\n",
"\n",
"The second answer would be an A-level answer. It highlights the ability of machine learning to automate processes and save money in the long run, as well as mentioning the reliability and efficiency of algorithms compared to human labor.\n",
"\n",
"The third answer would also be an A-level answer. It accurately describes the main innovation of machine learning as being prediction, and how it can lead to cost savings in time and money for businesses. It also hints at the potential for machine learning to improve forecasting capabilities.\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Conclusions based on Question 2\n",
"\n",
"Moving forward:\n",
"\n",
"\n",
"1. Train the model on the other questions provided in the case example\n",
"2. Request more student answers from Dr. Blocher\n",
"3. Request a grading rubric from Dr. Blocher for the case examples, which may help the model gain more context\n",
"\n"
],
"metadata": {
"id": "C7kDC2pe7bNd"
}
},
{
"cell_type": "markdown",
"source": [
"### Ouput for query with .pdf\n",
"\n",
"We also experimenting with using a .pdf file (before we figured out how to parse .json files). Here is one set of results.\n",
"\n",
"The first answer can be graded as a B. It touches on the benefits of machine learning such as improving access to data, efficiency in data management, and faster decision-making. However, the answer could be strengthened by providing more specific examples of how machine learning has benefited businesses.\n",
"\n",
"The second answer can be graded as an A. It provides a clear and concise explanation of how machine learning can automate processes and save companies money in the long run. The answer also acknowledges the initial costs of implementing machine learning but emphasizes the potential for increased profit margins.\n",
"\n",
"The third answer can also be graded as an A. It highlights the main innovation of machine learning which is prediction and how it can lead to cost and time savings for businesses. The answer is well-organized and provides a clear explanation of why machine learning is important for businesses in general.\n",
"\n",
"**Results:**\n",
"\n",
"The model did not perform as well as expected. It did not successfully grade any of the questions. The first qyestion should have been graded as an A, second as a B, and third as a C."
],
"metadata": {
"id": "iJTiGX4aZ91_"
}
},
{
"cell_type": "markdown",
"source": [
"### Output for .json queries\n",
"\n",
"**First query:**\n",
"\n",
"The first answer would be a B-level answer. It touches on the efficiency and time-saving benefits of machine learning, but could benefit from more specific examples and a deeper explanation of how it can improve business outcomes.\n",
"\n",
"The second answer would be an A-level answer. It provides a clear explanation of how machine learning can automate processes and improve efficiency, leading to cost savings and increased profits for businesses.\n",
"\n",
"The third answer would also be an A-level answer. It focuses on the predictive power of machine learning and how it can save time and money for businesses by providing more accurate forecasts. It also mentions the automation benefits of machine learning.\n",
"\n",
"Notes: Same results as first query using .pdf.\n",
"\n",
"**Second query:**\n",
"\n",
"Answer 1: B-level answer. The answer talks about how machine learning can automate processes and solve problems that were previously done by human labor. It also mentions how implementing machine learning can save money in the long run.\n",
"\n",
"Answer 2: A-level answer. The answer discusses how machine learning can help with data cleaning and validation, which can save time and enhance efficiency. It also mentions how models can handle more data than humans and can provide predictive results for decision-makers. Additionally, it talks about how machines can see patterns that are difficult for humans to spot, which can help with analysis and interpretation.\n",
"\n",
"Answer 3: C-level answer. The answer discusses how machine learning can provide accurate and powerful capabilities in forecasting, resulting in savings in time and money. It also talks about the three main innovations in machine learning, which are prediction, accuracy, and automation. The answer provides examples of different industries that have implemented machine learning, showcasing its importance in today's data-rich world.\n",
"\n",
"Notes: Graded one answer correctky (answer C).\n",
"\n",
"**Third query**\n",
"\n",
"The first answer would be a B-level answer. While it touches on some important points such as saving time on data cleaning and validation and using models for prediction, it lacks depth and doesn't provide specific examples or applications for businesses.\n",
"\n",
"The second answer would be an A-level answer. It provides a clear and concise explanation of how machine learning can automate processes and save companies money in the long run. It also acknowledges the initial costs of implementing machine learning and how it can increase profit margins.\n",
"\n",
"The third answer would also be a B-level answer. While it mentions the three main innovations of machine learning (cheap, more accurate, and automated predictions), it doesn't provide specific examples or applications for businesses and lacks depth in its explanation.\n",
"\n",
"Notes: None of the answers correct\n",
"\n",
"### General comments:\n",
"\n",
"None of the query results were consistent. There were also intsances where the output would be something like \"As an AI language model, I do not have the capability to...\". So more training and prompt engineering is certainly needed."
],
"metadata": {
"id": "FR4YMpPseoTE"
}
}
],
"metadata": {
"colab": {
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}