{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Clustering\n", "\n", "We use a simple k-means algorithm to demonstrate how clustering can be done. Clustering can help discover valuable, hidden groupings within the data. The dataset is created in the [Obtain_dataset Notebook](Obtain_dataset.ipynb)." ] }, { "cell_type": "code", "execution_count": 433, "metadata": {}, "outputs": [], "source": [ "# imports\n", "import numpy as np\n", "import pandas as pd\n", "from ast import literal_eval\n", "# load data\n", "datafile_path = \"../2-Data/dialogues_embededd.pkl\"\n", "df = pd.read_pickle(datafile_path)" ] }, { "cell_type": "code", "execution_count": 434, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | Description | \n", "Patient | \n", "Doctor | \n", "combined | \n", "n_tokens | \n", "embedding | \n", "
---|---|---|---|---|---|---|
255916 | \n", "What medicine is best for taking out pores and... | \n", "Hi, I am (Simran) 28 years (Female) hight 5/3 ... | \n", "Hi simran, your problem can be best sort out u... | \n", "Description: What medicine is best for taking ... | \n", "282 | \n", "[-0.2568769, 0.14872037, 0.17243277, 0.2426784... | \n", "
255917 | \n", "What causes non conception despite unprotected... | \n", "Hi, i am living with my partner from last six ... | \n", "do serum tsh and serum prolactin den do hsg on... | \n", "Description: What causes non conception despit... | \n", "195 | \n", "[-0.058888793, 0.084498726, -0.05540138, 0.021... | \n", "
255918 | \n", "How long after getting chicken pox is it safe ... | \n", "i am 25 yrs old. last year in June i had an IU... | \n", "Hi, I think you should keep a gap of 3 months ... | \n", "Description: How long after getting chicken po... | \n", "254 | \n", "[-0.12680525, -0.01062898, -0.25460148, 0.0511... | \n", "
255919 | \n", "How to remove unwanted hair without any side e... | \n", "hi doctor! i m 22 years old and i want to remo... | \n", "Hello, which part of the body do you have thes... | \n", "Description: How to remove unwanted hair witho... | \n", "110 | \n", "[0.12641467, -0.1543769, 0.42004266, 0.1722529... | \n", "
255920 | \n", "I am 15. Can i shave my pubic hair? | \n", "sorry,im just curious because im 15 and when i... | \n", "haha aw. Yes your supposed to shave, trim it f... | \n", "Description: I am 15. Can i shave my pubic hai... | \n", "223 | \n", "[-0.06219541, 0.020706447, 0.38799724, -0.0219... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
256911 | \n", "Why is hair fall increasing while using Bontre... | \n", "I am suffering from excessive hairfall. My doc... | \n", "Hello Dear Thanks for writing to us, we are he... | \n", "Description: Why is hair fall increasing while... | \n", "211 | \n", "[-0.17113408, 0.10835318, 0.33148944, -0.06146... | \n", "
256912 | \n", "Why was I asked to discontinue Androanagen whi... | \n", "Hi Doctor, I have been having severe hair fall... | \n", "hello, hair4u is combination of minoxid... | \n", "Description: Why was I asked to discontinue An... | \n", "154 | \n", "[-0.24637492, 0.031407423, -0.05137701, -0.301... | \n", "
256913 | \n", "Can Mintop 5% Lotion be used by women for seve... | \n", "Hi..i hav sever hair loss problem so consulted... | \n", "HI I have evaluated your query thoroughly you... | \n", "Description: Can Mintop 5% Lotion be used by w... | \n", "191 | \n", "[-0.32340947, 0.3667281, 0.3651925, -0.0989788... | \n", "
256914 | \n", "Is Minoxin 5% lotion advisable instead of Foli... | \n", "Hi, i am 25 year old girl, i am having massive... | \n", "Hello and Welcome to ‘Ask A Doctor’ service.I ... | \n", "Description: Is Minoxin 5% lotion advisable in... | \n", "232 | \n", "[-0.18737659, 0.12219846, 0.2365137, 0.1126744... | \n", "
256915 | \n", "Are Biotin supplements need to reduce severe h... | \n", "iam having hairfall for a decade.. but fews we... | \n", "you did'nt mention about thyroid problem ...us... | \n", "Description: Are Biotin supplements need to re... | \n", "213 | \n", "[-0.032349, -0.050617322, 0.30625877, 0.201994... | \n", "
1000 rows × 6 columns
\n", "\n", " | Description | \n", "Patient | \n", "Doctor | \n", "combined | \n", "n_tokens | \n", "embedding | \n", "
---|---|---|---|---|---|---|
255916 | \n", "What medicine is best for taking out pores and... | \n", "Hi, I am (Simran) 28 years (Female) hight 5/3 ... | \n", "Hi simran, your problem can be best sort out u... | \n", "Description: What medicine is best for taking ... | \n", "282 | \n", "[-0.2568769, 0.14872037, 0.17243277, 0.2426784... | \n", "
255917 | \n", "What causes non conception despite unprotected... | \n", "Hi, i am living with my partner from last six ... | \n", "do serum tsh and serum prolactin den do hsg on... | \n", "Description: What causes non conception despit... | \n", "195 | \n", "[-0.058888793, 0.084498726, -0.05540138, 0.021... | \n", "
255918 | \n", "How long after getting chicken pox is it safe ... | \n", "i am 25 yrs old. last year in June i had an IU... | \n", "Hi, I think you should keep a gap of 3 months ... | \n", "Description: How long after getting chicken po... | \n", "254 | \n", "[-0.12680525, -0.01062898, -0.25460148, 0.0511... | \n", "
255919 | \n", "How to remove unwanted hair without any side e... | \n", "hi doctor! i m 22 years old and i want to remo... | \n", "Hello, which part of the body do you have thes... | \n", "Description: How to remove unwanted hair witho... | \n", "110 | \n", "[0.12641467, -0.1543769, 0.42004266, 0.1722529... | \n", "
255920 | \n", "I am 15. Can i shave my pubic hair? | \n", "sorry,im just curious because im 15 and when i... | \n", "haha aw. Yes your supposed to shave, trim it f... | \n", "Description: I am 15. Can i shave my pubic hai... | \n", "223 | \n", "[-0.06219541, 0.020706447, 0.38799724, -0.0219... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
256911 | \n", "Why is hair fall increasing while using Bontre... | \n", "I am suffering from excessive hairfall. My doc... | \n", "Hello Dear Thanks for writing to us, we are he... | \n", "Description: Why is hair fall increasing while... | \n", "211 | \n", "[-0.17113408, 0.10835318, 0.33148944, -0.06146... | \n", "
256912 | \n", "Why was I asked to discontinue Androanagen whi... | \n", "Hi Doctor, I have been having severe hair fall... | \n", "hello, hair4u is combination of minoxid... | \n", "Description: Why was I asked to discontinue An... | \n", "154 | \n", "[-0.24637492, 0.031407423, -0.05137701, -0.301... | \n", "
256913 | \n", "Can Mintop 5% Lotion be used by women for seve... | \n", "Hi..i hav sever hair loss problem so consulted... | \n", "HI I have evaluated your query thoroughly you... | \n", "Description: Can Mintop 5% Lotion be used by w... | \n", "191 | \n", "[-0.32340947, 0.3667281, 0.3651925, -0.0989788... | \n", "
256914 | \n", "Is Minoxin 5% lotion advisable instead of Foli... | \n", "Hi, i am 25 year old girl, i am having massive... | \n", "Hello and Welcome to ‘Ask A Doctor’ service.I ... | \n", "Description: Is Minoxin 5% lotion advisable in... | \n", "232 | \n", "[-0.18737659, 0.12219846, 0.2365137, 0.1126744... | \n", "
256915 | \n", "Are Biotin supplements need to reduce severe h... | \n", "iam having hairfall for a decade.. but fews we... | \n", "you did'nt mention about thyroid problem ...us... | \n", "Description: Are Biotin supplements need to re... | \n", "213 | \n", "[-0.032349, -0.050617322, 0.30625877, 0.201994... | \n", "
1000 rows × 6 columns
\n", "\n", " | Description | \n", "Patient | \n", "Doctor | \n", "combined | \n", "n_tokens | \n", "embedding | \n", "Cluster | \n", "distance | \n", "
---|---|---|---|---|---|---|---|---|
255916 | \n", "What medicine is best for taking out pores and... | \n", "Hi, I am (Simran) 28 years (Female) hight 5/3 ... | \n", "Hi simran, your problem can be best sort out u... | \n", "Description: What medicine is best for taking ... | \n", "282 | \n", "[-0.2568769, 0.14872037, 0.17243277, 0.2426784... | \n", "7 | \n", "3.389531 | \n", "
255917 | \n", "What causes non conception despite unprotected... | \n", "Hi, i am living with my partner from last six ... | \n", "do serum tsh and serum prolactin den do hsg on... | \n", "Description: What causes non conception despit... | \n", "195 | \n", "[-0.058888793, 0.084498726, -0.05540138, 0.021... | \n", "5 | \n", "3.848544 | \n", "
255918 | \n", "How long after getting chicken pox is it safe ... | \n", "i am 25 yrs old. last year in June i had an IU... | \n", "Hi, I think you should keep a gap of 3 months ... | \n", "Description: How long after getting chicken po... | \n", "254 | \n", "[-0.12680525, -0.01062898, -0.25460148, 0.0511... | \n", "5 | \n", "3.287058 | \n", "
255919 | \n", "How to remove unwanted hair without any side e... | \n", "hi doctor! i m 22 years old and i want to remo... | \n", "Hello, which part of the body do you have thes... | \n", "Description: How to remove unwanted hair witho... | \n", "110 | \n", "[0.12641467, -0.1543769, 0.42004266, 0.1722529... | \n", "6 | \n", "3.875464 | \n", "
255920 | \n", "I am 15. Can i shave my pubic hair? | \n", "sorry,im just curious because im 15 and when i... | \n", "haha aw. Yes your supposed to shave, trim it f... | \n", "Description: I am 15. Can i shave my pubic hai... | \n", "223 | \n", "[-0.06219541, 0.020706447, 0.38799724, -0.0219... | \n", "6 | \n", "3.841674 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
256911 | \n", "Why is hair fall increasing while using Bontre... | \n", "I am suffering from excessive hairfall. My doc... | \n", "Hello Dear Thanks for writing to us, we are he... | \n", "Description: Why is hair fall increasing while... | \n", "211 | \n", "[-0.17113408, 0.10835318, 0.33148944, -0.06146... | \n", "6 | \n", "2.765943 | \n", "
256912 | \n", "Why was I asked to discontinue Androanagen whi... | \n", "Hi Doctor, I have been having severe hair fall... | \n", "hello, hair4u is combination of minoxid... | \n", "Description: Why was I asked to discontinue An... | \n", "154 | \n", "[-0.24637492, 0.031407423, -0.05137701, -0.301... | \n", "6 | \n", "3.314942 | \n", "
256913 | \n", "Can Mintop 5% Lotion be used by women for seve... | \n", "Hi..i hav sever hair loss problem so consulted... | \n", "HI I have evaluated your query thoroughly you... | \n", "Description: Can Mintop 5% Lotion be used by w... | \n", "191 | \n", "[-0.32340947, 0.3667281, 0.3651925, -0.0989788... | \n", "6 | \n", "3.229499 | \n", "
256914 | \n", "Is Minoxin 5% lotion advisable instead of Foli... | \n", "Hi, i am 25 year old girl, i am having massive... | \n", "Hello and Welcome to ‘Ask A Doctor’ service.I ... | \n", "Description: Is Minoxin 5% lotion advisable in... | \n", "232 | \n", "[-0.18737659, 0.12219846, 0.2365137, 0.1126744... | \n", "6 | \n", "2.536337 | \n", "
256915 | \n", "Are Biotin supplements need to reduce severe h... | \n", "iam having hairfall for a decade.. but fews we... | \n", "you did'nt mention about thyroid problem ...us... | \n", "Description: Are Biotin supplements need to re... | \n", "213 | \n", "[-0.032349, -0.050617322, 0.30625877, 0.201994... | \n", "6 | \n", "3.123004 | \n", "
1000 rows × 8 columns
\n", "