init

AI-Powered_Receipt_Bot.py

@@ -0,0 +1,49 @@
+import streamlit as st
+from utils import st_def
+st.set_page_config('AI-Powred 👋 Receipt Extract',  page_icon="🚀",)
+st_def.st_logo('Receipt Extract')
+st.image("./images/receipttextextraction.png")
+
+st.markdown("""
+    #### 🚀 Template-Based OCR (Optical Character Recognition) 🍨
+    Description: Using pre-defined templates to extract text from receipts based on the expected layout and structure.
+    
+    **Limitations**:
+    — Relies on consistent and standardized receipt formats, which is rarely the case in real-world scenarios.
+    — Struggles with variations in receipt layouts, such as different fonts, spacing, or orientations.
+    — Requires creating and maintaining a large number of templates to accommodate different receipt formats.
+    — Limited flexibility and adaptability to handle new or unseen receipt formats.
+
+        ### 📄Rule-Based Text Extraction📚:
+    Description: Defining a set of rules and regular expressions to extract specific information from receipts based on patterns and keywords.
+    **Limitations**:
+    — Requires extensive domain knowledge and manual effort to define and maintain the rules.
+    — Rules can become complex and difficult to manage as the variety of receipt formats increases.
+    — Struggles with handling variations in terminology, abbreviations, or language used in receipts.
+    — Limited scalability and adaptability to new receipt formats or changes in existing ones.    
+        
+        ### 🔍 Python Libraries for Traditional Machine Learning Approaches📰
+    `scikit-learn (sklearn)`: scikit-learn is a widely used Python library for machine learning tasks.
+    It provides a comprehensive set of tools for data preprocessing, feature extraction, model training, and evaluation.
+    scikit-learn offers various machine learning algorithms, including Support Vector Machines (SVM), Random Forests, and more.
+
+    `NLTK` (Natural Language Toolkit):NLTK is a popular Python library for natural language processing (NLP) tasks. It provides utilities for text preprocessing, tokenization, stemming, and feature extraction.
+    NLTK can be used in conjunction with scikit-learn for text-based machine learning tasks.
+
+    `spaCy`: spaCy is another powerful NLP library for Python.
+    It offers advanced features for text preprocessing, named entity recognition, part-of-speech tagging, and more.
+    spaCy can be used to extract additional features from the receipt text to enhance the machine learning models.
+
+    **Limitations of Traditional Approaches**:
+    The traditional approaches to receipt text extraction suffer from several limitations that hinder their effectiveness and scalability:
+
+    1. Lack of Flexibility: Traditional approaches struggle to handle the wide variety of receipt formats and layouts encountered in real-world scenarios. They often rely on fixed templates or rules, making them inflexible and difficult to adapt to new or unseen receipt formats.
+    2. Manual Effort and Domain Knowledge: Traditional approaches often require significant manual effort and domain expertise to define templates, rules, or features for text extraction. This process can be time-consuming and requires continuous updates and maintenance as receipt formats evolve.
+    3. Limited Scalability: As the volume and variety of receipts increase, traditional approaches face challenges in scaling efficiently. Manual data entry becomes impractical, and rule-based systems become complex and difficult to manage.
+    4. Sensitivity to Variations: Traditional approaches are sensitive to variations in receipt layouts, fonts, spacing, or terminology. They may struggle to accurately extract information when faced with inconsistencies or deviations from expected patterns.
+    5. Lack of Contextual Understanding: Traditional approaches often lack the ability to understand the contextual meaning and relationships between different elements in a receipt. They rely on predefined patterns and fail to capture the nuances and semantics of the text.
+    6. Limited Language Support: Traditional approaches may have limited support for multiple languages or may require separate models or rules for each language, making it challenging to process receipts from different regions or countries.
+
+   # 🍨 Text Extraction App Using Streamlit and OpenAI Vision
+    """)
+st.image("./images/zhang.gif")

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -1,13 +1,16 @@
----
-title: Receiptbot
-emoji: 🔥
-colorFrom: yellow
-colorTo: purple
-sdk: streamlit
-sdk_version: 1.33.0
-app_file: app.py
-pinned: false
-license: apache-2.0
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# AI-powered Receipt Bot
+
+The application is designed to process electronic and scanned receipts. 
+It sendsthese images to an LLM model, which extracts detailed information from the receipts. 
+Then, it stores this information in a database. 
+
+The app has two main parts: uploading and processing receipt images, and displaying the extracted data.
+
+The `process_image(image)` function takes an uploaded image, encodes it, and then sends a request to OpenAI’s API with the encoded image and the instructions on what information to extract.
+
+The response from OpenAI’s API, which includes the extracted data in JSON format, is then processed. 
+
+The function extracts the relevant data from this JSON and inserts it into the MySQL database.
+
+The database insertion involves two steps: first, inserting the receipt headerinformation, and then inserting details about each line item on the receipt.
+

Receipt Extract.py

@@ -0,0 +1,227 @@
+import streamlit as st
+import base64
+import requests
+import mysql.connector
+import json
+from datetime import datetime
+
+# Streamlit app title
+st.title("Receipt Extractor")
+
+# OpenAI API Key
+api_key = "Your Password"
+
+# Database connection parameters
+db_config = {
+    'host': 'localhost',
+    'user': 'root',
+    'password': 'Your Password',
+    'database': 'receipts'
+}
+
+var_for = """Given the receipt image provided, extract all relevant information and structure the output as detailed JSON that matches the database schema for storing receipt headers and line items. The receipt headers should include store name, slogan, address, store manager, phone number, transaction ID, date, time, cashier, subtotal, sales tax, total, gift card, charged amount, card type, authorization code, chip read, AID, issuer, policy ID, expiration date, survey message, survey website, user ID, password, and eligibility note. The line items should include SKU, description, details, and price for each item on the receipt. Exclude any sensitive information from the output. Format the JSON as follows:
+
+{
+  "receipt_headers": {
+    "store_name": "",
+    "slogan": "",
+    "address": "",
+    "store_manager": "",
+    "phone_number": "",
+    "transaction_id": "",+
+    "date": "",
+    "time": "",
+    "cashier": "",
+    "subtotal": 0,
+    "sales_tax": 0,
+    "total": 0,
+    "gift_card": 0,
+    "charged_amount": 0,
+    "card_type": "",
+    "auth_code": "",
+    "chip_read": "",
+    "aid": "",
+    "issuer": "",
+    "policy_id": "",
+    "expiration_date": "",
+    "survey_message": "",
+    "survey_website": "",
+    "user_id": "",
+    "password": "",
+    "eligibility_note": ""
+  },
+  "line_items": [
+    {
+      "sku": "",
+      "description": "",
+      "details": "",
+      "price": 0
+    }
+  ]
+}"""
+
+# Function to encode the image
+def encode_image(image):
+    return base64.b64encode(image.read()).decode('utf-8')
+
+# Function to process the uploaded image and update the database
+def process_image(image):
+    base64_image = encode_image(image)
+
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {api_key}"
+    }
+
+    payload = {
+        "model": "gpt-4-vision-preview",
+        "messages": [
+            {
+                "role": "user",
+                "content": [
+                    {
+                        "type": "text",
+                        "text": var_for
+                    },
+                    {
+                        "type": "image_url",
+                        "image_url": {
+                            "url": f"data:image/jpeg;base64,{base64_image}"
+                        }
+                    }
+                ]
+            }
+        ],
+        "max_tokens": 2048
+    }
+
+    response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
+    response_json = response.json()
+    receipt_data_str = response_json['choices'][0]['message']['content']
+
+    # Find the JSON string within the extracted content
+    receipt_data_json_str = receipt_data_str.split('```json')[1].split('```')[0].strip()
+    receipt_data = json.loads(receipt_data_json_str)
+
+    # Connect to the database
+    conn = mysql.connector.connect(**db_config)
+    cursor = conn.cursor()
+
+    # Insert into receipt_headers
+    header_insert_query = """
+    INSERT INTO receipt_headers (store_name, slogan, address, store_manager, phone_number, transaction_id, date, time, cashier, subtotal, sales_tax, total, gift_card, charged_amount, card_type, auth_code, chip_read, aid, issuer, policy_id, expiration_date, survey_message, survey_website, user_id, password, eligibility_note)
+    VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+    """
+
+    header_info = receipt_data['receipt_headers']
+    line_items = receipt_data['line_items']
+
+    # Format date, time, and expiration_date
+    formatted_date = datetime.strptime(header_info['date'], '%m/%d/%y').strftime('%Y-%m-%d')
+    formatted_time = datetime.strptime(header_info['time'], '%I:%M %p').strftime('%H:%M:%S')
+    formatted_expiration_date = datetime.strptime(header_info['expiration_date'], '%m/%d/%Y').strftime('%Y-%m-%d')
+
+    # Prepare header values
+    header_values = (
+        header_info['store_name'],
+        header_info['slogan'],
+        header_info['address'],
+        header_info['store_manager'],
+        header_info['phone_number'],
+        header_info['transaction_id'],
+        formatted_date,
+        formatted_time,
+        header_info['cashier'],
+        header_info['subtotal'],
+        header_info['sales_tax'],
+        header_info['total'],
+        header_info['gift_card'],
+        header_info['charged_amount'],
+        header_info['card_type'],
+        header_info['auth_code'],
+        header_info['chip_read'],
+        header_info['aid'],
+        header_info['issuer'],
+        header_info['policy_id'],
+        formatted_expiration_date,
+        header_info['survey_message'],
+        header_info['survey_website'],
+        header_info['user_id'],
+        header_info['password'],
+        header_info['eligibility_note']
+    )
+
+    # Insert header values
+    cursor.execute(header_insert_query, header_values)
+    receipt_id = cursor.lastrowid
+
+    # Prepare and insert line items
+    line_item_insert_query = """
+    INSERT INTO line_items (receipt_id, sku, description, details, price)
+    VALUES (%s, %s, %s, %s, %s)
+    """
+
+    for item in line_items:
+        price = float(item['price'])
+        line_item_values = (
+            receipt_id,
+            item['sku'],
+            item['description'],
+            item.get('details', ''),
+            price
+        )
+
+        cursor.execute(line_item_insert_query, line_item_values)
+
+    # Commit and close the connection
+    conn.commit()
+    cursor.close()
+    conn.close()
+
+    return receipt_data
+
+# Streamlit tabs
+tab1, tab2 = st.tabs(["Upload Receipt", "Display the Data"])
+
+with tab1:
+    st.header("Upload Receipt")
+    uploaded_file = st.file_uploader("Choose an image", type=["jpg", "jpeg", "png"])
+
+    if uploaded_file is not None:
+        # Display the uploaded image
+        st.image(uploaded_file, caption='Uploaded Receipt', use_column_width=True)
+
+        # Process the uploaded image
+        receipt_data = process_image(uploaded_file)
+
+        # Display success message
+        st.success("Message received successfully from the LLM.")
+
+        # Display the JSON output
+        st.json(receipt_data)
+
+
+with tab2:
+    st.header("Display the Data")
+
+    # Connect to the database
+    conn = mysql.connector.connect(**db_config)
+    cursor = conn.cursor()
+
+    # Fetch all records from the receipt_headers table, excluding the time column
+    cursor.execute("SELECT store_name, slogan, address, store_manager, phone_number, transaction_id, date, cashier, subtotal, sales_tax, total, gift_card, charged_amount, card_type, auth_code, chip_read, aid, issuer, policy_id, expiration_date, survey_message, survey_website, user_id, password, eligibility_note FROM receipt_headers;")
+    headers = cursor.fetchall()
+
+    # Display the headers in a table
+    st.subheader("Receipt Headers")
+    st.table(headers)
+
+    # Fetch and display all records from the line_items table
+    st.subheader("Line Items")
+    cursor.execute("SELECT * FROM line_items;")
+    items = cursor.fetchall()
+    st.table(items)
+
+    # Close the cursor and the connection
+    cursor.close()
+    conn.close()

packages.txt

@@ -0,0 +1,3 @@
+build-essential
+pkg-config
+default-libmysqlclient-dev

pages/1_📊Electronic_Receipt (e-Receipt).py

@@ -0,0 +1,38 @@
+import streamlit as st
+from utils import st_def, ai
+import openai, PyPDF2, os, time, pandas as pd
+
+st_def.st_logo(title='👋 e-Recepit Extract')
+tab1, tab2 = st.tabs(["Upload Receipt", "Display the Data"])
+page_text = []     #array for page
+
+with tab1:
+    st.markdown('An electronic receipt, commonly known as an e-receipt, is a digital version of a traditional paper receipt that is generated and delivered electronically. Instead of a tangible piece of paper, e-receipts are sent via electronic channels, such as email or mobile applications, as proof of a transaction. They include the same transaction information as paper receipts, such as the date, time, and items purchased.')
+    st.image("./data/invoice_Aaron Hawkins_38460.jpg")
+    pdf1 = st.file_uploader('Upload your e-Receipt: ', type='pdf')
+    #-----------------------------------------------
+    with st.spinner('loading ...'):
+        if pdf1:
+            doc_obj = PyPDF2.PdfReader(pdf1)
+            summary=' '
+            with st.spinner('Analyzing...'):
+                for i in range(0,len(doc_obj.pages)):
+                    # creating a page object
+                    pageObj = doc_obj.pages[i].extract_text()    # extract one page's text
+                    pageObj = pageObj.replace('\t\r','')     # tab, enter
+                    pageObj = pageObj.replace('\xa0','')     # non-breaking spaces
+                    # extracting text from page
+                    page_text.append(pageObj)                    # the whole pdf --> txt
+            # st.session_state['page_text'] = page_text
+            st.success("Load successfully. Continue to next tab: Display")
+        else:
+            st.info("waiting for uploading ...")
+    
+
+with tab2:
+    if page_text is not None:
+        # st.write(type(page_text))
+        st.code(f'raw data:  {page_text}')
+    else:
+        st.warning("Please upload a PDF to display the data.")
+    

pages/2_📑Scanned Receipt.py

@@ -0,0 +1,152 @@
+import streamlit as st, base64, json, re
+import mysql.connector
+from datetime import datetime
+from utils import ai, st_def, db
+import pandas as pd
+
+
+st_def.st_logo(title='👋 Scanned Recepit Extract')
+tab1, tab2, tab3,tab4 = st.tabs(["Upload", "Display", "Save", "Database"])
+uploaded_file = None
+base64_image = None
+model = 'gpt-4-turbo'
+
+with tab1:
+    st.image("./data/scanned_sample.png")
+    st.header("Upload Receipt")
+    uploaded_file = st.file_uploader("Upload scanned receipt: ", type=["jpg", "jpeg", "png", "pdf"])
+    if uploaded_file is not None:
+        st.image(uploaded_file, caption='Uploaded Receipt')
+        base64_image = base64.b64encode(uploaded_file.read()).decode('utf-8')
+        st.text(base64_image)
+        st.success("Load successfully. Continue to next tab: Display")
+
+with tab2:
+    if not base64_image:
+        st.error('Please upload scanned receipt first.')
+    else:
+        openai_api_key= st_def.st_sidebar()
+        if not openai_api_key:
+            st.info('Please enter OpenAI’s API key to continue extract the receipt uploaded.')
+        else: 
+            var_for = """Given the receipt image provided, extract all relevant information and structure the output as detailed JSON that matches the database schema for storing receipt headers and line items. The receipt headers should include store name, slogan, address, store manager, phone number, transaction ID, date, time, cashier, subtotal, sales tax, total, gift card, charged amount, card type, authorization code, chip read, AID, issuer, policy ID, expiration date, survey message, survey website, user ID, password, and eligibility note. The line items should include SKU, description, details, and price for each item on the receipt. Exclude any sensitive information from the output. Format the JSON as follows:
+                {"receipt_headers": {"store_name": "",                "slogan": "",                "address": "",                "store_manager": "",
+                        "phone_number": "",                "transaction_id": "",                "date": "",                "time": "",
+                        "cashier": "",                "subtotal": 0,                "sales_tax": 0,                "total": 0,
+                        "gift_card": 0,                "charged_amount": 0,                "card_type": "",                "auth_code": "",
+                        "chip_read": "",                "aid": "",                "issuer": "",                "policy_id": "",
+                        "expiration_date": "",                "survey_message": "",                "survey_website": "",                "user_id": "",
+                        "password": "",                "eligibility_note": ""            },
+                    "line_items": [{"sku": "",                "description": "",                "details": "",                "price": 0}]}"""    
+            # receipt_data_str = ai.ai_vision(var_for = var_for, openai_api_key=openai_api_key, model_v=model, base64_image=base64_image)
+            # with open('re.txt', 'w') as file:
+            #     file.write(receipt_data_str)
+            # st.write(receipt_data_str)
+
+            with open('re1.txt', 'r') as file:  receipt_data_str = file.read()
+            
+            start_index = receipt_data_str.find("{")    # Find the starting index of the JSON data (excluding the leading ```)
+            end_index = receipt_data_str.rfind("}")+1   # Find the ending index of the JSON data (excluding the trailing ```)
+            json_data = receipt_data_str[start_index:end_index] # Extract the JSON data as a substring
+
+            receipt_dict = json.loads(json_data)
+            
+            col1, col2 = st.columns(2)
+
+            with col1:
+                st.header("Scanned Receipt")
+                st.image(uploaded_file, caption='Uploaded Receipt')
+
+            with col2:
+                st.header("Extracted Data")
+                st.write(receipt_dict)
+            
+            db.mysql_insert_receipt(receipt_dict)
+
+            # Display success message
+            st.success("Message received successfully from the LLM.")
+
+    with tab3:
+        cursor, conn = db.mysql_conn()
+        st.header("Display the Data")
+        # Fetch all records from the receipt_headers table, excluding the time column
+        cursor.execute("SELECT store_name, slogan, address, store_manager, phone_number, transaction_id, date, cashier, subtotal, sales_tax, total, gift_card, charged_amount, card_type, auth_code, chip_read, aid, issuer, policy_id, expiration_date, survey_message, survey_website, user_id, password, eligibility_note FROM receipt_headers;")
+        headers = cursor.fetchall()
+
+        # Display the headers in a table
+        st.subheader("Receipt Headers")
+        st.table(headers)
+
+        # Fetch and display all records from the line_items table
+        st.subheader("Line Items")
+        cursor.execute("SELECT * FROM line_items;")
+        items = cursor.fetchall()
+        st.table(items)
+
+        # Close the cursor and the connection
+        cursor.close()
+        conn.close()
+        
+    with tab4:        
+        # Receipt headers data
+        receipt_headers_data = {
+            "store_name": "Starbucks E Food Court #16599",
+            "slogan": "OPERATED BY HMS",
+            "address": "Harry Reid Airport",
+            "store_manager": "",
+            "phone_number": "",
+            "transaction_id": "CHK 111704",
+            "date": "12/2/2022",
+            "time": "1:23 PM",
+            "cashier": "",
+            "subtotal": 6.00,
+            "sales_tax": 0.50,
+            "total": 6.50,
+            "gift_card": 0,
+            "charged_amount": 6.50,
+            "card_type": "MasterCard",
+            "auth_code": "",
+            "chip_read": "",
+            "aid": "",
+            "issuer": "",
+            "policy_id": "",
+            "expiration_date": "",
+            "survey_message": "We value your feedback! Scan the QR code below to share your experience.",
+            "survey_website": "https://hmshost.com/contact/",
+            "user_id": "",
+            "password": "",
+            "eligibility_note": ""
+        }
+
+        # Line items data
+        line_items_data = [
+            {
+                "sku": "",
+                "description": "1 GR GRN TEA LAT",
+                "details": "To Go",
+                "price": 6.00
+            },
+            {
+                "sku": "d34358axe",
+                "description": "2 GR APL",
+                "details": "To Go",
+                "price": 16.00
+            },
+            {
+                "sku": "12x3ce",
+                "description": "COF BAN SILK",
+                "details": "",
+                "price": 61.00
+            }
+        ]
+
+        # Create DataFrames
+        receipt_headers_df = pd.DataFrame(receipt_headers_data, index=[0])
+        line_items_df = pd.DataFrame(line_items_data)
+
+        # Display the DataFrames
+        st.write("Receipt Headers DataFrame:")
+        st.write(receipt_headers_df)
+
+        st.write("\nLine Items DataFrame:")
+        st.write(line_items_df)

re.txt

@@ -0,0 +1,50 @@
+{
+  "receipt_headers": {
+    "store_name": "Starbucks E Food Court #16599",
+    "slogan": "OPERATED BY HMS",
+    "address": "Harry Reid Airport",
+    "store_manager": "",
+    "phone_number": "",
+    "transaction_id": "CHK 111704",
+    "date": "12/2/2022",
+    "time": "1:23 PM",
+    "cashier": "",
+    "subtotal": 6.00,
+    "sales_tax": 0.50,
+    "total": 6.50,
+    "gift_card": 0,
+    "charged_amount": 6.50,
+    "card_type": "MasterCard",
+    "auth_code": "",
+    "chip_read": "",
+    "aid": "",
+    "issuer": "",
+    "policy_id": "",
+    "expiration_date": "",
+    "survey_message": "We value your feedback! Scan the QR code below to share your experience.",
+    "survey_website": "https://hmshost.com/contact/",
+    "user_id": "",
+    "password": "",
+    "eligibility_note": ""
+  },
+  "line_items": [
+    {
+      "sku": "",
+      "description": "1 GR GRN TEA LAT",
+      "details": "To Go",
+      "price": 6.00
+    }
+    {
+      "sku": "d34358axe",
+      "description": "2 GR APL",
+      "details": "To Go",
+      "price": 16.00
+    }
+    {
+      "sku": "12x3ce",
+      "description": "COF BAN SILK",
+      "details": "",
+      "price": 61.00
+    }
+  ]
+}

re1.txt

@@ -0,0 +1,44 @@
+Here is the extracted information from the receipt image provided, structured as a JSON object according to the given schema.
+
+```json
+{
+  "receipt_headers": {
+    "store_name": "Starbucks E Food Court #16599",
+    "slogan": "OPERATED BY HMS",
+    "address": "Harry Reid Airport",
+    "store_manager": "",
+    "phone_number": "",
+    "transaction_id": "CHK 111704",
+    "date": "12/2/2022",
+    "time": "1:23 PM",
+    "cashier": "",
+    "subtotal": 6.00,
+    "sales_tax": 0.50,
+    "total": 6.50,
+    "gift_card": 0,
+    "charged_amount": 6.50,
+    "card_type": "MasterCard",
+    "auth_code": "",
+    "chip_read": "",
+    "aid": "",
+    "issuer": "",
+    "policy_id": "",
+    "expiration_date": "",
+    "survey_message": "We value your feedback! Scan the QR code below to share your experience.",
+    "survey_website": "https://hmshost.com/contact/",
+    "user_id": "",
+    "password": "",
+    "eligibility_note": ""
+  },
+  "line_items": [
+    {
+      "sku": "",
+      "description": "1 GR GRN TEA LAT",
+      "details": "To Go",
+      "price": 6.00
+    }
+  ]
+}
+```
+
+Note: Some fields are left empty ("") as they are not provided in the receipt. Additional fields might be required to structure JSON perfectly according to a specified schema which were not visible or mentioned in the image of the receipt provided.

requirements.txt

@@ -0,0 +1,8 @@
+streamlit
+streamlit_extras
+openai
+tenacity
+PyPDF2
+mysql-connector-python
+SQLAlchemy
+mysqlclient

serialized_response.json

@@ -0,0 +1 @@
+"The information extracted from the receipt image is structured into a JSON format as specified:\n\n```json\n{\n  \"receipt_headers\": {\n    \"store_name\": \"Trader Joe's\",\n    \"slogan\": \"\",\n    \"address\": \"100 Front Street Santa Cruz, CA 95060\",\n    \"store_manager\": \"\",\n    \"phone_number\": \"831-425-0140\",\n    \"transaction_id\": \"****6000\",\n    \"date\": \"\",\n    \"time\": \"\",\n    \"cashier\": \"\",\n    \"subtotal\": 9.98,\n    \"sales_tax\": 0,\n    \"total\": 9.98,\n    \"gift_card\": 0,\n    \"charged_amount\": 9.98,\n    \"card_type\": \"MasterCard\",\n    \"auth_code\": \"N64K4H\",\n    \"chip_read\": \"\",\n    \"aid\": \"\",\n    \"issuer\": \"\",\n    \"policy_id\": \"\",\n    \"expiration_date\": \"\",\n    \"survey_message\": \"\",\n    \"survey_website\": \"\",\n    \"user_id\": \"\",\n    \"password\": \"\",\n    \"eligibility_note\": \"\"\n  },\n  \"line_items\": [\n    {\n      \"sku\": \"\",\n      \"description\": \"WRAP ORG SOUTHWEST CHICK\",\n      \"details\": \"\",\n      \"price\": 5.39\n    },\n    {\n      \"sku\": \"\",\n      \"description\": \"CUT FRUITFUL MEDLEY 16 O\",\n      \"details\": \"\",\n      \"price\": 3.99\n    }\n  ]\n}\n```\n\nNote:\n- Fields such as the \"slogan\", \"store_manager\", \"date\", \"time\", \"cashier\", \"sales tax\", \"chip_read\", \"aid\", \"issuer\", \"policy_id\", \"expiration_date\", \"survey_message\", \"survey_website\", \"user_id\", \"password\", and \"eligibility_note\" were not available or not applicable from the data provided in the receipt and are thus left empty or set to default values.\n- The SKU for each item is not listed on the receipt, so these entries remain empty.\n- The payment details include truncation of the transaction ID for security, and similar truncation is applied for some parts of the payment card number.\n- The address is inferred to include city and state based on the typical formatting found on receipts, despite them not being distinctly separated in the text."

utils/ai.py

@@ -0,0 +1,60 @@
+import openai
+from tenacity import retry, wait_random_exponential, stop_after_attempt
+# client = openai.OpenAI() # not acceptable in streamlit
+# curl https://api.openai.com/v1/models -H "Authorization: Bearer sk-proj-FcKBzwDnp"
+
+@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))
+def aichat(messages, openai_api_key):
+    try:
+        client = openai.OpenAI(api_key = openai_api_key)
+        response = client.chat.completions.create(
+            messages=messages,
+            model="gpt-3.5-turbo-0125",
+            # stream=True,
+            # max_tokens=2000
+        )
+        return response.choices[0].message.content
+    except Exception as e:
+        print("Unable to generate ChatCompletion response")
+        print(f"Exception: {e}")
+        return e
+
+
+@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))
+def ai_vision(var_for, openai_api_key, model_v, base64_image):
+    try:
+        client = openai.OpenAI(api_key = openai_api_key)
+        response = client.chat.completions.create(
+            model=model_v,
+            messages=[
+                {
+                "role": "user",
+                "content": [
+                    {"type": "text", "text": var_for},
+                    {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": f"data:image/jpeg;base64,{base64_image}",
+                    },
+                    },
+                ],
+                }
+            ],
+            max_tokens=1024,
+        )
+
+        return response.choices[0].message.content
+        # return response.choices[0]
+    except Exception as e:
+        print("Unable to generate ChatCompletion response")
+        print(f"Exception: {e}")
+        return e
+
+
+def get_embedding(text, model="text-embedding-3-small"):
+    # client = openai.OpenAI(api_key = openai_api_key)
+    text = text.replace("\n", " ")
+    # return client.embeddings.create(input = [text], model=model).data[0].embedding
+
+# text = "test embedding"
+# embeddings = get_embedding(text)

utils/db.py

@@ -0,0 +1,140 @@
+import os, mysql.connector, streamlit as st
+from datetime import datetime
+
+def mysql_conn():
+    conn = mysql.connector.connect(
+                host="mysql-omni-omni.b.aivencloud.com",
+                port='21906',
+                user="avnadmin",
+                password= os.environ.get('MYSQL_PWD'),
+                database = 'defaultdb',            )
+    cursor = conn.cursor()
+    return cursor, conn
+
+def mysql_check():
+    cursor, conn = mysql_conn()
+    cursor.execute("SHOW TABLES;")
+    # cursor.execute("DESCRIBE receipt_headers_allvarchar;")
+    cursor.close()
+    conn.close()
+
+
+def mysql_create_receipt_table():
+    cursor, conn = mysql_conn()
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS receipt_headers (
+            receipt_id INT AUTO_INCREMENT PRIMARY KEY,
+            store_name VARCHAR(255),                slogan VARCHAR(255),                address VARCHAR(255),                store_manager VARCHAR(255),                phone_number VARCHAR(50),
+            transaction_id VARCHAR(255),                date DATE,                time TIME,                cashier VARCHAR(255),                subtotal DECIMAL(10,2),
+            sales_tax DECIMAL(10,2),            total DECIMAL(10,2),            gift_card DECIMAL(10,2),            charged_amount DECIMAL(10,2),            card_type VARCHAR(50),
+            auth_code VARCHAR(50),                chip_read VARCHAR(50),                aid VARCHAR(50),                issuer VARCHAR(255),                policy_id VARCHAR(50),
+            expiration_date DATE,                survey_message TEXT,                survey_website VARCHAR(255),                user_id VARCHAR(255),                password VARCHAR(255),                eligibility_note TEXT            )
+        """)
+    # Create line_items table
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS line_items (
+            line_item_id INT AUTO_INCREMENT PRIMARY KEY,
+            receipt_id INT,                sku VARCHAR(255),                description VARCHAR(255),                details TEXT,                price DECIMAL(10,2),
+            FOREIGN KEY (receipt_id) REFERENCES receipt_headers(receipt_id)            )
+        """)
+    conn.commit()
+    cursor.close()
+    conn.close()
+
+
+def mysql_create_receipt_table_allvarchar():
+    cursor, conn = mysql_conn()
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS receipt_headers_allvarchar (
+            receipt_id INT AUTO_INCREMENT PRIMARY KEY,
+            store_name VARCHAR(255),                slogan VARCHAR(255),                address VARCHAR(255),                store_manager VARCHAR(255),                phone_number VARCHAR(50),
+            transaction_id VARCHAR(255),                date VARCHAR(50),                time VARCHAR(50),                cashier VARCHAR(255),                subtotal DECIMAL(10,2),
+            sales_tax DECIMAL(10,2),            total DECIMAL(10,2),            gift_card DECIMAL(10,2),            charged_amount DECIMAL(10,2),            card_type VARCHAR(50),
+            auth_code VARCHAR(50),                chip_read VARCHAR(50),                aid VARCHAR(50),                issuer VARCHAR(255),                policy_id VARCHAR(50),
+            expiration_date VARCHAR(50),                survey_message TEXT,                survey_website VARCHAR(255),                user_id VARCHAR(255),                password VARCHAR(255),                eligibility_note TEXT            )
+        """)
+    # Create line_items table
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS line_items_allvarchar (
+            line_item_id INT AUTO_INCREMENT PRIMARY KEY,
+            receipt_id INT,     sku VARCHAR(255),   description VARCHAR(255),   details TEXT,   price DECIMAL(10,2),
+            FOREIGN KEY (receipt_id) REFERENCES receipt_headers(receipt_id))
+        """)
+    conn.commit()
+    cursor.close()
+    conn.close()
+
+def mysql_insert_receipt(receipt_data):
+    cursor, conn = mysql_conn()
+    # Insert into receipt_headers
+    header_insert_query = """
+        INSERT INTO receipt_headers (store_name, slogan, address, store_manager, phone_number, transaction_id, date, time, cashier, subtotal, sales_tax, total, gift_card, charged_amount, card_type, auth_code, chip_read, aid, issuer, policy_id, expiration_date, survey_message, survey_website, user_id, password, eligibility_note)
+        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+    """
+    header_info = receipt_data['receipt_headers']
+    line_items = receipt_data['line_items']
+    # Format date, time, and expiration_date
+    formatted_date = datetime.strptime(header_info['date'], '%m/%d/%Y').strftime('%Y-%m-%d')
+    formatted_time = datetime.strptime(header_info['time'], '%I:%M %p').strftime('%H:%M:%S')
+    # formatted_expiration_date = datetime.strptime(header_info['expiration_date'], '%m/%d/%Y').strftime('%Y-%m-%d')
+    formatted_expiration_date = datetime.strptime(header_info['expiration_date'], '%m/%d/%Y').strftime('%Y-%m-%d') if header_info['expiration_date'] else '01/01/2023'
+    formatted_expiration_date = datetime.strptime(header_info['expiration_date'], '%m/%d/%Y') if header_info['expiration_date'] else datetime.strptime('01/01/1000', '%m/%d/%Y')
+    
+    # Prepare header values
+    header_values = (
+        header_info['store_name'],
+        header_info['slogan'],
+        header_info['address'],
+        header_info['store_manager'],
+        header_info['phone_number'],
+        header_info['transaction_id'],
+        formatted_date,
+        formatted_time,
+        header_info['cashier'],
+        header_info['subtotal'],
+        header_info['sales_tax'],
+        header_info['total'],
+        header_info['gift_card'],
+        header_info['charged_amount'],
+        header_info['card_type'],
+        header_info['auth_code'],
+        header_info['chip_read'],
+        header_info['aid'],
+        header_info['issuer'],
+        header_info['policy_id'],
+        formatted_expiration_date,
+        header_info['survey_message'],
+        header_info['survey_website'],
+        header_info['user_id'],
+        header_info['password'],
+        header_info['eligibility_note']
+    )
+
+    # Insert header values
+    cursor.execute(header_insert_query, header_values)
+    receipt_id = cursor.lastrowid
+
+    # Prepare and insert line items
+    line_item_insert_query = """
+    INSERT INTO line_items (receipt_id, sku, description, details, price)
+    VALUES (%s, %s, %s, %s, %s)
+    """
+
+    for item in line_items:
+        price = float(item['price'])
+        line_item_values = (
+            receipt_id,
+            item['sku'],
+            item['description'],
+            item.get('details', ''),
+            price
+        )
+
+        cursor.execute(line_item_insert_query, line_item_values)
+
+    # Commit and close the connection
+    conn.commit()
+    cursor.close()
+    conn.close()
+    
+    

utils/st_def.py

@@ -0,0 +1,64 @@
+import streamlit as st
+from streamlit_extras.add_vertical_space import add_vertical_space
+
+def st_sidebar():
+    with st.sidebar:
+        # store_link = st.text_input("Enter Your Store URL:",   value="http://hypech.com/StoreSpark", disabled=True, key="store_link")
+        openai_api_key = st.text_input("OpenAI API Key", key="chatbot_api_key", type="password")
+        st.write("[Get an OpenAI API key](https://platform.openai.com/account/api-keys)")
+        add_vertical_space(2)
+        st.write('Made with ❤️ by [aiXpertLab](https://hypech.com)')
+    return openai_api_key
+   
+def st_logo(title="aiXpert!"):
+    st.title(title)
+
+    st.markdown("""
+    <style>
+        [data-testid="stSidebarNav"] {
+            background-image: url(https://hypech.com/images/logo/st_receiptbot.png);
+            background-size: 300px; /* Set the width and height of the image */
+            background-repeat: no-repeat;
+            padding-top: 80px;
+            background-position: 15px 10px;
+        }
+    </style>
+    """,
+    unsafe_allow_html=True,
+)
+
+
+def st_text_preprocessing_contents():
+    st.markdown("""
+        - Normalize Text
+        - Remove Unicode Characters
+        - Remove Stopwords
+        - Perform Stemming and Lemmatization
+    """)    
+
+
+def st_read_pdf():
+    st.markdown("""
+Because OpenAI has a limit on the input prompt size, we would like to send the data to be summarized in parts. 
+There can be multiple ways to split the text. For the sake of simplicity, we will divide the whole book on the basis of pages. 
+A **better strategy** will be to split it on the basis of paragraphs. However, it will increase the number of API calls increasing the overall time.
+
+We will store each page in a list and then summarize it.
+    """)    
+    st.image("./images/book.png")
+
+def st_summary():
+    st.markdown("Now we will start prompting. This is a matter of experiment to figure out the best prompt. However, there are a few basic guidelines on how to do it efficiently. In some upcoming articles, we will discuss the art of prompting in more detail. You can use the prompt for now, which has worked well for me. ")
+    # st.image("./images/featureengineering.png")
+
+def st_case_study():
+        st.image("./images/NLP-Pipeline.png")
+        # main_contents="""
+        #     ### 🚀 Bridge the Gap: Chatbots for Every Store 🍨
+        #     Tired of missing out on sales due to limited customer support options? Struggling to keep up with growing customer inquiries? Store Spark empowers you to seamlessly integrate a powerful ChatGPT-powered chatbot into your website, revolutionizing your customer service and boosting engagement. No coding required! No modifications for current site needed!
+        #     ### 📄Key Features📚:
+        #     -  🔍 No Coding Required: Say goodbye to developer fees and lengthy website updates. Store Spark’s user-friendly API ensures a smooth integration process.
+        #     -  📰 Empower Your Business: Offer instant customer support, improve lead generation, and boost conversion rates — all with minimal setup effort.
+        #     -  🍨 Seamless Integration: Maintain your existing website design and user experience. Store Spark seamlessly blends in, providing a unified customer journey.
+        #     """
+