Upload 7 files

Blueprint.md

@@ -0,0 +1,409 @@
+# Technical Blueprint for Innovative Idea Generator Chatbot (Updated)
+
+## Table of Contents
+
+1. [Introduction](#introduction)
+2. [High-Level Overview](#high-level-overview)
+   - [Purpose and Goals](#purpose-and-goals)
+   - [Key Functionalities](#key-functionalities)
+3. [System Architecture](#system-architecture)
+   - [Components](#components)
+   - [Data Flow](#data-flow)
+4. [Detailed Module Design](#detailed-module-design)
+   - [Chatbot Interaction Module](#chatbot-interaction-module)
+     - [Stage-Based Interaction](#stage-based-interaction)
+     - [Dynamic Follow-ups](#dynamic-follow-ups)
+     - [Context Preservation](#context-preservation)
+   - [Data Collection and Refinement Module](#data-collection-and-refinement-module)
+     - [Input Parsing](#input-parsing)
+     - [Data Storage](#data-storage)
+     - [Data Refinement](#data-refinement)
+   - [Stage Management Module](#stage-management-module)
+     - [Stage Buttons Interface](#stage-buttons-interface)
+     - [Function Modularity](#function-modularity)
+   - [Web Search Integration Module](#web-search-integration-module)
+     - [LLM-Triggered Web Searches](#llm-triggered-web-searches)
+     - [User-Triggered Web Searches](#user-triggered-web-searches)
+   - [Database Management Module](#database-management-module)
+     - [Data Storage Format](#data-storage-format)
+     - [Dynamic Editing](#dynamic-editing)
+5. [Implementation Details](#implementation-details)
+   - [Python and Gradio Usage](#python-and-gradio-usage)
+   - [Module and Function Organization](#module-and-function-organization)
+   - [Error Handling and Validation](#error-handling-and-validation)
+   - [LLM Context and Memory Management](#llm-context-and-memory-management)
+   - [User Interface Enhancements](#user-interface-enhancements)
+6. [Additional Considerations](#additional-considerations)
+   - [Scalability and Extensibility](#scalability-and-extensibility)
+   - [Security and Privacy](#security-and-privacy)
+   - [Testing and Validation](#testing-and-validation)
+7. [Conclusion](#conclusion)
+
+---
+
+## Introduction
+
+This blueprint outlines the technical design for the **Innovative Idea Generator Chatbot**, an application aimed at transforming user ideas into well-defined project proposals ready for prototyping. The chatbot guides users through a structured, stage-based workflow, refining their inputs, and dynamically storing data for further editing and retrieval. The application is developed using **Python** and **Gradio**, adhering strictly to Python-based frameworks without the use of JavaScript, Next.js, or similar technologies.
+
+---
+
+## High-Level Overview
+
+### Purpose and Goals
+
+- **Objective**: Assist users in refining raw ideas into structured project proposals.
+- **Outcome**: Generate a comprehensive proposal by collecting, refining, and storing user inputs through guided, stage-by-stage interactions.
+
+### Key Functionalities
+
+1. **Stage-Based Interaction**: Users navigate through the idea development process using individual stage buttons.
+2. **Dynamic Follow-ups**: Trigger additional questions based on user responses for clarification.
+3. **Data Refinement**: Rewrite user inputs for clarity and store them dynamically.
+4. **Context Preservation**: Maintain conversation history for seamless interactions.
+5. **Web Search Integration**: Allow both LLM and users to perform web searches during the interaction.
+6. **Data Storage and Editing**: Store all collected data in a dynamic database accessible for modifications.
+7. **Stage Navigation with Buttons**: Replace progress tracking bar with a checklist of clickable stage buttons.
+8. **Stage-Specific Processing**: LLM processes one stage at a time based on user-activated buttons.
+9. **Customizable AI Settings**: Allow users to select models and adjust generation parameters.
+10. **API Key Management**: Enable users to securely input their Hugging Face API keys for model interactions.
+
+---
+
+## System Architecture
+
+### Components
+
+1. **User Interface (UI)**: Built with Gradio, providing a stage-based button interface and corresponding text fields.
+2. **Chatbot Engine**: Manages conversations, including stage-specific prompts and response handling.
+3. **Language Model (LLM) Interaction**: Utilizes an LLM for generating responses and processing inputs.
+4. **Data Storage**: A database (SQL or CSV) for storing refined user inputs and conversation history.
+5. **Web Search Module**: Integrates web search capabilities using APIs (e.g., DuckDuckGo Search).
+6. **API Management Module**: Handles secure input and storage of user-provided API keys.
+7. **Stage Management Interface**: Manages the display and interaction of stage buttons and corresponding text fields.
+
+### Data Flow
+
+1. **User Activation**: User clicks a stage button to initiate a specific stage.
+2. **LLM Processing**: Chatbot focuses on the selected stage, prompting the user for relevant information.
+3. **User Input**: User provides input for the stage.
+4. **Data Refinement**: LLM refines the input for clarity and coherence.
+5. **Form Population**: User presses the 'Fill Out Form' button to populate the corresponding text field with refined data.
+6. **Data Storage**: Refined inputs are stored in the database.
+7. **Web Search**: Triggered by the LLM or user, with results incorporated into the conversation.
+8. **Stage Completion**: User marks the stage as complete before moving to the next stage.
+9. **Feedback Loop**: Process repeats until all stages are completed.
+
+---
+
+## Detailed Module Design
+
+### Chatbot Interaction Module
+
+#### Stage-Based Interaction
+
+- **Stage Buttons**: Ten individual buttons represent each main question/stage. Users click these buttons to activate and work on specific stages.
+- **Exclusive Focus**: Upon activation of a stage button (e.g., 'Idea Name'), the LLM concentrates solely on tasks related to that stage.
+- **Sequential Processing**: Users can navigate stages in any order, allowing flexibility in the workflow.
+
+#### Dynamic Follow-ups
+
+- **Conditional Questions**: Based on user responses within a stage, the chatbot may ask additional questions for further clarification.
+- **LLM Analysis**: The LLM analyzes responses to determine if more information is needed.
+- **Implementation**:
+  - Use conditional logic to assess the completeness of user inputs.
+  - Trigger follow-up prompts when necessary to ensure comprehensive data collection.
+
+#### Context Preservation
+
+- **Conversation History**: Maintain a log of all interactions to provide context for each stage.
+- **LLM Access**: Pass relevant conversation history to the LLM for consistent and informed responses.
+- **Implementation**:
+  - Utilize a list or database to store message histories.
+  - Ensure that the LLM receives pertinent history relevant to the current stage without exceeding token limits.
+
+### Data Collection and Refinement Module
+
+#### Input Parsing
+
+- **Structured Data Extraction**: Parse LLM responses to extract relevant information specific to each stage.
+- **Tags and Markers**: Use specific markers (e.g., `<form_data></form_data>`) to delineate structured data within responses.
+
+#### Data Storage
+
+- **Database Choice**: Utilize SQL or CSV files for robust data storage.
+- **Dynamic Updates**: Allow real-time updates to the stored data as users complete or modify stages.
+- **Implementation**:
+  - Define a data model using Pydantic or similar for validation.
+  - Implement functions to read and write data to the database seamlessly.
+
+#### Data Refinement
+
+- **LLM Rewriting**: The LLM refines user inputs for clarity, coherence, and conciseness within each stage.
+- **Validation**: Ensure the refined data aligns with the expected format and requirements of the stage.
+- **Implementation**:
+  - Employ the LLM to generate refined versions of user inputs.
+  - Update the database with the refined data upon user confirmation.
+
+### Stage Management Module
+
+#### Stage Buttons Interface
+
+- **Checklist of Stages**: Replace the traditional progress tracking bar with a checklist comprising individual stage buttons.
+- **User Activation**: Users manually activate each stage by clicking the corresponding button, initiating focused processing by the LLM.
+- **'Fill Out Form' Button**: After completing a stage, users press the 'Fill Out Form' button to populate the associated text field with refined data.
+- **Implementation**:
+  - Design the UI to display ten stage buttons, each linked to a specific question.
+  - Ensure that each button triggers the LLM to handle only the tasks pertinent to that stage.
+
+#### Function Modularity
+
+- **Compartmentalization**: Each stage is implemented as a separate function or class method to ensure modularity.
+- **Independent Modification**: Allows individual stages to be updated or modified without impacting others.
+- **Implementation**:
+  - Define separate methods for each stage within the chatbot class.
+  - Utilize inheritance or composition for shared functionalities across stages.
+
+### Web Search Integration Module
+
+#### LLM-Triggered Web Searches
+
+- **Automatic Searches**: The LLM determines when external information is necessary and triggers web searches accordingly.
+- **Search Execution**: Perform searches using APIs and integrate the results into the conversation context.
+- **Implementation**:
+  - LLM outputs a special token or command when a web search is needed.
+  - Chatbot captures this command and performs the search using the designated API.
+
+#### User-Triggered Web Searches
+
+- **User Commands**: Users can initiate searches by typing '@' followed by their query.
+- **Query Optimization**: The LLM rewrites user queries to enhance search effectiveness.
+- **Implementation**:
+  - Detect messages starting with '@' to identify user-initiated searches.
+  - Pass the query through the LLM for optimization before performing the search.
+
+### Database Management Module
+
+#### Data Storage Format
+
+- **Structured Data**: Store data in well-defined tables or a structured SQLite database to ensure data integrity.
+- **Fields**: Include all necessary fields corresponding to the ten main questions/stages.
+- **Implementation**:
+  - Define a comprehensive schema for the SQL database.
+  - Use an Object-Relational Mapping (ORM) tool like SQLAlchemy for efficient database interactions.
+
+#### Dynamic Editing
+
+- **User Modifications**: Allow users to edit previous inputs through the UI, updating the corresponding text fields.
+- **Real-Time Updates**: Reflect changes immediately in the database to maintain data consistency.
+- **Implementation**:
+  - Provide editable text fields beneath each stage button.
+  - Implement functions to update the database upon user edits, ensuring that the refined data remains accurate and up-to-date.
+
+---
+
+## Implementation Details
+
+### Python and Gradio Usage
+
+- **Framework**: Utilize Gradio for the UI, providing a web-based interface with stage buttons and corresponding text fields.
+- **Libraries**:
+  - **LLM Interaction**: Use `huggingface_hub` for LLM API calls.
+  - **Data Models**: Use `pydantic` for data validation and modeling.
+  - **Web Search**: Use `duckduckgo_search` API for web searches.
+  - **Environment Variables**: Use `python-dotenv` for managing API keys and configuration.
+
+### Module and Function Organization
+
+- **Main Modules**:
+  - `main.py`: Entry point of the application.
+  - `chatbot.py`: Contains the `InnovativeIdeaChatbot` class and related functions.
+  - `data_models.py`: Defines data models using Pydantic.
+  - `utils.py`: Utility functions for parsing, web search, etc.
+  - `config.py`: Configuration settings and constants.
+- **Function Structure**:
+  - Each function should have clear input/output parameters.
+  - Include docstrings with type hints for clarity and maintainability.
+  - Ensure that each stage's function handles only its specific tasks to maintain modularity.
+
+### Error Handling and Validation
+
+- **User Input Validation**: Check for empty, invalid, or malformed inputs before processing.
+- **API Error Handling**: Catch exceptions from API calls and provide user-friendly error messages or prompts.
+- **Data Validation**: Use Pydantic models to validate data before storage to ensure data integrity.
+- **Implementation**:
+  - Utilize try-except blocks around critical sections, especially around API interactions.
+  - Provide default values or prompts to guide users in case of errors or invalid inputs.
+  - Implement validation checks before updating the database with refined data.
+
+### LLM Context and Memory Management
+
+- **Conversation Length**: Manage the size of the conversation history to stay within LLM token limits, ensuring efficient processing.
+- **Contextual Responses**: Ensure the LLM receives sufficient context relevant to the current stage without overwhelming it with unnecessary information.
+- **Implementation**:
+  - Implement a sliding window mechanism to include only recent and relevant messages in the LLM prompt.
+  - Summarize older parts of the conversation if necessary to maintain context within token limits.
+  - Ensure that each stage's context is isolated to prevent cross-stage contamination.
+
+### User Interface Enhancements
+
+- **Stage Buttons**: Design intuitive and clearly labeled buttons for each of the ten stages, allowing easy navigation.
+- **Text Fields**: Place editable text fields beneath each stage button to display and allow modifications of the refined data.
+- **'Fill Out Form' Button**: Include a dedicated button for users to populate the corresponding text field with the refined data after completing a stage.
+- **Visual Indicators**: Implement visual cues (e.g., color changes, checkmarks) to indicate the completion status of each stage.
+- **Responsive Design**: Ensure the UI is responsive and user-friendly across different devices and screen sizes.
+- **Implementation**:
+  - Utilize Gradio's layout features to arrange buttons and text fields logically.
+  - Incorporate CSS styling within Gradio to enhance the visual appeal and usability of the interface.
+  - Ensure that the UI updates dynamically based on user interactions and stage completions.
+
+### API Key Management
+
+- **Secure Input**: Allow users to input their Hugging Face API keys securely through the UI.
+- **Environment Variables**: Store API keys using environment variables to prevent exposure and enhance security.
+- **Implementation**:
+  - Provide a secure input field in the UI for users to enter their API keys.
+  - Utilize `python-dotenv` to manage and access API keys within the application without hardcoding them.
+  - Ensure that API keys are never exposed in logs or error messages.
+
+---
+
+# Appendices
+
+## Appendix A: Data Model Definition
+
+### InnovativeIdeaForm (Pydantic BaseModel)
+
+Defines the structure for storing refined user inputs.
+
+```python
+from pydantic import BaseModel, Field
+from typing import List, Optional
+
+class InnovativeIdeaForm(BaseModel):
+    idea_name: Optional[str] = Field(None, description="Name of the innovative idea")
+    idea_overview: Optional[str] = Field(None, description="Simple overview of the idea")
+    problem_solution: Optional[str] = Field(None, description="Customer problem and how the idea addresses it")
+    target_customers: Optional[str] = Field(None, description="Potential customers and markets")
+    market_value: Optional[str] = Field(None, description="Estimated market value")
+    existing_solutions: Optional[str] = Field(None, description="Similar existing ideas or products")
+    unique_value: Optional[str] = Field(None, description="How the idea is better or different")
+    technical_challenges: Optional[str] = Field(None, description="Difficult technical problems")
+    legal_barriers: Optional[str] = Field(None, description="Possible legal issues")
+    data_dependencies: Optional[str] = Field(None, description="Required data types")
+    team_roles: Optional[List[str]] = Field(None, description="Required job roles")
+    timeline: Optional[str] = Field(None, description="Estimated time to build main components")
+    additional_info: Optional[str] = Field(None, description="Any additional information")
+```
+
+## Appendix B: Function Signatures
+
+### Chatbot Class Methods
+
+```python
+from typing import List, Dict, Tuple, Optional, Union
+
+class InnovativeIdeaChatbot:
+    def __init__(self):
+        """
+        Initializes the chatbot with necessary configurations and data storage.
+        """
+        pass
+
+    def get_initial_greeting(self) -> str:
+        """
+        Returns the initial greeting message to the user.
+        """
+        pass
+
+    def activate_stage(self, stage_name: str) -> Optional[str]:
+        """
+        Activates the specified stage and returns the prompt for that stage.
+
+        Args:
+            stage_name (str): The name of the stage to activate.
+
+        Returns:
+            Optional[str]: The prompt message for the activated stage.
+        """
+        pass
+
+    def process_stage_input(self, stage_name: str, message: str) -> Tuple[str, Optional[str]]:
+        """
+        Processes the user's input for a specific stage and returns the refined data.
+
+        Args:
+            stage_name (str): The name of the current stage.
+            message (str): The user's input message.
+
+        Returns:
+            Tuple[str, Optional[str]]: A tuple containing the refined data and an optional follow-up prompt.
+        """
+        pass
+
+    def fill_out_form(self, stage_name: str) -> Optional[str]:
+        """
+        Populates the corresponding text field with refined data for the specified stage.
+
+        Args:
+            stage_name (str): The name of the stage to fill out.
+
+        Returns:
+            Optional[str]: The refined data to be displayed in the text field.
+        """
+        pass
+
+    def update_idea_form(self, stage_name: str, form_data: str):
+        """
+        Updates the idea form with the provided refined data for a specific stage.
+
+        Args:
+            stage_name (str): The name of the stage being updated.
+            form_data (str): The refined data to update in the form.
+        """
+        pass
+
+    def reset(self):
+        """
+        Resets the chatbot state for a new session, clearing all stored data and history.
+        """
+        pass
+```
+
+## Appendix C: Sample System Prompt Template
+
+```text
+You are an AI assistant named Myamoto, designed to help users refine their innovative ideas. Your task is to guide them through the ideation process, stage by stage, asking relevant questions and providing insightful feedback at each stage. Be encouraging, creative, and analytical in your responses. Remember to:
+
+1. Analyze the user's input carefully and extract relevant information for the current stage.
+2. Provide constructive feedback and suggestions to improve the idea.
+3. Ask specific, targeted follow-up questions to gather more details if needed.
+4. Offer creative ideas or alternatives to enhance the user's concept.
+5. Maintain a supportive and enthusiastic tone throughout the conversation.
+6. When you feel the current stage has been thoroughly explored, explicitly state "stage complete" or "ready to move on" in your response.
+7. Break down your thinking process into steps, enclosed in <step></step> tags.
+8. Provide a reflection on your thought process, enclosed in <reflection></reflection> tags.
+9. For each stage, provide a refined, clear, and concise version of the user's input. This refined version should capture the essence of the user's idea while improving its clarity and coherence.
+10. If you think additional information from a web search might be helpful, suggest it to the user. Ask them to use '@' followed by their search query to perform a web search.
+11. Always include a <form_data></form_data> section in your response with the extracted information for the current stage.
+12. Be explicit when a stage is complete and indicate that you're ready to move to the next stage.
+
+Please format your response as follows:
+<answer>
+[Your main response here, including follow-up questions and web search suggestions if applicable]
+[If the stage is complete, include a clear statement like "This stage is now complete. We're ready to move on to the next stage."]
+</answer>
+
+<form_data>
+[Extracted information for the current stage in a structured format]
+</form_data>
+
+<reflection>
+[Your reflection on the thought process]
+</reflection>
+
+<step>[Step 1 of your thinking process]</step>
+<step>[Step 2 of your thinking process]</step>
+...
+
+Current stage: {{current_stage}}
+```

chatbot.py

@@ -0,0 +1,165 @@
+from typing import List, Tuple, Dict, Any, Optional
+from data_models import IdeaForm, IDEA_STAGES
+from config import DEFAULT_SYSTEM_PROMPT, STAGES
+from utils import (
+    get_llm_response, extract_form_data, save_idea_to_database,
+    load_idea_from_database, update_idea_in_database,
+    get_db, clear_database, init_db, create_tables,
+    perform_web_search, optimize_search_query,
+    SessionLocal, InnovativeIdea
+)
+
+class InnovativeIdeaChatbot:
+    def __init__(self):
+        create_tables()
+        init_db()
+        self.idea_form = IdeaForm()
+        self.chat_history = []
+        self.idea_id = None
+        self.db = SessionLocal()
+        self.current_stage = None
+        self.api_key = None
+        self.add_system_message(self.get_initial_greeting())
+
+    def __del__(self):
+        if hasattr(self, 'db'):
+            self.db.close()
+
+    def get_initial_greeting(self) -> str:
+        return "Welcome to the Innovative Idea Generator! Please select a stage to begin."
+
+    def add_system_message(self, message: str):
+        self.chat_history.append(("System", message))
+
+    def set_api_key(self, api_key: str):
+        self.api_key = api_key
+
+    def activate_stage(self, stage_name: str) -> Optional[str]:
+        self.current_stage = stage_name
+        for stage in STAGES:
+            if stage["name"] == stage_name:
+                return f"Let's work on the '{stage_name}' stage. {stage['question']}"
+        return None
+
+    def process_stage_input(self, stage_name: str, message: str, model: str, system_prompt: str, thinking_budget: int) -> Tuple[List[Tuple[str, str]], Dict[str, Any]]:
+        if self.current_stage != stage_name:
+            activation_message = self.activate_stage(stage_name)
+            if activation_message is None:
+                error_message = f"Error: Unable to activate stage '{stage_name}'. Please check if the stage name is correct."
+                self.chat_history.append(("System", error_message))
+                return self.chat_history, self.idea_form.dict()
+            self.chat_history.append(("System", activation_message))
+
+        # Check for web search request
+        if message.startswith('@'):
+            search_query = message[1:].strip()
+            optimized_query = optimize_search_query(search_query, model)
+            search_results = perform_web_search(optimized_query)
+            self.chat_history.append(("Human", message))
+            self.chat_history.append(("AI", f"Here are the search results for '{optimized_query}':\n\n{search_results}"))
+            return self.chat_history, self.idea_form.dict()
+
+        # Generate the prompt for the current stage
+        stage_prompt = self.generate_prompt_for_stage(stage_name)
+
+        # Combine the user's input with the stage prompt
+        combined_prompt = f"{stage_prompt}\n\nUser input: {message}"
+
+        # Get LLM response
+        llm_response = get_llm_response(combined_prompt, model, thinking_budget, self.api_key)
+
+        # Add the interaction to chat history
+        self.chat_history.append(("Human", message))
+        self.chat_history.append(("AI", llm_response))
+
+        # Extract form data from the LLM response
+        form_data = extract_form_data(llm_response)
+        
+        # Update the idea form
+        if stage_name in form_data:
+            setattr(self.idea_form, stage_name.lower().replace(" ", "_"), form_data[stage_name])
+
+        return self.chat_history, self.idea_form.dict()
+
+    def fill_out_form(self, current_stage: str, model: str, thinking_budget: int) -> Dict[str, str]:
+        form_data = {}
+        for stage in STAGES:
+            stage_name = stage["name"]
+            if stage_name == current_stage:
+                # Generate new data for the current stage
+                form_data[stage_name] = self.generate_form_data(stage_name, model, thinking_budget)
+            else:
+                # Use existing data for other stages
+                form_data[stage_name] = getattr(self.idea_form, stage["field"], "")
+        
+        # Update the idea form
+        for stage in STAGES:
+            setattr(self.idea_form, stage["field"], form_data[stage["name"]])
+        
+        # Save to database
+        if self.idea_id:
+            update_idea_in_database(self.idea_id, self.idea_form, self.db)
+        else:
+            self.idea_id = save_idea_to_database(self.idea_form, self.db)
+
+        return form_data
+
+    def generate_prompt_for_stage(self, stage: str) -> str:
+        for s in STAGES:
+            if s["name"] == stage:
+                prompt_text = s.get('question') or s.get('description', '')
+                return f"We are currently working on the '{stage}' stage. {prompt_text}"
+        return f"We are currently working on the '{stage}' stage. Please provide relevant information."
+
+    def reset(self):
+        self.chat_history = []
+        self.idea_form = IdeaForm()
+        self.idea_id = None
+        self.current_stage = None
+        self.add_system_message(self.get_initial_greeting())
+        clear_database(self.db)
+        return self.chat_history, self.idea_form.dict()
+
+    def update_idea_form(self, stage_name: str, form_data: str):
+        setattr(self.idea_form, stage_name.lower().replace(" ", "_"), form_data)
+        if self.idea_id:
+            update_idea_in_database(self.idea_id, self.idea_form, self.db)
+        else:
+            self.idea_id = save_idea_to_database(self.idea_form, self.db)
+
+    def generate_form_data(self, stage: str, model: str, thinking_budget: int) -> str:
+        # Prepare the conversation history for the LLM
+        conversation = "\n".join([f"{role}: {message}" for role, message in self.chat_history])
+        
+        prompt = f"""
+        Based on the following conversation, extract the relevant information for the '{stage}' stage of the innovative idea:
+
+        {conversation}
+
+        Please provide a concise summary for the '{stage}' stage, focusing only on the information relevant to this stage.
+        """
+
+        # Get LLM response
+        llm_response = get_llm_response(prompt, model, thinking_budget, self.api_key)
+
+        # Extract form data from the LLM response
+        form_data = extract_form_data(llm_response)
+        
+        return form_data.get(stage, "")
+
+    def start_over(self):
+        self.reset()
+        clear_database(self.db)
+        self.idea_id = None
+        
+        # Reinitialize with a new empty idea
+        new_idea = InnovativeIdea()
+        self.db.add(new_idea)
+        self.db.commit()
+        self.idea_id = new_idea.id
+        
+        self.idea_form = IdeaForm()
+        self.chat_history = []
+        self.add_system_message(self.get_initial_greeting())
+        
+        return self.chat_history, self.idea_form.dict()

config.py

@@ -0,0 +1,49 @@
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+API_TOKEN = os.getenv("HUGGINGFACE_API_TOKEN")
+
+MODELS = [
+    "NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO",
+    # Add other models here if needed
+]
+
+STAGES = [
+    {"name": "Idea Name", "field": "idea_name", "question": "Do you have a name for this idea?", "example": "Sharks with Laserbeams"},
+    {"name": "Idea Overview", "field": "idea_overview", "question": "Tell me about your idea, give a very simple overview as you might explain to younger child.", "example": "My idea is to attach laserbeams to sharks to create the ultimate evil weapon. The laserbeams are powered by a battery that are recharged by the shark's swimming motion. The laserbeams are used to attack enemy submarines."},
+    {"name": "Problem Solution", "field": "problem_solution", "question": "Tell me what Customer problem your idea addresses, and how does your idea address it?", "example": "Our customers' secret lairs are often attacked by submarines. If our clients had sharks with laserbeams they could attack and destroy the submarines, saving the secret lair from being discovered."},
+    {"name": "Target Customers", "field": "target_customers", "question": "What customers might be interested in this idea? Or is this a internal cost-savings / efficiency idea?", "example": "Evil geniuses, tyrannical dictators, rogue government agencies"},
+    {"name": "Market Value", "field": "market_value", "question": "What markets or industries is this idea for?", "example": "Defense Contractors, submarine manufacturers"},
+    {"name": "Existing Solutions", "field": "existing_solutions", "question": "What are some similar ideas or products that already exist today?", "example": "Nuclear Attack submarines and giant squids are somewhat similar to Shark with Laserbeams."},
+    {"name": "Unique Value", "field": "unique_value", "question": "How is your idea better or different?", "example": "Shark with Laserbeams are cheaper than Nuclear Attack submarines and the lasers are more powerful than squid tentacles."},
+    {"name": "Technical Challenges", "field": "technical_challenges", "question": "Are there any difficult technical problems with making this idea happen?", "example": "We would need to design a waterproof housing for the laser and batteries. We might need to experiment with how powerful lasers need to be when underwater in order to melt submarine steel hulls. We need to explore how to capture sharks and install the waterproof harnesses for the lasers."},
+    {"name": "Legal Barriers", "field": "legal_barriers", "question": "Any legal barriers?", "example": "We will need to obtain a weapons export license if we want to sell outside the United States."},
+    {"name": "Data Dependencies", "field": "data_dependencies", "question": "Any data dependencies?", "example": "We might need to get data on underwater motion vectors for creating the aiming mechanism for the lasers."},
+    {"name": "Team Roles", "field": "team_roles", "question": "What kind of people / job roles do you need to make this idea happen?", "example": "We will need marine biologists and fishermen for obtaining sharks. We will need electrical engineers, optical engineers and software engineers to create the laserbeam weapon. We will need project manager to help manage the budget and schedule. And we need a QA tester to ensure the laser is working underwater."},
+    {"name": "Timeline", "field": "timeline", "question": "About how long will it take to build the main components of your idea?", "example": "It will take a 2-3weeks to capture a shark. It will take 2-3 weeks to modify the harness and calibrate the laserbeam. I estimate it will take 3 months to train the shark to attack submarines."},
+    {"name": "Additional Info", "field": "additional_info", "question": "Anything else you'd want to add to help us understand and value your idea?", "example": "We might consider adding optional racing stripes or bioluminescent patterns ot the sharks as an upgrade with charge the customer for. We could explore a subscription model where we send the customers a new shark each quarter to replace sharks that age out of the program."}
+]
+
+DEFAULT_SYSTEM_PROMPT = """
+You are Myamoto, an AI assistant designed to guide users through the innovative idea development process. Your task is to help refine and expand their ideas, focusing strictly on one stage at a time. Be encouraging, creative, and analytical in your responses. Remember to:
+
+1. Analyze the user's input carefully and extract relevant information for the current stage only.
+2. Provide constructive feedback and suggestions to improve the idea, specific to the current stage.
+3. If the user's response is insufficient, ask follow-up questions to gather more details.
+4. Do not move to the next stage automatically. The user must explicitly indicate when they want to move on.
+5. Maintain a supportive and enthusiastic tone throughout the conversation.
+6. Always include a <form_data></form_data> section in your response with the extracted information for the current stage.
+7. If the user tries to skip ahead or provide information for future stages, gently redirect them to the current stage.
+8. Use analogies, examples, or thought-provoking questions to help users think more deeply about their ideas.
+9. When you feel the current stage has been thoroughly explored, explicitly state "stage complete" or "ready to move on" in your response.
+10. Break down your thinking process into steps, enclosed in <step></step> tags.
+11. Provide a reflection on your thought process, enclosed in <reflection></reflection> tags.
+12. If you think additional information from a web search might be helpful, suggest it to the user. Ask them to use '@' followed by their search query to perform a web search.
+
+Current stage: {current_stage}
+Stage prompt: {stage_prompt}
+"""
+
+# Other configuration variables can be added here

data_models.py

@@ -0,0 +1,61 @@
+from pydantic import BaseModel, Field
+from typing import List, Optional
+from sqlalchemy import Column, Integer, String, Text
+from sqlalchemy.ext.declarative import declarative_base
+
+Base = declarative_base()
+
+class IdeaStage(BaseModel):
+    name: str
+    field: str
+    question: str
+    example: str
+
+class IdeaForm(BaseModel):
+    idea_name: Optional[str] = Field(None, description="Name of the innovative idea")
+    idea_overview: Optional[str] = Field(None, description="Simple overview of the idea")
+    problem_solution: Optional[str] = Field(None, description="Customer problem and how the idea addresses it")
+    target_customers: Optional[str] = Field(None, description="Potential customers and markets")
+    market_value: Optional[str] = Field(None, description="Estimated market value")
+    existing_solutions: Optional[str] = Field(None, description="Similar existing ideas or products")
+    unique_value: Optional[str] = Field(None, description="How the idea is better or different")
+    technical_challenges: Optional[str] = Field(None, description="Difficult technical problems")
+    legal_barriers: Optional[str] = Field(None, description="Possible legal issues")
+    data_dependencies: Optional[str] = Field(None, description="Required data types")
+    team_roles: Optional[List[str]] = Field(None, description="Required job roles")
+    timeline: Optional[str] = Field(None, description="Estimated time to build main components")
+    additional_info: Optional[str] = Field(None, description="Any additional information")
+
+class InnovativeIdea(Base):
+    __tablename__ = 'innovative_ideas'
+
+    id = Column(Integer, primary_key=True, index=True)
+    idea_name = Column(String, index=True)
+    idea_overview = Column(Text)
+    problem_solution = Column(Text)
+    target_customers = Column(Text)
+    market_value = Column(String)
+    existing_solutions = Column(Text)
+    unique_value = Column(Text)
+    technical_challenges = Column(Text)
+    legal_barriers = Column(Text)
+    data_dependencies = Column(Text)
+    team_roles = Column(Text)
+    timeline = Column(Text)
+    additional_info = Column(Text)
+
+IDEA_STAGES = [
+    IdeaStage(name="Idea Name", field="idea_name", question="Do you have a name for this idea?", example="Sharks with Laserbeams"),
+    IdeaStage(name="Idea Overview", field="idea_overview", question="Tell me about your idea, give a very simple overview as you might explain to younger child.", example="My idea is to attach laserbeams to sharks to create the ultimate evil weapon. The laserbeams are powered by a battery that are recharged by the shark's swimming motion. The laserbeams are used to attack enemy submarines."),
+    IdeaStage(name="Problem Solution", field="problem_solution", question="Tell me what Customer problem your idea addresses, and how does your idea address it?", example="Our customers' secret lairs are often attacked by submarines. If our clients had sharks with laserbeams they could attack and destroy the submarines, saving the secret lair from being discovered."),
+    IdeaStage(name="Target Customers", field="target_customers", question="What customers might be interested in this idea? Or is this a internal cost-savings / efficiency idea?", example="Evil geniuses, tyrannical dictators, rogue government agencies"),
+    IdeaStage(name="Market Value", field="market_value", question="What markets or industries is this idea for?", example="Defense Contractors, submarine manufacturers"),
+    IdeaStage(name="Existing Solutions", field="existing_solutions", question="What are some similar ideas or products that already exist today?", example="Nuclear Attack submarines and giant squids are somewhat similar to Shark with Laserbeams."),
+    IdeaStage(name="Unique Value", field="unique_value", question="How is your idea better or different?", example="Shark with Laserbeams are cheaper than Nuclear Attack submarines and the lasers are more powerful than squid tentacles."),
+    IdeaStage(name="Technical Challenges", field="technical_challenges", question="Are there any difficult technical problems with making this idea happen?", example="We would need to design a waterproof housing for the laser and batteries. We might need to experiment with how powerful lasers need to be when underwater in order to melt submarine steel hulls. We need to explore how to capture sharks and install the waterproof harnesses for the lasers."),
+    IdeaStage(name="Legal Barriers", field="legal_barriers", question="Any legal barriers?", example="We will need to obtain a weapons export license if we want to sell outside the United States."),
+    IdeaStage(name="Data Dependencies", field="data_dependencies", question="Any data dependencies?", example="We might need to get data on underwater motion vectors for creating the aiming mechanism for the lasers."),
+    IdeaStage(name="Team Roles", field="team_roles", question="What kind of people / job roles do you need to make this idea happen?", example="We will need marine biologists and fishermen for obtaining sharks. We will need electrical engineers, optical engineers and software engineers to create the laserbeam weapon. We will need project manager to help manage the budget and schedule. And we need a QA tester to ensure the laser is working underwater."),
+    IdeaStage(name="Timeline", field="timeline", question="About how long will it take to build the main components of your idea?", example="It will take a 2-3weeks to capture a shark. It will take 2-3 weeks to modify the harness and calibrate the laserbeam. I estimate it will take 3 months to train the shark to attack submarines."),
+    IdeaStage(name="Additional Info", field="additional_info", question="Anything else you'd want to add to help us understand and value your idea?", example="We might consider adding optional racing stripes or bioluminescent patterns ot the sharks as an upgrade with charge the customer for. We could explore a subscription model where we send the customers a new shark each quarter to replace sharks that age out of the program.")
+]

main.py

@@ -0,0 +1,194 @@
+import gradio as gr
+from utils import SessionLocal, InnovativeIdea, ensure_database_exists, verify_database, initialize_database
+from data_models import IdeaForm
+from chatbot import InnovativeIdeaChatbot
+from config import MODELS, DEFAULT_SYSTEM_PROMPT, STAGES
+import logging
+from typing import Dict, Any
+
+# Set up logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+
+def create_gradio_interface():
+    innovative_chatbot = InnovativeIdeaChatbot()
+    default_stage = STAGES[0]["name"]
+
+    # Load initial data from the database
+    db = SessionLocal()
+    initial_idea = db.query(InnovativeIdea).first()
+    db.close()
+
+    if initial_idea is None:
+        logging.warning("No initial idea found in the database. Creating a new one.")
+        initial_idea = InnovativeIdea()
+
+    def chatbot_function(message, history, model, system_prompt, thinking_budget, current_stage):
+        try:
+            chat_history, form_data = innovative_chatbot.process_stage_input(current_stage, message, model, system_prompt, thinking_budget)
+            
+            history.append((message, chat_history[-1][1]))
+            
+            # Update the database with the new form data
+            db = SessionLocal()
+            idea = db.query(InnovativeIdea).first()
+            for key, value in form_data.items():
+                setattr(idea, key, value)
+            db.commit()
+            db.close()
+            
+            return history, form_data.get(current_stage, ""), ""  # Return an empty string to clear the input
+        except Exception as e:
+            logging.error(f"An error occurred in chatbot_function: {str(e)}", exc_info=True)
+            return history + [(None, f"An error occurred: {str(e)}")], "", ""  # Return an empty string to clear the input
+
+    def fill_form(stage, model, thinking_budget):
+        form_data = innovative_chatbot.fill_out_form(stage, model, thinking_budget)
+        return form_data
+
+    def clear_chat():
+        # Reset the database to an empty form
+        db = SessionLocal()
+        idea = db.query(InnovativeIdea).first()
+        empty_form = IdeaForm()
+        for key, value in empty_form.dict().items():
+            setattr(idea, key, value)
+        db.commit()
+        db.close()
+        
+        return innovative_chatbot.reset()
+
+    def update_form_field(stage_name: str, value: str) -> str:
+        db = SessionLocal()
+        idea = db.query(InnovativeIdea).first()
+        field_name = next(stage["field"] for stage in STAGES if stage["name"] == stage_name)
+        setattr(idea, field_name, value)
+        db.commit()
+        db.close()
+        return value
+
+    def update_placeholders(form_data: Dict[str, str]) -> Dict[str, Any]:
+        return {
+            stage["name"]: gr.update(value=form_data.get(stage["name"], ""), placeholder=form_data.get(stage["name"], stage["example"]))
+            for stage in STAGES
+        }
+
+    def start_over():
+        chat_history, form_data = innovative_chatbot.start_over()
+        return (
+            [],  # Clear the chatbot
+            "",  # Clear the message input
+            *[form_data.get(stage["field"], "") for stage in STAGES],  # Reset all form fields
+            gr.update(value=default_stage)  # Reset the stage selection
+        )
+
+    with gr.Blocks(theme=gr.themes.Soft()) as demo:
+        gr.Markdown("# Innovative Idea Generator")
+        
+        mode = gr.Radio(["Chatbot", "Direct Input"], label="Mode", value="Chatbot")
+        
+        with gr.Row():
+            with gr.Column(scale=2):
+                chatbot = gr.Chatbot(label="Conversation", height=500)
+                msg = gr.Textbox(label="Your input", placeholder="Type your brilliant idea here...")
+                
+                with gr.Row():
+                    submit = gr.Button("Submit")
+                    clear = gr.Button("Clear Chat")
+                    start_over_btn = gr.Button("Start Over")
+
+            with gr.Column(scale=1):
+                stages = gr.Radio(
+                    choices=[stage["name"] for stage in STAGES],
+                    label="Ideation Stages",
+                    value=default_stage
+                )
+                form_fields = {
+                    stage["name"]: gr.Textbox(
+                        label=stage["question"],
+                        placeholder=stage["example"],
+                        value=getattr(initial_idea, stage["field"], ""),
+                        visible=(stage["name"] == default_stage),
+                        interactive=False
+                    ) for stage in STAGES
+                }
+                fill_form_btn = gr.Button("Fill out Form")
+                submit_form_btn = gr.Button("Submit Form", visible=False)
+
+        with gr.Accordion("Advanced Settings", open=False):
+            model = gr.Dropdown(choices=MODELS, label="Select Model", value=MODELS[0])
+            system_prompt = gr.Textbox(label="System Prompt", value=DEFAULT_SYSTEM_PROMPT, lines=5)
+            thinking_budget = gr.Slider(minimum=1, maximum=4098, value=2048, step=1, label="Max New Tokens")
+            api_key = gr.Textbox(label="Hugging Face API Key", type="password")
+
+        # Event handlers
+        msg.submit(chatbot_function, 
+                   inputs=[msg, chatbot, model, system_prompt, thinking_budget, stages],
+                   outputs=[chatbot, form_fields[default_stage], msg])
+        
+        submit.click(chatbot_function, 
+                     inputs=[msg, chatbot, model, system_prompt, thinking_budget, stages],
+                     outputs=[chatbot, form_fields[default_stage], msg])
+        
+        def fill_and_update_form(stage, model, thinking_budget):
+            form_data = fill_form(stage, model, thinking_budget)
+            updated_fields = {}
+            for stage_name, value in form_data.items():
+                updated_value = update_form_field(stage_name, value)
+                updated_fields[stage_name] = updated_value
+            return update_placeholders(updated_fields)
+
+        fill_form_btn.click(fill_and_update_form,
+                            inputs=[stages, model, thinking_budget],
+                            outputs=list(form_fields.values()))
+        
+        clear.click(clear_chat,
+                    outputs=[chatbot] + list(form_fields.values()))
+
+        # Update form field visibility based on selected stage
+        stages.change(
+            lambda s: [gr.update(visible=(stage["name"] == s)) for stage in STAGES],
+            inputs=[stages],
+            outputs=list(form_fields.values())
+        )
+
+        # Update API key when changed
+        api_key.change(innovative_chatbot.set_api_key, inputs=[api_key])
+
+        # Toggle between chatbot and direct input mode
+        def toggle_mode(new_mode):
+            if new_mode == "Direct Input":
+                return [gr.update(visible=False)] * 3 + [gr.update(interactive=True)] * len(STAGES) + [gr.update(visible=True)]
+            else:
+                return [gr.update(visible=True)] * 3 + [gr.update(interactive=False)] * len(STAGES) + [gr.update(visible=False)]
+
+        mode.change(
+            toggle_mode,
+            inputs=[mode],
+            outputs=[chatbot, msg, submit] + list(form_fields.values()) + [submit_form_btn]
+        )
+
+        # Handle direct form submission
+        submit_form_btn.click(
+            lambda *values: values,
+            inputs=[form_fields[stage["name"]] for stage in STAGES],
+            outputs=[form_fields[stage["name"]] for stage in STAGES]
+        )
+
+        # Add this new event handler for the Start Over button
+        start_over_btn.click(
+            start_over,
+            outputs=[chatbot, msg] + [form_fields[stage["name"]] for stage in STAGES] + [stages]
+        )
+
+    return demo
+
+def main():
+    ensure_database_exists()
+    initialize_database()
+    verify_database()
+    demo = create_gradio_interface()
+    return demo 
+
+if __name__ == "__main__":
+    demo = main()
+    demo.launch()

requirements.txt

@@ -0,0 +1,6 @@
+gradio
+pydantic
+huggingface_hub
+duckduckgo_search
+python-dotenv
+sqlalchemy

utils.py

@@ -0,0 +1,456 @@
+import re
+from typing import Dict, List, Optional, Any
+from pydantic import ValidationError
+from data_models import IdeaForm, IDEA_STAGES, InnovativeIdea, Base
+from config import API_TOKEN, STAGES 
+from huggingface_hub import InferenceClient
+from duckduckgo_search import DDGS
+from sqlalchemy import create_engine, delete, inspect
+from sqlalchemy.orm import sessionmaker, Session, declarative_base
+import sys
+import sqlite3
+from sqlalchemy.exc import OperationalError, PendingRollbackError
+import logging
+import os
+
+# Database configuration
+DATABASE_URL = "sqlite:///./innovative_ideas.db"
+engine = create_engine(DATABASE_URL)
+SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
+Base = declarative_base()
+
+def get_llm_response(context: str, model: str, max_tokens: int, api_key: Optional[str] = None) -> str:
+    client = InferenceClient(model=model, token=api_key or API_TOKEN)
+    
+    messages = [
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": context}
+    ]
+    
+    full_response = ""
+    for message in client.chat_completion(
+        messages=messages,
+        max_tokens=max_tokens,
+        temperature=0.7,
+        top_p=0.95,
+        stream=True,
+    ):
+        content = message.choices[0].delta.content
+        if content:
+            full_response += content
+            print(content, end="", flush=True)
+    
+    print()  # New line after the full response
+    return full_response
+
+def extract_form_data(llm_response: str) -> Dict[str, Any]:
+    form_data = {}
+    
+    # Define patterns for each field
+    patterns = {stage["field"]: rf'{stage["name"]}:?\s*(.*?)(?:\n\n|\Z)' for stage in STAGES}
+    
+    # Extract data for each field
+    for field, pattern in patterns.items():
+        match = re.search(pattern, llm_response, re.DOTALL | re.IGNORECASE)
+        if match:
+            form_data[field] = match.group(1).strip()
+    
+    # Special handling for team_roles (convert to list)
+    if 'team_roles' in form_data:
+        form_data['team_roles'] = [role.strip() for role in form_data['team_roles'].split(',')]
+    
+    return form_data
+
+def perform_web_search(query: str, max_results: int = 3) -> str:
+    with DDGS() as ddgs:
+        results = list(ddgs.text(query, max_results=max_results))
+    
+    formatted_results = ""
+    for i, result in enumerate(results, 1):
+        formatted_results += f"{i}. {result['title']}\n   {result['body']}\n   URL: {result['href']}\n\n"
+    
+    return formatted_results
+
+def optimize_search_query(query: str, model: str) -> str:
+    client = InferenceClient(model=model, token=API_TOKEN)
+    prompt = f"Optimize the following search query to get better web search results:\n\n{query}\n\nOptimized query:"
+    
+    response = client.text_generation(prompt, max_new_tokens=50, temperature=0.7)
+    return response.strip()
+
+def refine_user_input(user_input: str, current_stage: str, model: str) -> str:
+    client = InferenceClient(model=model, token=API_TOKEN)
+    prompt = f"""
+    Refine and clarify the following user input for the '{current_stage}' stage of an innovative idea:
+    
+    {user_input}
+    
+    Please provide a clear, concise, and well-structured version of this input.
+    Ensure that the refined input is specific to the current stage and does not include information for other stages.
+    """
+    messages = [
+        {"role": "system", "content": "You are a helpful AI assistant specializing in refining innovative ideas."},
+        {"role": "user", "content": prompt}
+    ]
+    
+    full_response = ""
+    for message in client.chat_completion(
+        messages=messages,
+        max_tokens=200,
+        temperature=0.7,
+        stream=True,
+    ):
+        content = message.choices[0].delta.content
+        if content:
+            full_response += content
+    
+    return full_response.strip()
+
+def update_idea_form(form: IdeaForm, new_data: Dict) -> Optional[IdeaForm]:
+    """
+    Updates the IdeaForm with new data and validates it.
+    """
+    try:
+        updated_data = form.dict()
+        updated_data.update(new_data)
+        return IdeaForm(**updated_data)
+    except ValidationError as e:
+        print(f"Validation error: {e}")
+        return None
+
+def update_checklist(idea_form: IdeaForm) -> str:
+    """
+    Generates a checklist based on the current state of the idea form.
+    """
+    checklist = []
+    for field, value in idea_form.dict().items():
+        if not value:
+            checklist.append(f"[ ] Complete the '{field.replace('_', ' ').title()}' section")
+        elif isinstance(value, str) and len(value) < 50:
+            checklist.append(f"[?] Consider expanding the '{field.replace('_', ' ').title()}' section")
+        else:
+            checklist.append(f"[x] '{field.replace('_', ' ').title()}' section completed")
+    
+    return "\n".join(checklist)
+
+def generate_follow_up_questions(idea_form: IdeaForm, current_stage: int) -> List[str]:
+    """
+    Generates follow-up questions based on the current stage and form state.
+    """
+    questions = []
+    current_stage_name = IDEA_STAGES[current_stage]
+    form_dict = idea_form.dict()
+
+    if current_stage_name == "Idea Name":
+        if not form_dict['idea_name']:
+            questions.append("What would be a catchy and memorable name for your idea?")
+        elif len(form_dict['idea_name']) > 50:
+            questions.append("Can you shorten your idea name to make it more concise?")
+
+    elif current_stage_name == "Idea Overview":
+        if not form_dict['idea_overview']:
+            questions.append("Can you provide a brief overview of your idea in simple terms?")
+        elif len(form_dict['idea_overview']) < 100:
+            questions.append("Can you elaborate more on how your idea works?")
+
+    elif current_stage_name == "Problem Solution":
+        if not form_dict['problem_solution']:
+            questions.append("What specific problem does your idea solve?")
+        if 'how' not in form_dict['problem_solution'].lower():
+            questions.append("How exactly does your idea solve the problem you've identified?")
+
+    elif current_stage_name == "Target Customers":
+        if not form_dict['target_customers']:
+            questions.append("Who are the main users or customers for your idea?")
+        elif len(form_dict['target_customers']) < 50:
+            questions.append("Can you be more specific about your target customers? Consider demographics, industries, or use cases.")
+
+    elif current_stage_name == "Market Value":
+        if not form_dict['market_value']:
+            questions.append("What industries or markets could benefit from your idea?")
+        if 'size' not in form_dict['market_value'].lower():
+            questions.append("Can you estimate the size of the market for your idea?")
+
+    elif current_stage_name == "Existing Solutions":
+        if not form_dict['existing_solutions']:
+            questions.append("Are there any existing products or services similar to your idea?")
+        elif 'different' not in form_dict['existing_solutions'].lower():
+            questions.append("How is your idea different from these existing solutions?")
+
+    elif current_stage_name == "Unique Value":
+        if not form_dict['unique_value']:
+            questions.append("What makes your idea unique or better than existing solutions?")
+        if 'why' not in form_dict['unique_value'].lower():
+            questions.append("Why would customers choose your solution over alternatives?")
+
+    elif current_stage_name == "Technical Challenges":
+        if not form_dict['technical_challenges']:
+            questions.append("What are the main technical challenges in developing your idea?")
+        if 'overcome' not in form_dict['technical_challenges'].lower():
+            questions.append("How might these technical challenges be overcome?")
+
+    elif current_stage_name == "Legal Barriers":
+        if not form_dict['legal_barriers']:
+            questions.append("Are there any legal or regulatory issues that could affect your idea?")
+        if 'address' not in form_dict['legal_barriers'].lower():
+            questions.append("How might you address these legal or regulatory issues?")
+
+    elif current_stage_name == "Data Dependencies":
+        if not form_dict['data_dependencies']:
+            questions.append("Does your idea rely on any specific data sources?")
+        if 'acquire' not in form_dict['data_dependencies'].lower():
+            questions.append("How will you acquire or generate the necessary data for your idea?")
+
+    elif current_stage_name == "Team Roles":
+        if not form_dict['team_roles']:
+            questions.append("What key roles or expertise are needed to develop and implement your idea?")
+        elif len(form_dict['team_roles']) < 3:
+            questions.append("Are there any additional roles or skills that would be beneficial for your team?")
+
+    elif current_stage_name == "Timeline":
+        if not form_dict['timeline']:
+            questions.append("Can you provide a rough timeline for developing your idea?")
+        if 'milestone' not in form_dict['timeline'].lower():
+            questions.append("What are the main milestones in your development timeline?")
+
+    elif current_stage_name == "Additional Info":
+        if not form_dict['additional_info']:
+            questions.append("Is there any additional information that's important for understanding or implementing your idea?")
+        elif len(form_dict['additional_info']) < 100:
+            questions.append("Are there any other aspects of your idea that you'd like to elaborate on?")
+
+    return questions
+
+def save_idea_to_database(form: IdeaForm, db: Session) -> int:
+    """
+    Saves the IdeaForm to the SQLite database.
+    Returns the ID of the inserted row.
+    """
+    db_idea = InnovativeIdea(
+        idea_name=form.idea_name,
+        idea_overview=form.idea_overview,
+        problem_solution=form.problem_solution,
+        target_customers=form.target_customers,
+        market_value=form.market_value,
+        existing_solutions=form.existing_solutions,
+        unique_value=form.unique_value,
+        technical_challenges=form.technical_challenges,
+        legal_barriers=form.legal_barriers,
+        data_dependencies=form.data_dependencies,
+        team_roles=','.join(form.team_roles) if form.team_roles else None,
+        timeline=form.timeline,
+        additional_info=form.additional_info
+    )
+    db.add(db_idea)
+    db.commit()
+    db.refresh(db_idea)
+    return db_idea.id
+
+def load_idea_from_database(idea_id: int, db: Session) -> Optional[IdeaForm]:
+    """
+    Loads an IdeaForm from the SQLite database by ID.
+    """
+    db_idea = db.query(InnovativeIdea).filter(InnovativeIdea.id == idea_id).first()
+    if db_idea:
+        return IdeaForm(
+            idea_name=db_idea.idea_name,
+            idea_overview=db_idea.idea_overview,
+            problem_solution=db_idea.problem_solution,
+            target_customers=db_idea.target_customers,
+            market_value=db_idea.market_value,
+            existing_solutions=db_idea.existing_solutions,
+            unique_value=db_idea.unique_value,
+            technical_challenges=db_idea.technical_challenges,
+            legal_barriers=db_idea.legal_barriers,
+            data_dependencies=db_idea.data_dependencies,
+            team_roles=db_idea.team_roles.split(',') if db_idea.team_roles else None,
+            timeline=db_idea.timeline,
+            additional_info=db_idea.additional_info
+        )
+    return None
+
+def update_idea_in_database(idea_id: int, form: IdeaForm, db: Session) -> bool:
+    """
+    Updates an existing idea in the database.
+    """
+    db_idea = db.query(InnovativeIdea).filter(InnovativeIdea.id == idea_id).first()
+    if db_idea:
+        for field, value in form.dict(exclude_unset=True).items():
+            setattr(db_idea, field, value)
+        db.commit()
+        return True
+    return False
+
+def extract_search_query(message: str) -> Optional[str]:
+    """
+    Extract a search query from a user message if it starts with '@'.
+    """
+    if message.startswith('@'):
+        return message[1:].strip()
+    return None
+
+def create_tables():
+    Base.metadata.create_all(bind=engine)
+
+def get_db():
+    db = SessionLocal()
+    try:
+        yield db
+    finally:
+        db.close()
+
+def verify_database():
+    try:
+        inspector = inspect(engine)
+        if not inspector.has_table("innovative_ideas"):
+            logging.error("InnovativeIdea table does not exist")
+            return False
+        
+        columns = inspector.get_columns("innovative_ideas")
+        expected_columns = [field for field in IdeaForm.__fields__]
+        actual_columns = [column['name'] for column in columns if column['name'] != 'id']
+        
+        if set(expected_columns) != set(actual_columns):
+            logging.error(f"Mismatch in table columns. Expected: {expected_columns}, Actual: {actual_columns}")
+            return False
+        
+        # Check if there's at least one row in the table
+        db = SessionLocal()
+        idea_count = db.query(InnovativeIdea).count()
+        db.close()
+        
+        if idea_count == 0:
+            logging.warning("No ideas found in the database")
+            # Instead of returning False, we'll create an initial idea
+            db = SessionLocal()
+            initial_idea = InnovativeIdea()
+            db.add(initial_idea)
+            db.commit()
+            db.close()
+            logging.info("Created initial idea in the database")
+        
+        return True
+    except Exception as e:
+        logging.error(f"Database verification failed: {str(e)}", exc_info=True)
+        return False
+
+def initialize_database():
+    try:
+        # Create tables
+        Base.metadata.create_all(bind=engine)
+        logging.info("Database tables created successfully.")
+        
+        # Check if the table is empty
+        db = SessionLocal()
+        existing_idea = db.query(InnovativeIdea).first()
+        
+        if existing_idea is None:
+            # Create an empty IdeaForm
+            empty_form = IdeaForm()
+            
+            # Save the empty form to the database
+            db_idea = InnovativeIdea(**empty_form.dict())
+            db.add(db_idea)
+            db.commit()
+            db.refresh(db_idea)
+            logging.info("Empty idea form added to the database.")
+        else:
+            logging.info("Database already contains data.")
+        
+        db.close()
+        return True
+    except Exception as e:
+        logging.error(f"Failed to initialize database: {str(e)}")
+        return False
+
+def init_db():
+    Base.metadata.create_all(bind=engine)
+
+def get_example_idea() -> Dict[str, str]:
+    """
+    Returns an example idea to guide users and the LLM.
+    """
+    return {
+        "idea_name": "Sharks with Laserbeams",
+        "idea_overview": "My idea is to attach laserbeams to sharks to create the ultimate evil weapon. The laserbeams are powered by a battery that are recharged by the shark's swimming motion. The laserbeams are used to attack enemy submarines.",
+        "problem_solution": "Our customers' secret lairs are often attacked by submarines. If our clients had sharks with laserbeams they could attack and destroy the submarines, saving the secret lair from being discovered.",
+        "target_customers": "Evil geniuses, tyrannical dictators, rogue government agencies",
+        "market_value": "Defense Contractors, submarine manufacturers",
+        "existing_solutions": "Nuclear Attack submarines and giant squids are somewhat similar to Shark with Laserbeams.",
+        "unique_value": "Shark with Laserbeams are cheaper than Nuclear Attack submarines and the lasers are more powerful than squid tentacles.",
+        "technical_challenges": "We would need to design a waterproof housing for the laser and batteries. We might need to experiment with how powerful lasers need to be when underwater in order to melt submarine steel hulls. We need to explore how to capture sharks and install the waterproof harnesses for the lasers.",
+        "legal_barriers": "We will need to obtain a weapons export license if we want to sell outside the United States.",
+        "data_dependencies": "We might need to get data on underwater motion vectors for creating the aiming mechanism for the lasers.",
+        "team_roles": "Marine biologists, fishermen, electrical engineers, optical engineers, software engineers, project manager, QA tester",
+        "timeline": "It will take 2-3 weeks to capture a shark. It will take 2-3 weeks to modify the harness and calibrate the laserbeam. I estimate it will take 3 months to train the shark to attack submarines.",
+        "additional_info": "We might consider adding optional racing stripes or bioluminescent patterns to the sharks as an upgrade with charge the customer for. We could explore a subscription model where we send the customers a new shark each quarter to replace sharks that age out of the program."
+    }
+
+def generate_prompt_for_stage(stage: str, current_form: Dict[str, Any]) -> str:
+    """
+    Generates a prompt for the current stage, including the example idea.
+    """
+    example_idea = get_example_idea()
+    prompt = f"We are currently in the '{stage}' stage of developing an innovative idea. "
+    prompt += "Please provide information for this stage, keeping in mind the following example:\n\n"
+
+    if stage == "Idea Name":
+        prompt += f"Example: {example_idea['idea_name']}\n"
+        prompt += "What's the name of your innovative idea? Keep it short and catchy."
+    elif stage == "Idea Overview":
+        prompt += f"Example: {example_idea['idea_overview']}\n"
+        prompt += "Provide a simple overview of your idea, as if explaining it to a 10-year-old child. What does your idea do?"
+    elif stage == "Problem Solution":
+        prompt += f"Example: {example_idea['problem_solution']}\n"
+        prompt += "What specific problem does your idea solve? How does it solve this problem?"
+    elif stage == "Target Customers":
+        prompt += f"Example: {example_idea['target_customers']}\n"
+        prompt += "Who are the main customers or users for your idea? Be as specific as possible."
+    elif stage == "Market Value":
+        prompt += f"Example: {example_idea['market_value']}\n"
+        prompt += "What industries or markets could benefit from your idea? How large is this market?"
+    elif stage == "Existing Solutions":
+        prompt += f"Example: {example_idea['existing_solutions']}\n"
+        prompt += "Are there any existing products or services similar to your idea? How are they different?"
+    elif stage == "Unique Value":
+        prompt += f"Example: {example_idea['unique_value']}\n"
+        prompt += "What makes your idea unique or better than existing solutions? Why would customers choose your solution?"
+    elif stage == "Technical Challenges":
+        prompt += f"Example: {example_idea['technical_challenges']}\n"
+        prompt += "What are the main technical challenges in developing your idea? How might these be overcome?"
+    elif stage == "Legal Barriers":
+        prompt += f"Example: {example_idea['legal_barriers']}\n"
+        prompt += "Are there any legal or regulatory issues that could affect your idea? How might these be addressed?"
+    elif stage == "Data Dependencies":
+        prompt += f"Example: {example_idea['data_dependencies']}\n"
+        prompt += "Does your idea rely on any specific data sources? How will you acquire or generate this data?"
+    elif stage == "Team Roles":
+        prompt += f"Example: {example_idea['team_roles']}\n"
+        prompt += "What key roles or expertise are needed to develop and implement your idea? List specific job titles or skills."
+    elif stage == "Timeline":
+        prompt += f"Example: {example_idea['timeline']}\n"
+        prompt += "Provide a rough timeline for developing your idea. What are the main milestones and how long might each take?"
+    elif stage == "Additional Info":
+        prompt += f"Example: {example_idea['additional_info']}\n"
+        prompt += "Is there any additional information that's important for understanding or implementing your idea?"
+
+    prompt += "\n\nPlease provide your response for this stage, considering the example provided and the current state of your idea."
+    return prompt
+
+def ensure_database_exists():
+    db_path = DATABASE_URL.replace("sqlite:///", "")
+    if not os.path.exists(db_path):
+        logging.info(f"Database file not found. Creating new database at {db_path}")
+        open(db_path, 'a').close()  # Create an empty file
+    else:
+        logging.info(f"Database file found at {db_path}")
+
+def clear_database(db: Session):
+    try:
+        # Delete all records from the InnovativeIdea table
+        db.query(InnovativeIdea).delete()
+        db.commit()
+        logging.info("Database cleared successfully.")
+    except Exception as e:
+        db.rollback()
+        logging.error(f"Error clearing database: {str(e)}")