---
title: Faheem
emoji: 🔥
colorFrom: gray
colorTo: yellow
sdk: gradio
sdk_version: 5.18.0
app_file: app.py
pinned: false
short_description: This project uses AI to transcribe and summarize media conte
---

Check out the configuration reference at https://huggingface.co./docs/hub/spaces-config-reference

# AI-Powered Audio & Video to Text Conversion

## Project Description
This project enables users to convert audio and video content into text using AI. Additionally, it allows interaction with the extracted text through question-answering, summarization, and text-to-speech functionalities.

## Introduction
With the increasing amount of digital content, users often face challenges in accessing information from audio and video files efficiently. This project aims to facilitate information retrieval by leveraging AI to transcribe content into text, summarize key points, and provide an interactive Q&A system. Users can also listen to the summarized text if preferred.

## Project Objectives
1. **Enhance the speed of information retrieval** by enabling quick searches within extracted text.
2. **Provide an interactive experience** where users can ask questions about the content and receive instant answers.
3. **Support both Arabic and English languages** to accommodate a wider user base.
4. **Utilize AI to analyze and comprehend content**, allowing effective extraction and interaction with information.

## Features
- **Speech-to-Text**: Extracts text from audio and video files.
- **Text Summarization**: Generates concise summaries of extracted text.
- **Question Answering**: Answers user queries based on extracted text.
- **Text-to-Speech**: Converts text into speech for an auditory experience.
- **Multilingual Support**: Works in both Arabic and English.

## Technology Stack
### 1. Programming Language
- **Python**: The core programming language used for this project due to its extensive support for AI and data processing libraries.

### 2. Libraries & Frameworks
- **Gradio**: Used for creating an interactive web-based UI for users to upload files and interact with the extracted text.
- **Hugging Face Transformers**: Provides pre-trained models for automatic speech recognition, summarization, translation, and question-answering.
- **MoviePy**: Extracts audio from video files for further processing.
- **Librosa & SoundFile**: Handles audio processing, including loading, resampling, and segmenting audio clips.
- **gTTS (Google Text-to-Speech)**: Converts text into spoken words.
- **LangDetect**: Detects the language of the extracted text to provide appropriate processing.

## Code Breakdown
### 1. **Setting Up the Environment**
The script starts by checking if a GPU (CUDA) is available, which significantly speeds up AI model inference.
```python
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
```

### 2. **Loading AI Models**
Several AI models are loaded to handle different functionalities:
- **Whisper (Speech Recognition)**: Converts audio into text.
- **BART (Summarization)**: Generates concise summaries.
- **Helsinki-NLP (Translation)**: Translates text between Arabic and English.
- **BERT (Question Answering)**: Finds answers from extracted text.
```python
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="openai/whisper-medium", device=0 if device == "cuda" else -1)
bart_model = AutoModelForSeq2SeqLM.from_pretrained("ahmedabdo/arabic-summarizer-bart")
bart_tokenizer = AutoTokenizer.from_pretrained("ahmedabdo/arabic-summarizer-bart")
translate_ar_to_en = pipeline("translation", model="Helsinki-NLP/opus-mt-ar-en")
translate_en_to_ar = pipeline("translation", model="Helsinki-NLP/opus-mt-en-ar")
qa_pipeline = pipeline("question-answering", model="deepset/bert-base-cased-squad2", tokenizer="deepset/bert-base-cased-squad2")
```

### 3. **Audio Processing**
Handles both audio and video files. If a video is uploaded, it extracts the audio and converts it into a compatible format.
```python
from moviepy.video.io.VideoFileClip import VideoFileClip
import librosa
import soundfile as sf

def convert_audio_to_text(uploaded_file):
    if not uploaded_file:
        return "⛔ Please upload a file first"

    input_path = uploaded_file if isinstance(uploaded_file, str) else uploaded_file.name
    output_path = "/tmp/processed.wav"

    if input_path.split('.')[-1].lower() in ['mp4', 'avi', 'mov', 'mkv']:
        VideoFileClip(input_path).audio.write_audiofile(output_path, codec='pcm_s16le')
    else:
        output_path = input_path

    audio, rate = librosa.load(output_path, sr=16000)
    return pipe(output_path)["text"]
```

### 4. **Summarization**
Summarizes extracted text using the pre-trained BART model.
```python
def summarize_text(text):
    inputs = bart_tokenizer(text, return_tensors="pt", max_length=1024, truncation=True).to(device)
    summary_ids = bart_model.generate(inputs.input_ids, max_length=150, num_beams=4, early_stopping=True)
    return bart_tokenizer.decode(summary_ids[0], skip_special_tokens=True)
```

### 5. **Question Answering**
Users can input a question based on the extracted text, and the system finds the most relevant answer.
```python
def answer_question(text, question):
    translated_context = translate_ar_to_en(text)[0]['translation_text']
    translated_question = translate_ar_to_en(question)[0]['translation_text']
    results = qa_pipeline({'question': translated_question, 'context': translated_context}, top_k=3)
    best_result = max(results, key=lambda res: res['score'])
    return translate_en_to_ar(best_result['answer'])[0]['translation_text']
```

### 6. **Text-to-Speech**
Converts text into an audio file using gTTS.
```python
def text_to_speech(text):
    tts = gTTS(text=text, lang='ar' if detect(text) == 'ar' else 'en', slow=False)
    output = "/tmp/tts.wav"
    tts.save(output)
    return output
```

## Installation & Setup
### Prerequisites
- Python 3.8+
- Pip package manager
- GPU (optional but recommended for better performance)

### Installation Steps
1. Clone the repository:
   ```sh
   git clone https://github.com/your-repo/ai-audio-video-text.git
   cd ai-audio-video-text
   ```
2. Install dependencies:
   ```sh
   pip install -r requirements.txt
   ```
3. Run the application:
   ```sh
   python main.py
   ```

## Contributing
Contributions are welcome! Feel free to fork the repository and submit pull requests.

## License
This project is licensed under the MIT License.

## Contact
For inquiries, contact
[ٍShatha khaled] –[ٍShathah2030@gmail.com]
[Sharifah Malhan] – [Sh.sa.mal7@gmail.com]