---
library_name: transformers
license: mit
datasets:
- c3rl/IIIT-INDIC-HW-WORDS-Hindi
language:
- ne
metrics:
- cer
base_model:
- google/vit-base-patch16-224-in21k
- amitness/roberta-base-ne
pipeline_tag: image-to-text
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->



## Model Details


# TrOCR Devanagari - Handwritten Text Recognition

## Overview
TrOCR Devanagari is an end-to-end Vision Encoder-Decoder model built to recognize and convert handwritten Devanagari script (specifically for Nepali language) into machine-readable text. It leverages a Vision Transformer (ViT) as the encoder and uses a transformer-based decoder (NepBERT) to produce textual output. This project aims to assist in digitizing handwritten Nepali documents.

## Model Architecture
The model pipeline includes the following steps:
1. **Text Detection:** Extracts regions of interest from scanned handwritten documents.
2. **Image Preprocessing:** Resizes and pads input images to feed into the model.
3. **Text Recognition:** Uses the TrOCR-based Vision Encoder Decoder model to predict handwritten text.
4. **UI Interface (Optional):** Displays the results and enables user interaction with the system.

## Model Information
- **Model Name:** TrOCR Devanagari
- **Developed by:** Anil Paudel, Aayush Puri, Yubaraj Sigdel
- **Language:** Nepali
- **License:** MIT (tentative)
- **Model Type:** Vision Encoder Decoder
- **Repository:** [paudelanil/trocr-devanagari-2](https://huggingface.co./paudelanil/trocr-devanagari-2)
- **Training Data:** IIIT-HW Dataset
- **Evaluation Metric:** CER (Character Error Rate)
- **Hardware Used:** NVIDIA RTX A4500

## Getting Started

### Installation

To use the model, ensure you have the following Python packages installed:
```bash
pip install torch transformers pillow
```

### Preprocessing Function

The image preprocessing function is used to resize images to the target size while maintaining the aspect ratio and padding the remaining space.

```python
from PIL import Image

def preprocess_image(image):
    target_size = (224, 224)
    original_size = image.size

    aspect_ratio = original_size[0] / original_size[1]
    if aspect_ratio > 1:
        new_width = target_size[0]
        new_height = int(target_size[0] / aspect_ratio)
    else:
        new_height = target_size[1]
        new_width = int(target_size[1] * aspect_ratio)

    resized_img = image.resize((new_width, new_height))

    padding_width = target_size[0] - new_width
    padding_height = target_size[1] - new_height
    pad_left = padding_width // 2
    pad_top = padding_height // 2

    pad_image = Image.new('RGB', target_size, (255, 255, 255))
    pad_image.paste(resized_img, (pad_left, pad_top))
    return pad_image
```

### Prediction Code

Here’s how you can use the model for text recognition:

```python
import torch
from PIL import Image
from transformers import AutoTokenizer, VisionEncoderDecoderModel, ViTFeatureExtractor, TrOCRProcessor

# Load the model and processor
tokenizer = AutoTokenizer.from_pretrained("aayushpuri01/TrOCR-Devanagari")
model1 = VisionEncoderDecoderModel.from_pretrained("aayushpuri01/TrOCR-Devanagari")
feature_extractor1 = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224-in21k')
processor1 = TrOCRProcessor(feature_extractor=feature_extractor1, tokenizer=tokenizer)

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model1.to(device)

# Prediction function
def predict(image):
    # Preprocess the image
    image = Image.open(image).convert("RGB")
    image = preprocess_image(image)
    pixel_values = processor1(image, return_tensors="pt").pixel_values.to(device)
    
    # Generate text from the image
    generated_ids = model1.generate(pixel_values)
    generated_text = processor1.batch_decode(generated_ids, skip_special_tokens=True)[0]
    
    return generated_text
```

### Usage Example

```python
# Load and predict
image_path = "path_to_your_image.jpg"
predicted_text = predict(image_path)
print("Predicted Text:", predicted_text)
```

## Training Hyperparameters
```python
training_args = Seq2SeqTrainingArguments(
    predict_with_generate=True,
    evaluation_strategy="steps",
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    output_dir='/workspace/checkpoint-save/',
    save_total_limit=2,
    logging_steps=2,
    save_steps=1000,
    eval_steps=1000,
    save_strategy="steps",
    load_best_model_at_end=True,
    metric_for_best_model="cer",
    greater_is_better=False,
    num_train_epochs=15
)
```

## License
The model is shared under the MIT license. For details, see the [LICENSE](LICENSE) file.

## Acknowledgments
This model is based on the 🤗 Transformers library, and uses the ViT encoder and NepBERT decoder architecture. Special thanks to the IIIT-HW dataset contributors.

---

Feel free to explore the project and contribute to the repository!