Florence-2-large-TableDetection

This model is a fine-tuned version of microsoft/Florence-2-large-ft on ucsahin/pubtables-detection-1500-samples dataset. It achieves the following results on the evaluation set:

• Loss: 0.7601

microsoft/Florence-2-large-ft can detect various objects in zero-shot setting with the task prompt "<OD>". Please check Florence-2-large sample inference for how to use Florence-2 model in inference. However, the ft-base model is not able to detect tables on a given image.

The following Colab notebook showcases how you can finetune the model with your custom data to detect objects.

Florence2-Object Detection-Finetuning-HF-Trainer.ipynb

Model description

• This model is a multimodal language model fine-tuned for the task of detecting tables in images given textual prompts. The model utilizes a combination of image and text inputs to predict bounding boxes around tables within the provided images.
• The primary purpose of this model is to assist in automating the process of table detection within images. It can be utilized in various applications such as document processing, data extraction, and image analysis, where identifying tables within images is essential.

How to Get Started with the Model

In Transformers, you can load the model and inference as follows: (Note that trust_remote_code=True is needed to run the model. It will only download the external custom codes from the original HuggingFaceM4/Florence-2-DocVQA.)

from transformers import AutoProcessor, AutoModelForCausalLM
import matplotlib.pyplot as plt
import matplotlib.patches as patches

model_id = "ucsahin/Florence-2-large-TableDetection"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, device_map="cuda") # load the model on GPU
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

def run_example(task_prompt, image, max_new_tokens=128):
    prompt = task_prompt
    inputs = processor(text=prompt, images=image, return_tensors="pt")
    generated_ids = model.generate(
      input_ids=inputs["input_ids"].cuda(),
      pixel_values=inputs["pixel_values"].cuda(),
      max_new_tokens=max_new_tokens,
      early_stopping=False,
      do_sample=False,
      num_beams=3,
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
    parsed_answer = processor.post_process_generation(
        generated_text,
        task=task_prompt,
        image_size=(image.width, image.height)
    )
    return parsed_answer

def plot_bbox(image, data):
   # Create a figure and axes
    fig, ax = plt.subplots()
    # Display the image
    ax.imshow(image)
    # Plot each bounding box
    for bbox, label in zip(data['bboxes'], data['labels']):
        # Unpack the bounding box coordinates
        x1, y1, x2, y2 = bbox
        # Create a Rectangle patch
        rect = patches.Rectangle((x1, y1), x2-x1, y2-y1, linewidth=1, edgecolor='r', facecolor='none')
        # Add the rectangle to the Axes
        ax.add_patch(rect)
        # Annotate the label
        plt.text(x1, y1, label, color='white', fontsize=8, bbox=dict(facecolor='red', alpha=0.5))
    # Remove the axis ticks and labels
    ax.axis('off')
    # Show the plot
    plt.show()

########### Inference
from datasets import load_dataset

dataset = load_dataset("ucsahin/pubtables-detection-1500-samples")

example_id = 5
image = dataset["train"][example_id]["image"]

parsed_answer = run_example("<OD>", image=image)
plot_bbox(image, parsed_answer["<OD>"])

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
1.3199	1.0	169	1.0372
0.7922	2.0	338	0.9169
0.6824	3.0	507	0.8411
0.6109	4.0	676	0.8168
0.5752	5.0	845	0.7915
0.5605	6.0	1014	0.7862
0.5291	7.0	1183	0.7740
0.517	8.0	1352	0.7683
0.5139	9.0	1521	0.7642
0.5005	10.0	1690	0.7601

Framework versions

• Transformers 4.42.0.dev0
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1