arig23498/ft-pali-gemma

Notebooks for fine tuning pali gemma

Fine Tuning PaliGemma

What is PaliGemma?

PaliGemma is a new family of vision-language models from Google. These models can process both images and text to produce text outputs.

Google has released three types of PaliGemma models:

1. Pretrained (pt) models: Trained on large datasets without task-specific tuning.
2. Mix models: A combination of pre-trained and fine-tuned elements.
3. Fine-tuned (ft) models: Optimized for specific tasks with additional training.

Each type comes in different resolutions and multiple precisions for convenience. All models are available on the Hugging Face Hub with model cards, licenses, and integration with transformers.

Fine-Tuning Methods

1. JAX Fine-Tuning Script
2. Fine-tuning using HuggingFace transformers
3. Fine-tuning using Vanilla Pytorch scripts (shown here) a. Fine tune with an Image Captioning Dataset (vanilla_ft.py) b. Fine tune with an Object Detection Dataset (object_detection_ft.py)

Results

Image Captioning

In the script provided we have used the tuxemon dataset, from the diffusers team. The dataset comprises of images of tuxemons (a spin off of pokemons) and their captions.

Before Fine Tuning	After Fine Tuning

Object Detection

While I could not find a document that provides pointers to train the model on a detection dataset, diving in the official big vision space made it really clear. Taking inspiration from the space, I have create a script to format any object detection dataset (here the dataset is based on the coco format) to the format PaliGemma is trained on.

You can find the dataset creation script here: create_od_dataset.py.

After the dataset is created run the fine tuning script object_detection_ft.py and run the model.

Before Fine Tuning	After Fine Tuning

Citation

If you like our work and would use it please cite us! ^_^

@misc{github_repository,
  author = {Aritra Roy Gosthipaty, Ritwik Raha}, 
  title = {ft-pali-gemma}, 
  publisher = {{GitHub}(https://github.com)},
  howpublished = {\url{https://github.com/ariG23498/ft-pali-gemma/edit/main/README.md}},
  year = {2024}  
}

ft-pali-gemma's Issues

removing train parameter on collate_fn for huggingface's Trainer

Thank you for sharing the finetuning code on paligemma, this really helps me!

I have a question for collate_fn.

Seeing the example on huggingface and yours, I would like to know if it is possible to use Trainer from huggingface instead of using the normal loop like yours.

I am not sure on how to remove the train=True parameter on your collate_fn function

    collate_fn_trainer = partial(collate_fn, 
                                 image_title="image", 
                                 prompt="Detect license plate.", 
                                 suffix_title="label_for_paligemma", 
                                 processor=processor, 
                                 device=device, 
                                 train=True) -> how to 
    trainer = Trainer(
            model=model,
            train_dataset=train_dataset,
            eval_dataset=validation_dataset,
            data_collator=collate_fn_trainer,
            args=args
            )