CLIP's Visual Embedding Projector is a Few-shot Cornucopia (ProLIP)

TL;DR: CLIP projects the visual embeddings to the shared latent space using a linear projection layer. ProLIP simply fine-tunes this layer with a zero-shot regularization loss. ProLIP is a strong alternative to linear probing, prompt tuning and CLIP-adapters, and is robust to the learning rate. It also significantely outperforms prompt tuning on test-time adaptation.

Table of Content

Installation

Dependencies

Create a conda environment and install dependencies:

git clone https://github.com/astra-vision/ProLIP.git
cd ProLIP
conda create -n prolip python=3.8
conda activate prolip
# Install the according versions of torch and torchvision
conda install pytorch torchvision cudatoolkit

Datasets

Follow DATASET.md to install the datasets.

Running ProLIP

Before running ProLIP, make sure to change root_path in the configuration files at configs/experiments to the path of the datasets.

Saving Pre-projection Features

First, start by saving the pre-projection features; i.e. the features to which the linear projection of CLIP is applied.

bash scripts/save_features.sh

You can change the backbone in save_features.yaml if you want its corresponding features.

From now on, you can directly train the projection layer using these features, requiring up to 2 seconds per training.

Few-shot Classification with Few-shot Validation

To obtain the results of few-shot classification with a few-shot validation set for hyperparameter selection (which corresponds to the setting of LP++), please run:

bash scripts/few_shot_few_val.sh

These experiments correspond to Table 1, Table 20 and Table 21 of the paper.

ProLIP sensitivity to hyperparameters

To obtain the results of few-shot classification for different combinations of hyperparameters (i.e. Learning rate (lr) and