SyntHesIzed Prompts (SHIP)

Introducing a groundbreaking technique named SyntHesIzed Prompts (SHIP), this paper aims to elevate existing fine-tuning methods. Fine-tuning involves training a pre-trained model on a smaller, task-specific dataset to enhance its applicability for specific tasks. However, a challenge arises when certain classes lack data, making it difficult to effectively train the model for those classes.

To tackle this issue, the researchers propose training a generative model capable of synthesizing features for categories without available data. This involves generating representations for classes that are absent in the training dataset, proving particularly valuable in scenarios where obtaining real data for certain classes poses challenges.

The key approach involves leveraging a variational autoencoder (VAE) framework, known for its ease of training and effectiveness in low-data scenarios, compared to models requiring adversarial training. By fine-tuning CLIP (Contrastive Language–Image Pretraining) using both the original labeled features and the newly synthesized features, the researchers strive to achieve state-of-the-art performance across various tasks.

The proposed model architecture ingeniously combines the VAE framework with CLIP to extract and reconstruct image features. During training, the model learns to encode features into a latent space and subsequently reconstruct them. During the generation phase, this learned encoding is utilized to synthesize features for new classes. The innovative CLIP-based generator, comprising a lightweight MLP and a frozen CLIP text encoder, plays a pivotal role in transforming the latent code and constructing the final prompts for feature reconstruction.

The experimental results illustrate that the novel method, SHIP, significantly enhances performance in new classes across different datasets. The paper diligently compares the results with other methods, demonstrating SHIP's superiority in generating features for categories lacking data.

In conclusion, this paper introduces SyntHesIzed Prompts (SHIP), an ingenious approach that enhances fine-tuning methods. By synthesizing features for categories without data and harnessing the power of the CLIP model, the researchers achieved state-of-the-art performance across various tasks. Future research may delve into exploring SHIP's applicability in dense prediction tasks.