📖 Abstract:

In recent years, protein-text language models are widely used to solve protein understanding tasks. Current approaches focus on integrating protein-related knowledge into LLMs through continued pretraining or multi-modal alignment, enabling LLMs to jointly understand protein sequences and textual descriptions.

However, by analysing existing model architectures and text-based protein understanding benchmarks, our analysis reveals significant data leakage in current text-based protein benchmarks, and metrics like ROUGE, BLEU are inadequate for evaluation in this domain.

To address these limitations, we reorganize existing datasets and introduce a novel OOD dataset, Prot-Inst-OOD, along with a new evaluation metric, Entity-BLEU. Furthermore, we propose a retrieval-enhanced method that significantly outperforms fine-tuned LLMs in protein-to-text generation, demonstrating both high accuracy and efficiency in training-free scenarios.

‼️ Data Leakage in Existing Protein-to-Text Benchmark

We evaluated four widely used benchmarks for text-based protein understanding: the protein comprehension tasks from Mol-Instructions [1], UniProtQA [2], Swiss-Prot Protein Caption [3], and ProteinKG25 [4].

[1] Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models
[2] BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine
[3] ProtT3: Protein-to-Text Generation for Text-based Protein Understanding
[4] OntoProtein: Protein Pretraining With Ontology Embedding

For sequence retrieval, we used MMSeqs2 with the following command:

mmseqs easy-search --max-accept 1 -e 1e5 -v 0 test_seqs.fasta train_seqs.fasta result.m8 tmp  

For each protein sequence in the test set, the label of its most similar counterpart in the training set was assigned as the predicted output. Note that we process different subtasks separately, instead of retrieving from mixed candidates.

The results, shown in the table below, demonstrate that all current LLM-based models perform worse than retrieval-based models.

We also analyzed data leakage rates, defined as the probability of obtaining identical labels using the retrieval method. For Mol-Instructions, we only consider metadata matches, ignoring differences in response phrasing, as shown in the table below. The results indicate that data leakage is prevalent in almost all benchmarks, with UniProtQA-Protein Family being the most severe case, where 97.7% of the test set can be predicted by retrieval.

（Left: Leakage Rate of different datasets; Right: An example sample of data leakage.）

Based on the above findings, we propose an Out-of-Distribution (OOD) split that is based on sequence similarity and removes samples in the training set that are highly similar to those in the test set. This split is designed to mitigate data leakage issues and provide a more accurate evaluation of model performance.

OOD datasets can be downloaded from Huggingface-link.

📊 New Metrics

Problems in Existing Metrics

Here is an example of the evaluation results using ROUGE-L and BLEU metrics on a sample protein sequence:

Bold: Matched Part    Italic: Mismatched Part

It is evident that the first prediction, which is the true answer, has a low ROUGE-L and BLEU score due to the lack of exact matches in the generated text. In contrast, the second prediction, which is incorrect, achieves high scores despite containing incorrect information. This highlights the inadequacy of these metrics for evaluating protein-to-text generation tasks.

Entity-BLEU

Entity-BLEU is a metric specifically designed for biological question answering, where standard NLP metrics like ROUGE and BLEU often fail to reflect the true quality of predictions. Unlike traditional metrics that treat all tokens equally, Entity-BLEU focuses on the correct identification of biological entities, such as protein domains or enzyme names, regardless of their order in the text.

The calculation process is as follows:

1. Entity Extraction: Biological entities are extracted from both the predicted and reference answers using a curated knowledge base
2. BLEU Calculation: The standard BLEU score is then computed, but instead of using the raw text, it operates on the sequences of extracted entities. This makes the metric order-invariant and robust to variations in phrasing.