Abstract
The exponential availability of protein sequences has led to the dominance of the pretraining-then-finetuning paradigm for protein function prediction. However, finetuning a pretrained protein language model for diverse downstream tasks requires annotated protein data tailored to each task. To avoid the redundant individual finetuning, we propose a methodology that unifies various Protein function prediction tasks via Text Matching (named ProTeM). This method first transforms simple numeric or category labels from disparate protein datasets into textual descriptions, imbued with rich semantics. We then harness a pretrained large language model, which is proficient in comprehensive language understanding, to capture intrinsic interconnections among varied protein functions and facilitate the alignment between text and protein. During inference, we employ the paradigm of text matching to predict the protein functionalities. Extensive experiments demonstrate that ProTeM achieves performance on par with individually finetuned models, and outshines the model based on conventional multi-task learning. Moreover, ProTeM unveils an enhanced capacity for protein representation, surpassing state-of-the-art PLMs.
M. Qin and X. Li—Contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Almagro Armenteros, J.J., Sønderby, C.K., Sønderby, S.K., Nielsen, H., Winther, O.: Deeploc: prediction of protein subcellular localization using deep learning. Bioinformatics 33(21), 3387–3395 (2017)
Bonetta, R., Valentino, G.: Machine learning techniques for protein function prediction. Proteins: Structure, Function Bioinform. 88(3), 397–413 (2020)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020)
Chung, Y.A., Zhu, C., Zeng, M.: Splat: Speech-language joint pre-training for spoken language understanding. arXiv preprint arXiv:2010.02295 (2020)
Elnaggar, A., et al.: Prottrans: toward understanding the language of life through self-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 44(10) (2022). https://doi.org/10.1109/TPAMI.2021.3095381
Gligorijević, V., et al.: Structure-based protein function prediction using graph convolutional networks. Nat. Commun. 12(1), 3168 (2021)
Hu, E.J., et al.: Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
Jumper, J., et al.: Highly accurate protein structure prediction with alphafold. Nature 596(7873), 583–589 (2021)
Khurana, S., Rawi, R., Kunji, K., Chuang, G.Y., Bensmail, H., Mall, R.: Deepsol: a deep learning framework for sequence-based protein solubility prediction. Bioinformatics 34(15), 2605–2613 (2018)
Krogh, A., Larsson, B., Von Heijne, G., Sonnhammer, E.L.: Predicting transmembrane protein topology with a hidden Markov model: application to complete genomes. J. Mol. Biol. 305(3), 567–580 (2001)
Li, L.H., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965–10975 (2022)
Lin, Z., et al.: Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379(6637), 1123–1130 (2023)
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. CoRR: abs/2107.13586 (2021)
Luo, H., et al.: Clip4clip: an empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508, 293–304 (2022)
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., Rives, A.: Language models enable zero-shot prediction of the effects of mutations on protein function. Adv. Neural. Inf. Process. Syst. 34, 29287–29303 (2021)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)
Radivojac, P., et al.: A large-scale evaluation of computational protein function prediction. Nat. Methods 10(3), 221–227 (2013)
Rao, R., et al.: Evaluating protein transfer learning with tape. Adv. Neural. Inf. Process. Syst. 32, 9689 (2019)
Rives, A., et al.: Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. 118(15), e2016239118 (2021)
Schick, T., Schütze, H.: Exploiting cloze-questions for few-shot text classification and natural language inference. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255–269 (2021)
Singh, A., et al.: Flava: a foundational language and vision alignment model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638–15650 (2022)
Touvron, H., et al.: Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
Wang, Z., et al.: Instructprotein: aligning human and protein language via knowledge instruction. arXiv preprint arXiv:2310.03269 (2023)
Wang, Z., et al.: Multi-level protein structure pre-training via prompt learning. In: The Eleventh International Conference on Learning Representations (2022)
Wu, F., de Boer, R., van der Klei, I.J.: Gluing yeast peroxisomes–composition and function of membrane contact sites. J. Cell Sci. 136(11), jcs259440 (2023)
Xu, M., Yuan, X., Miret, S., Tang, J.: Protst: Multi-modality learning of protein sequences and biomedical texts. arXiv preprint arXiv:2301.12040 (2023)
Xu, M., et al.: Peer: a comprehensive and multi-task benchmark for protein sequence understanding. Adv. Neural. Inf. Process. Syst. 35, 35156–35173 (2022)
Zhang, N., et al.: Ontoprotein: protein pretraining with gene ontology embedding. arXiv preprint arXiv:2201.11147 (2022)
Zhang, Y., Yang, Q.: A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34(12), 5586–5609 (2021)
Zhang, Z., et al.: Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203.06125 (2022)
Zhou, C., et al.: Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 (2023)
Zhou, H.Y., Fu, Y., Zhang, Z., Cheng, B., Yu, Y.: Protein representation learning via knowledge enhanced primary structure reasoning. In: The Eleventh International Conference on Learning Representations (2022)
Acknowledgements
This work is supported by National Natural Science Foundation of China (U23A20496, 62302433, 62301480), Hangzhou West Lake Pearl Project Leading Innovative Youth Team Project (TD2023017), and Zhejiang Provincial Natural Science Foundation of China (LQ24F020007). Some figures has been designed using images from Flaticon.com and icons8.com.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Qin, M. et al. (2024). ProTeM: Unifying Protein Function Prediction via Text Matching. In: Wand, M., Malinovská, K., Schmidhuber, J., Tetko, I.V. (eds) Artificial Neural Networks and Machine Learning – ICANN 2024. ICANN 2024. Lecture Notes in Computer Science, vol 15023. Springer, Cham. https://doi.org/10.1007/978-3-031-72353-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-72353-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72352-0
Online ISBN: 978-3-031-72353-7
eBook Packages: Computer ScienceComputer Science (R0)