iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://doi.org/10.1007/s10618-020-00704-w
Active learning for hierarchical multi-label classification | Data Mining and Knowledge Discovery Skip to main content

Advertisement

Springer Nature Link
Log in

Active learning for hierarchical multi-label classification

  • Published:
Data Mining and Knowledge Discovery Aims and scope Submit manuscript

Abstract

Due to technological advances, a massive amount of data is produced daily, presenting challenges for application areas where data needs to be labelled by a domain specialist or by expensive procedures, in order to be useful for supervised machine learning purposes. In order to select which data points will provide more information when labelled, one can make use of active learning methods. Active learning (AL) is a subfield of machine learning which addresses methods to build models with fewer, but more representative instances. Even though AL has been vastly studied, it has not been thoroughly investigated in hierarchical multi-label classification, a learning task where multiple class labels can be assigned to an instance and these labels are hierarchically structured. In this work, we provide a public framework containing baseline and state-of-the-art algorithms suitable for this task. Additionally, we also propose a new algorithm, namely Hierarchical Query-By-Committee (H-QBC), which is validated on datasets from different domains. Our results show that H-QBC is capable of providing superior predictive performance results compared to its competitors, while being computationally efficient and parameter free.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://geneontology.org/.

  2. https://itec.kuleuven-kulak.be/supportingmaterial.

  3. https://dtai.cs.kuleuven.be/clus/hmcdatasets/.

  4. https://dtai.cs.kuleuven.be/clus/hmc-ens/.

  5. http://kt.ijs.si/DragiKocev/PhD/resources/doku.php?id=hmc_classification.

  6. https://itec.kuleuven-kulak.be/supportingmaterial.

References

  • Athanasopoulos G, Gamakumara P, Panagiotelis A, Hyndman RJ, Affan M (2020) Hierarchical forecasting. Springer, Cham, pp 689–719

    Google Scholar 

  • Bekker J, Davis J (2018) Learning from positive and unlabeled data: a survey. CoRR, arXiv:1811.04820

  • Borchani H, Varando G, Bielza C, Larrañaga P (2015) A survey on multi-output regression. Wiley Int Rev Data Min Knowl Discov 5(5):216–233

    Article  Google Scholar 

  • Brinker K (2006) On active learning in multi-label classification. In: From data and information analysis to knowledge engineering. Springer, Berlin, pp 206–213

  • Cerri R, Barros RC, de Carvalho ACPLF (2012) A genetic algorithm for hierarchical multi-label classification. In: Proceedings of the 27th annual ACM symposium on applied computing, SAC ’12. ACM, New York, pp 250–255

  • Cerri R, Barros R, de Carvalho A (2015) Hierarchical classification of gene ontology-based protein functions with neural networks. In: Neural networks (IJCNN), 2015 international joint conference on, pp 1–8

  • Cerri R, Barros RC, de Carvalho AC, Jin Y (2016) Reduction strategies for hierarchical multi-label classification in protein function prediction. BMC Bioinf 17(1):373

    Article  Google Scholar 

  • Cerri R, Basgalupp MP, Barros RC, de Carvalho AC (2019) Inducing hierarchical multi-label classification rules with genetic algorithms. Appl Soft Comput 77:584–604

    Article  Google Scholar 

  • Chakraborty S, Balasubramanian V, Panchanathan S (2011) Optimal batch selection for active learning in multi-label classification. In: Proceedings of the 19th ACM international conference on multimedia, MM ’11. ACM, New York, pp 1413–1416

  • Chakraborty S, Balasubramanian V, Sankar AR, Panchanathan S, Ye J (2015) Batchrank: a novel batch mode active learning framework for hierarchical classification. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 99–108

  • Cheng Y, Zhang K, Xie Y, Agrawal A, Choudhary A (2012) On active learning in hierarchical classification. In: Proceedings of the 21st ACM international conference on information and knowledge management. ACM, pp 2467–2470

  • Cheng Y, Chen Z, Fei H, Wang F, Choudhary A (2014) Batch mode active learning with hierarchical-structured embedded variance. In: Proceedings of the 2014 SIAM international conference on data mining. SIAM, pp 10–18

  • Cherman EA, Papanikolaou Y, Tsoumakas G, Monard MC (2019) Multi-label active learning: key issues and a novel query strategy. Evol Syst 10:63–78

    Article  Google Scholar 

  • Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  • Duin JD (2017) Hierarchical active learning application to mitochondrial disease. University of Nebraska, Tese de Doutorado

    Google Scholar 

  • Gargiulo F, Silvestri S, Ciampi M, Pietro GD (2019) Deep neural network for hierarchical extreme multi-label text classification. Appl Soft Comput 79:125–138

    Article  Google Scholar 

  • Guo A, Wu J, Sheng VS, Zhao P, Cui Z (2017) Multi-label active learning with low-rank mapping for image classification. In: 2017 IEEE international conference on multimedia and expo (ICME), pp 259–264

  • Hoi SCH, Jin R, Zhu J, Lyu MR (2006) Batch mode active learning and its application to medical image classification. In: Proceedings of the 23rd international conference on machine learning, ICML ’06. Association for Computing Machinery, New York, pp 417–424

  • Hung C-W, Lin H-T (2011) Multi-label active learning with auxiliary learner. In: Hsu C-N, Lee WS (eds) Proceedings of the Asian conference on machine learning, volume 20 of Proceedings of machine learning research, PMLR. South Garden Hotels and Resorts, Taoyuan, pp 315–332

  • Huang S, Zhou Z (2013) Active query driven by uncertainty and diversity for incremental multi-label learning. In: 2013 IEEE 13th international conference on data mining, pp 1079–1084

  • Huang S, Jin R, Zhou Z (2014) Active learning by querying informative and representative examples. IEEE Trans Pattern Anal Mach Intell 36(10):1936–1949

    Article  Google Scholar 

  • Jiao Y, Zhao P, Wu J, Xian X, Xu H, Cui Z (2014) Active multi-label learning with optimal label subset selection. In: Luo X, Yu JX, Li Z (eds) Advanced data mining and applications. Springer, Cham, pp 523–534

    Chapter  Google Scholar 

  • Klimt B, Yang Y (2004) The enron corpus: a new dataset for email classification research. In: ECML ’04: proceedings of the 18th European conference on machine learning—LNCS 3201. Springer, Berlin, pp 217–226

  • Kocev D, Vens C, Struyf J, Džeroski S (2013) Tree ensembles for predicting structured outputs. Pattern Recogn 46(3):817–833

    Article  Google Scholar 

  • Krawczyk B, Minku LL, Gama J, Stefanowski J, Woźniak M (2017) Ensemble learning for data stream analysis: a survey. Inf Fusion 37:132–156

    Article  Google Scholar 

  • Levatić J, Ceci M, Kocev D, Džeroski S (2017) Self-training for multi-target regression with tree ensembles. Knowl-Based Syst 123:41–60

    Article  Google Scholar 

  • Levatić J, Kocev D, Ceci M, Džeroski S (2018) Semi-supervised trees for multi-target regression. Inf Sci 450:109–127

    Article  MathSciNet  Google Scholar 

  • Lewis DD, Catlett J (1994) Heterogeneous uncertainty sampling for supervised learning. In: Proceedings of the eleventh international conference on machine learning, pp 148–156

  • Lewis DD, Yang Y, Rose TG, Li F (2004) Rcv1: a new benchmark collection for text categorization research. J Mach Learn Res 5:361–397

    Google Scholar 

  • Li X, Guo Y (2013) Active learning with multi-label SVM classification. In: IJCAI international joint conference on artificial intelligence, pp 1479–1485

  • Li X, Wang L, Sung E (2004) Multilabel SVM active learning for image classification. In: 2004 international conference on image processing, 2004. ICIP ’04, vol 4, pp 2207–2210

  • Li X, Kuang D, Ling CX (2012) Active learning for hierarchical text classification. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, Berlin, pp 14–25

  • Li X, Ling CX, Wang H (2013) Effective top-down active learning for hierarchical text classification. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, Berli, pp 233–244

  • Mileski V, Džeroski S, Kocev D (2017) Predictive clustering trees for hierarchical multi-target regression. In: Adams N, Tucker A, Weston D (eds) Advances in Intelligent Data Analysis, vol XVI. Springer, Cham, pp 223–234

  • Mo Y, Scott SD, Downey D (2016) Learning hierarchically decomposable concepts with active over-labeling. In: 2016 IEEE 16th international conference on data mining (ICDM), pp 340–349

  • Nakano FK, Pinto WJ, Pappa GL, Cerri R (2017) Top-down strategies for hierarchical classification of transposable elements with neural networks. In: International joint conference on neural networks (IJCNN), pp 2539–2546

  • Nakano FK, Lietaert M, Vens C (2019) Machine learning for discovering missing or wrong protein function annotations. BMC Bioinf 20(1):485

    Article  Google Scholar 

  • Pliakos K, Vens C (2018) Mining features for biomedical data using clustering tree ensembles. J Biomed Inf 85:40–48

    Article  Google Scholar 

  • Qi G-J, Hua X-S, Rui Y, Tang J, Zhang H-J (2008) Two-dimensional active learning for image classification. In: 2008 IEEE conference on computer vision and pattern recognition, pp 1–8

  • Qian C, Yu Y, Zhou Z-H (2015) Subset selection by pareto optimization. In: Proceedings of the 28th International conference on neural information processing systems—vol 1, NIPS’15. MIT Press, Cambridge, pp 1774–1782

  • Reyes O, Morell C, Ventura S (2018) Effective active learning strategy for multi-label learning. Neurocomputing 273:494–508

    Article  Google Scholar 

  • Ribeiro MT, Singh S, Guestrin C (2016) “why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135–1144

  • Rubens N, Kaplan D, Sugiyama M (2011) Active learning in recommender systems. In: Kantor P, Ricci F, Rokach L, Shapira B (eds) Recommender systems handbook. Springer, Berlin, pp 735–767

    Chapter  Google Scholar 

  • Schietgat L, Vens C, Struyf J, Blockeel H, Kocev D, Džeroski S (2010) Predicting gene function using hierarchical multi-label decision tree ensembles. BMC Bioinf 11(1):2

    Article  Google Scholar 

  • Settles B (2010) Active learning literature survey. University of Wisconsin, Madison 52(55–66):11

  • Seung HS, Opper M, Sompolinsky H (1992) Query by committee. In: Proceedings of the fifth annual workshop on computational learning theory. ACM, pp 287–294

  • Štrumbelj E, Kononenko I (2014) Explaining prediction models and individual predictions with feature contributions. Knowl Inf Syst 41(3):647–665

    Article  Google Scholar 

  • Valentini G (2010) True path rule hierarchical ensembles for genome-wide gene function prediction. IEEE/ACM Trans Comput Biol Bioinf 8(3):832–847

    Article  Google Scholar 

  • van Engelen JE, Hoos HH (2020) A survey on semi-supervised learning. Mach Learn 109:373–440

    Article  MathSciNet  Google Scholar 

  • Vasisht D, Damianou A, Varma M, Kapoor A (2014) Active learning for sparse Bayesian multilabel classification. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’14. ACM, New York, pp 472–481

  • Vens C, Struyf J, Schietgat L, Džeroski S, Blockeel H (2008) Decision trees for hierarchical multi-label classification. Mach Learn 73:185–214

    Article  Google Scholar 

  • Wang X, Zhao H, Lu B-L (2011) Enhanced k-nearest neighbour algorithm for large-scale hierarchical multi-label classification. In: Proceedings of the joint ECML/PKDD PASCAL workshop on large-scale hierarchical classification

  • Wehrmann J, Cerri R, Barros R (2018) Hierarchical multi-label classification networks. In: Dy J, Krause A (eds) Proceedings of the 35th international conference on machine learning, volume 80 of proceedings of machine learning research (PMLR), Stockholmsmässan, Stockholm, pp 5075–5084

  • Wu J, Sheng VS, Zhang J, Zhao P, Cui Z (2014) Multi-label active learning for image classification. In: 2014 IEEE international conference on image processing (ICIP), pp 5227–5231

  • Wu J, Ye C, Sheng VS, Zhang J, Zhao P, Cui Z (2017) Active learning with label correlation exploration for multi-label image classification. IET Comput Vis 11(7):577–584

    Article  Google Scholar 

  • Wu J, Guo A, Sheng VS, Zhao P, Cui Z (2018) An active learning approach for multi-label image classification with sample noise. Int J Pattern Recogn Artif Intell 32(03):1850005

    Article  MathSciNet  Google Scholar 

  • Yan Y, Huang S-J (2018) Cost-effective active learning for hierarchical multi-label classification. IJCAI, pp 2962–2968

  • Yan Y, Rosales R, Fung G, Dy JG (2011) Active learning from crowds. In: Proceedings of the 28th international conference on international conference on machine learning, ICML’11. Omnipress, Madison, pp 1161–1168

  • Yang B, Sun J-T, Wang T, Chen Z (2009) Effective multi-label active learning for text classification. In: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’09. ACM, New York, pp 917–926

  • Yang K, Ren J, Zhu Y, Zhang W (2018) Active learning for wireless IoT intrusion detection. IEEE Wirel Commun 25(6):19–25

    Article  Google Scholar 

  • Ye C, Wu J, Sheng V, Zhao P, Cui Z (2015a) Multi-label active learning with label correlation for image classification, pp 3437–3441

  • Ye C, Wu J, Sheng VS, Zhao S, Zhao P, Cui Z (2015b) Multi-label active learning with chi-square statistics for image classification. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, ICMR ’15. Association for Computing Machinery, New York, pp 583–586

  • Yu G, Fu G, Wang J, Zhao Y (2017) Newgoa: predicting new go annotations of proteins by bi-random walks on a hybrid graph. IEEE/ACM Trans Comput Biol Bioinf 15(4):1390–1402

    Article  Google Scholar 

  • Zeng C, Zhou W, Li T, Shwartz L, Grabarnik GY (2017) Knowledge guided hierarchical multi-label classification over ticket data. IEEE Trans Netw Serv Manag 14(2):246–260

    Article  Google Scholar 

  • Zhang M-L (2009) ML-RBF: RBF neural networks for multi-label learning. Neural Process Lett 29:61–74

    Article  Google Scholar 

  • Zhang B, Wang Y, Wang W (2012) Batch mode active learning for multi-label image classification with informative label correlation mining. In: 2012 IEEE workshop on the applications of computer vision (WACV), pp 401–407

  • Zhang B, Wang Y, Chen F (2014) Multilabel image classification via high-order label correlation driven active learning. IEEE Trans Image Process 23(3):1430–1441

    Article  MathSciNet  Google Scholar 

  • Zhang Z, Zhang J, Liu Y, Wang Z, Deng L (2017) Ontological function annotation of long non-coding RNAs through hierarchical multi-label classification. Bioinformatics 34(10):1750–1757

    Article  Google Scholar 

  • Zhao Y, Wang J, Chen J, Zhang X, Guo M, Yu G (2020) A literature review of gene function prediction by modeling gene ontology. Front Genet 11:400

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to FAPESP (Grants #2016/12489-2 and #2017/19264-9), Research Fund Flanders (FWO) and Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brasil (CAPES)—Finance Code 001 for providing financial support.

Author information

Authors and Affiliations

  1. Department of Public Health and Primary Care, KU Leuven Campus KULAK, Etienne Sabbelaan 53, 8500, Kortrijk, Belgium

    Felipe Kenji Nakano & Celine Vens

  2. Itec, imec Research Group at KU Leuven, Etienne Sabbelaan 53, 8500, Kortrijk, Belgium

    Felipe Kenji Nakano & Celine Vens

  3. Department of Computer Science, Federal University of São Carlos, Rodovia Washington Luís, Km 235, São Carlos, SP, 13565-905, Brazil

    Ricardo Cerri

Authors
  1. Felipe Kenji Nakano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ricardo Cerri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Celine Vens
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felipe Kenji Nakano.

Additional information

Responsible editor: Ira Assent, Carlotta Domeniconi, Aristides Gionis, Eyke Hüllermeier.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 269 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nakano, F.K., Cerri, R. & Vens, C. Active learning for hierarchical multi-label classification. Data Min Knowl Disc 34, 1496–1530 (2020). https://doi.org/10.1007/s10618-020-00704-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10618-020-00704-w

Keywords

Search

Navigation