iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://doi.org/10.1007/978-3-031-69583-4_28
MPR: An MPI Framework for Distributed Self-adaptive Stream Processing | SpringerLink
Skip to main content
Springer Nature Link
Log in

MPR: An MPI Framework for Distributed Self-adaptive Stream Processing

  • Conference paper
  • First Online:
Euro-Par 2024: Parallel Processing (Euro-Par 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14803))

Included in the following conference series:

  • 385 Accesses

Abstract

Stream processing systems must often cope with workloads varying in content, format, size, and input rate. The high variability and unpredictability make statically fine-tuning them very challenging. Our work addresses this limitation by providing a new framework and runtime system to simplify implementing and assessing new self-adaptive algorithms and optimizations. We implement a prototype on top of MPI called MPR and show its functionality. We focus on horizontal scaling by supporting the addition and removal of processes during execution time. Experiments reveal that MPR can achieve performance similar to that of a handwritten static MPI application. We also assess MPR’s adaptation capabilities, showing that it can readily re-configure itself, with the help of a self-adaptive algorithm, in response to workload variations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/GMAP/MPR.

References

  1. Apache Software Foundation. Apache Flink, December 2022

    Google Scholar 

  2. Apache Software Foundation. Apache Spark, December 2022

    Google Scholar 

  3. Bingmann, T., et al.: Thrill: high-performance algorithmic distributed batch data processing with C++. In: International Conference on Big Data, pp. 172–183 (2016)

    Google Scholar 

  4. Chandy, K.M., Lamport, L.: Distributed snapshots: determining global states of distributed systems. ACM Trans. Comput. Syst. 3, 63–75 (1985)

    Article  Google Scholar 

  5. Ciechanowicz, P., Kuchen, H.: Enhancing Muesli’s data parallel skeletons for multi-core computer architectures. In: International Conference on High Performance Computing and Communications, pp. 108–113 (2010)

    Google Scholar 

  6. Falcou, J., Sérot, J., Chateau, T., Lapresté, J.-T.: Quaff: efficient C++ design for parallel skeletons. Parallel Comput. 32, 604–615 (2006)

    Article  Google Scholar 

  7. Hori, A., et al.: An international survey on MPI users. Parallel Comput. 108, 1–13 (2021)

    Article  MathSciNet  Google Scholar 

  8. Kalavri, V., Liagouris, J., Hoffmann, M., Dimitrova, D., Forshaw, M., Roscoe, T.: Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In: International Conference on Operating Systems Design and Implementation, pp. 783–798 (2018)

    Google Scholar 

  9. Löff, J., Hoffmann, R.B., Pieper, R., Griebler, D., Fernandes, L.G.: DSParLib: a C++ template library for distributed stream parallelism. Int. J. Parallel Prog. 50, 454–485 (2022)

    Article  Google Scholar 

  10. López-Gómez, J., Fernández Muñoz, J., del Rio Astorga, D., Dolz, M.F., Garcia, J.D.: Exploring stream parallel patterns in distributed MPI environments. Parallel Comput. 84, 24–36 (2019)

    Google Scholar 

  11. Mancini, E.P., Marsh, G., Panda, D.K.: An MPI-stream hybrid programming model for computational clusters. In: International Conference on Cluster, Cloud and Grid Computing, pp. 323–330 (2010)

    Google Scholar 

  12. Morisawa, Y., Suzuki, M., Kitahara, T.: Flexible executor allocation without latency increase for stream processing in apache spark. In: International Conference on Big Data, pp. 2198–2206 (2020)

    Google Scholar 

  13. Rivas-Gomez, S., et al.: MPI windows on storage for HPC applications. Parallel Comput. 77, 38–56 (2018)

    Article  MathSciNet  Google Scholar 

  14. Tonci, N., Torquati, M., Mencagli, G., Danelutto, M.: Distributed-memory fastflow building blocks. Int. J. Parallel Prog. 51, 1–21 (2022)

    Article  Google Scholar 

  15. Van Dongen, G., Van Den Poel, D.: Influencing factors in the scalability of distributed stream processing jobs. IEEE Access 9, 109413–109431 (2021)

    Article  Google Scholar 

  16. Wagner, A., Rostoker, C.: A lightweight stream-processing library using MPI. In: International Symposium on Parallel and Distributed Processing, pp. 1–8 (2009)

    Google Scholar 

Download references

Acknowledgments

The research presented in this paper was supported by Oracle (ERO project 1332), the Swiss National Science Foundation (project 200020_188688), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001, and LAD-PUCRS.

Author information

Authors and Affiliations

  1. Faculty of Informatics, Università della Svizzera italiana (USI), Lugano, Switzerland

    Júnior Löff & Walter Binder

  2. School of Technology, Pontifical Catholic University of Rio Grande do Sul (PUCRS), Porto Alegre, Brazil

    Júnior Löff, Dalvan Griebler & Luiz Gustavo Fernandes

Authors
  1. Júnior Löff
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dalvan Griebler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luiz Gustavo Fernandes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Walter Binder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Júnior Löff .

Editor information

Editors and Affiliations

  1. University Carlos III of Madrid, Madrid, Spain

    Jesus Carretero

  2. University of Oregon, Eugene, OR, USA

    Sameer Shende

  3. University Carlos III of Madrid, Madrid, Spain

    Javier Garcia-Blas

  4. TU Wien, Vienna, Austria

    Ivona Brandic

  5. Universidad Complutense de Madrid, Madrid, Spain

    Katzalin Olcoz

  6. Université Grenoble Alpes, Saint Martin d'Hères, France

    Martin Schreiber

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Löff, J., Griebler, D., Fernandes, L.G., Binder, W. (2024). MPR: An MPI Framework for Distributed Self-adaptive Stream Processing. In: Carretero, J., Shende, S., Garcia-Blas, J., Brandic, I., Olcoz, K., Schreiber, M. (eds) Euro-Par 2024: Parallel Processing. Euro-Par 2024. Lecture Notes in Computer Science, vol 14803. Springer, Cham. https://doi.org/10.1007/978-3-031-69583-4_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-69583-4_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-69582-7

  • Online ISBN: 978-3-031-69583-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation