iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://unpaywall.org/10.1007/S10723-023-09730-6
A Federated Deep Reinforcement Learning-based Low-power Caching Strategy for Cloud-edge Collaboration | Journal of Grid Computing Skip to main content
Springer Nature Link
Log in

A Federated Deep Reinforcement Learning-based Low-power Caching Strategy for Cloud-edge Collaboration

  • Research
  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

In the era of ubiquitous network devices, an exponential increase in content requests from user equipment (UE) calls for optimized caching strategies within a cloud-edge integration. This approach is critical to handling large numbers of requests. To enhance caching efficiency, federated deep reinforcement learning (FDRL) is widely used to adjust caching policies. Nonetheless, for improved adaptability in dynamic scenarios, FDRL generally demands extended and online deep training, incurring a notable energy overhead when contrasted with rule-based approaches. With the aim of achieving a harmony between caching efficiency and training energy expenditure, we integrate a content request latency model, a deep reinforcement learning model based on markov decision processes (MDP), and a two-stage training energy consumption model. Together, these components define a new average delay and training energy gain (ADTEG) challenge. To address this challenge, we put forth a innovative dynamic federated optimization strategy. This approach refines the pre-training phase through the use of cluster-based strategies and parameter transfer methodologies. The online training phase is improved through a dynamic federated framework and an adaptive local iteration count. The experimental findings affirm that our proposed methodology reduces the training energy outlay while maintaining caching efficacy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Xia, X., Chen, F., He, Q., Grundy, J., Abdelrazek, M., Shen, J., Bouguettaya, A., Jin, H.: Formulating cost-effective data distribution strategies online for edge cache systems. IEEE Trans. Parallel Distrib. Syst. 33(12), 4270–4281 (2022)

    Article  Google Scholar 

  2. Taleb, T., Samdanis, K., Mada, B., Flinck, H., Dutta, S., Sabella, D.: On multi-access edge computing: a survey of the emerging 5g network edge cloud architecture and orchestration. IEEE Communications Surveys & Tutorials 19(3), 1657–1681 (2017)

    Article  Google Scholar 

  3. Li, X., Yi, W., Chi, H.-L., Wang, X., Chan, A.P.: A critical review of virtual and augmented reality (vr/ar) applications in construction safety. Autom. Constr. 86, 150–162 (2018)

    Article  Google Scholar 

  4. Mao, Y., You, C., Zhang, J., Huang, K., Letaief, K.B.: A survey on mobile edge computing: the communication perspective. IEEE Communications Surveys & Tutorials 19(4), 2322–2358 (2017)

    Article  Google Scholar 

  5. Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: vision and challenges. IEEE Internet Things J. 3(5), 637–646 (2016)

    Article  Google Scholar 

  6. Xu, X., Liu, Q., Luo, Y., Peng, K., Zhang, X., Meng, S., Qi, L.: A computation offloading method over big data for iot-enabled cloud-edge computing. Futur. Gener. Comput. Syst. 95, 522–533 (2019)

    Article  Google Scholar 

  7. Mudassar, B.A., Ko, J.H., Mukhopadhyay, S.: Edge-cloud collaborative processing for intelligent internet of things: a case study on smart surveillance. In: 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), pp. 1–6 (2018)

  8. Yousefpour, A., Fung, C., Nguyen, T., Kadiyala, K., Jalali, F., Niakanlahiji, A., Kong, J., Jue, J.P.: All one needs to know about fog computing and related edge computing paradigms: a complete survey. J. Syst. Architect. 98, 289–330 (2019)

  9. Thai, M.-T., Lin, Y.-D., Lai, Y.-C., Chien, H.-T.: Workload and capacity optimization for cloud-edge computing systems with vertical and horizontal offloading. IEEE Trans. Netw. Serv. Manage. 17(1), 227–238 (2019)

    Article  Google Scholar 

  10. Asim, M., Wang, Y., Wang, K., Huang, P.-Q.: A review on computational intelligence techniques in cloud and edge computing. IEEE Transactions on Emerging Topics in Computational Intelligence 4(6), 742–763 (2020)

    Article  Google Scholar 

  11. Ferrer, A.J., Marquès, J.M., Jorba, J.: Towards the decentralised cloud: survey on approaches and challenges for mobile, ad hoc, and edge computing. ACM Comput. Surv. (CSUR) 51(6), 1–36 (2019)

    Article  Google Scholar 

  12. Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: vision and challenges. IEEE Internet Things J. 3(5), 637–646 (2016)

    Article  Google Scholar 

  13. Sriram, G.: Edge computing vs. cloud computing: an overview of big data challenges and opportunities for large enterprises. Int. Res. J. Mod. Eng. Technol. Sci. 4(1), 1331–1337 (2022)

  14. Yao, J., Zhang, S., Yao, Y., Wang, F., Ma, J., Zhang, J., Chu, Y., Ji, L., Jia, K., Shen, T., et al.: Edge-cloud polarization and collaboration: a comprehensive survey for ai. IEEE Trans. Knowl. Data, Eng (2022)

    Google Scholar 

  15. He, X., Lu, H., Du, M., Mao, Y., Wang, K.: Qoe-based task offloading with deep reinforcement learning in edge-enabled internet of vehicles. IEEE Trans. Intell. Transp. Syst. 22(4), 2252–2261 (2020)

    Article  Google Scholar 

  16. Xia, X., Chen, F., He, Q., Grundy, J., Abdelrazek, M., Jin, H.: Online collaborative data caching in edge computing. IEEE Trans. Parallel Distrib. Syst. 32(2), 281–294 (2020)

    Article  Google Scholar 

  17. Saputra, Y.M., Hoang, D.T., Nguyen, D.N., Dutkiewicz, E.: A novel mobile edge network architecture with joint caching-delivering and horizontal cooperation. IEEE Trans. Mob. Comput. 20(1), 19–31 (2019)

    Article  Google Scholar 

  18. Zhao, J., Sun, X., Li, Q., Ma, X.: Edge caching and computation management for real-time internet of vehicles: an online and distributed approach. IEEE Trans. Intell. Transp. Syst. 22(4), 2183–2197 (2020)

    Article  Google Scholar 

  19. Chien, W.-C., Weng, H.-Y., Lai, C.-F.: Q-learning based collaborative cache allocation in mobile edge computing. Futur. Gener. Comput. Syst. 102, 603–610 (2020)

    Article  Google Scholar 

  20. Thar, K., Tran, N.H., Oo, T.Z., Hong, C.S.: Deepmec: mobile edge caching using deep learning. IEEE Access 6, 78260–78275 (2018)

    Article  Google Scholar 

  21. Zhao, Y., Li, R., Wang, C., Wang, X., Leung, V.C.: Neighboring-aware caching in heterogeneous edge networks by actor-attention-critic learning. In: ICC 2021-IEEE International Conference on Communications, pp. 1–6. IEEE (2021)

  22. Wang, X., Wang, C., Li, X., Leung, V.C., Taleb, T.: Federated deep reinforcement learning for internet of things with decentralized cooperative edge caching. IEEE Internet Things J. 7(10), 9441–9455 (2020)

    Article  Google Scholar 

  23. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37(3), 50–60 (2020)

    Article  CAS  Google Scholar 

  24. Sanguanpuak, T., Niyato, D., Rajatheva, N., Latva-Aho, M.: Radio resource sharing and edge caching with latency constraint for local 5g operator: geometric programming meets stackelberg game. IEEE Trans. Mob. Comput. 20(2), 707–721 (2019)

    Article  Google Scholar 

  25. Fu, Y., Yu, Q., Quek, T.Q., Wen, W.: Revenue maximization for content-oriented wireless caching networks (cwcns) with repair and recommendation considerations. IEEE Trans. Wireless Commun. 20(1), 284–298 (2020)

    Article  Google Scholar 

  26. Li, Y., Hu, S., Li, G.: Cvc: a collaborative video caching framework based on federated learning at the edge. IEEE Trans. Netw. Serv, Manag (2021)

    Google Scholar 

  27. Daghero, F., Pagliari, D.J., Poncino, M.: Energy-efficient deep learning inference on edge devices. In: Advances in Computers vol. 122, pp. 247–301. Elsevier, ??? (2021)

  28. Zhong, S., Zhang, K., Bagheri, M., Burken, J.G., Gu, A., Li, B., Ma, X., Marrone, B.L., Ren, Z.J., Schrier, J., et al.: Machine learning: new ideas and tools in environmental science and engineering. Environ. Sci. Technol. 55(19), 12741–12754 (2021)

    CAS  PubMed  Google Scholar 

  29. Cao, K., Liu, Y., Meng, G., Sun, Q.: An overview on edge computing research. IEEE Access 8, 85714–85728 (2020)

    Article  Google Scholar 

  30. Osibo, B.K., Jin, Z., Ma, T., Marah, B.D., Zhang, C., Jin, Y.: An edge computational offloading architecture for ultra-low latency in smart mobile devices. Wireless Netw. 28(5), 2061–2075 (2022)

    Article  Google Scholar 

  31. Lei, M., Li, Q., Ge, X., Pandharipande, A.: Partially collaborative edge caching based on federated deep reinforcement learning. IIEEE Trans. Veh, Technol (2022)

    Google Scholar 

  32. Ricardo, G.I., Tuholukova, A., Neglia, G., Spyropoulos, T.: Caching policies for delay minimization in small cell networks with coordinated multi-point joint transmissions. IEEE/ACM Trans. Networking 29(3), 1105–1115 (2021)

    Article  Google Scholar 

  33. Li, L., Zhao, G., Blum, R.S.: A survey of caching techniques in cellular networks: research issues and challenges in content placement and delivery strategies. IEEE Commun. Surv. Tutor. 20(3), 1710–1732 (2018)

    Article  Google Scholar 

  34. Avrachenkov, K., Goseling, J., Serbetci, B.: A low-complexity approach to distributed cooperative caching with geographic constraints. Proceedings of the ACM on Measurement and Analysis of Computing Systems 1(1), 1–25 (2017)

    Article  Google Scholar 

  35. Ioannidis, S., Yeh, E.: Adaptive caching networks with optimality guarantees. IEEE/ACM Trans. Networking 26(2), 737–750 (2018)

    Article  Google Scholar 

  36. Rathore, S., Ryu, J.H., Sharma, P.K., Park, J.H.: Deepcachnet: a proactive caching framework based on deep learning in cellular networks. IEEE Netw. 33(3), 130–138 (2019)

    Article  Google Scholar 

  37. Blasco, P., Gündüz, D.: Learning-based optimization of cache content in a small cell base station. In: 2014 IEEE International Conference on Communications (ICC), pp. 1897–1903. IEEE (2014)

  38. Wan, Z., Li, Y.: Deep reinforcement learning-based collaborative video caching and transcoding in clustered and intelligent edge b5g networks. Wirel. Commun. Mob. Comput. 2020 (2020)

  39. Sun, Y., Peng, M., Mao, S.: Deep reinforcement learning-based mode selection and resource management for green fog radio access networks. IEEE Internet Things J. 6(2), 1960–1971 (2018)

    Article  Google Scholar 

  40. Wang, Z., Xu, H., Liu, J., Huang, H., Qiao, C., Zhao, Y.: Resource-efficient federated learning with hierarchical aggregation in edge computing. In: IEEE INFOCOM 2021-IEEE Conference on Computer Communications, pp. 1–10. IEEE (2021)

  41. Mills, J., Hu, J., Min, G.: Multi-task federated learning for personalised deep neural networks in edge computing. IEEE Trans. Parallel Distrib. Syst. 33(3), 630–641 (2021)

    Article  Google Scholar 

  42. Chen, J., Ran, X.: Deep learning with edge computing: a review. Proc. IEEE 107(8), 1655–1674 (2019)

    Article  Google Scholar 

  43. Li, C., Zhang, Y., Gao, X., Luo, Y.: Energy-latency tradeoffs for edge caching and dynamic service migration based on dqn in mobile edge computing. J. Parallel Distrib. Comput. 166, 15–31 (2022)

    Article  Google Scholar 

  44. Vallero, G., Deruyck, M., Meo, M., Joseph, W.: Base station switching and edge caching optimisation in high energy-efficiency wireless access network. Comput. Netw. 192, 108100 (2021)

    Article  Google Scholar 

  45. Yin, B., Chen, Z., Tao, M.: Joint user scheduling and resource allocation for federated learning over wireless networks. In: GLOBECOM 2020-2020 IEEE Global Communications Conference, pp. 1–6. IEEE (2020)

  46. Zheng, J., Li, K., Tovar, E., Guizani, M.: Federated learning for energy-balanced client selection in mobile edge computing. In: 2021 International Wireless Communications and Mobile Computing (IWCMC), pp. 1942–1947. IEEE (2021)

  47. Rong, Z., Rappaport, T.S.: Wireless Communications: Principles and Practice, Solutions Manual. Prentice Hall, ??? (1996)

  48. Li, C., Tang, J., Tang, H., Luo, Y.: Collaborative cache allocation and task scheduling for data-intensive applications in edge computing environment. Futur. Gener. Comput. Syst. 95, 249–264 (2019)

    Article  CAS  Google Scholar 

  49. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  CAS  PubMed  Google Scholar 

  50. Hasselt, H.: Double q-learning. IEEE Intell, Syst (2010)

    Google Scholar 

  51. Yang, Z., Chen, M., Saad, W., Hong, C.S., Shikh-Bahaei, M.: Energy efficient federated learning over wireless communication networks. IEEE Trans. Wireless Commun. 20(3), 1935–1949 (2020)

    Article  Google Scholar 

  52. Mao, Y., Zhang, J., Letaief, K.B.: Dynamic computation offloading for mobile-edge computing with energy harvesting devices. IEEE J. Sel. Areas Commun. 34(12), 3590–3605 (2016)

    Article  Google Scholar 

  53. Jaleel, A., Theobald, K.B., Steely, S.C., Jr., Emer, J.: High performance cache replacement using re-reference interval prediction (rrip). ACM SIGARCH computer architecture news 38(3), 60–71 (2010)

    Article  Google Scholar 

  54. Ahmed, M., Traverso, S., Giaccone, P., Leonardi, E., Niccolini, S.: Analyzing the performance of lru caches under non-stationary traffic patterns. arXiv:1301.4909 (2013)

  55. Rossi, D., Rossini, G.: Caching performance of content centric networks under multi-path routing (and more). Relatório técnico, Telecom ParisTech 2011, 1–6 (2011)

    Google Scholar 

  56. McMahan, H.B., Moore, E., Ramage, D., Arcas, B.A.: Federated learning of deep networks using model averaging. arXiv: Learning (2016)

  57. Xu, S., Liu, X., Guo, S., Qiu, X., Meng, L.: Mecc: a mobile edge collaborative caching framework empowered by deep reinforcement learning. IEEE Netw. 35(4), 176–183 (2021)

    Article  Google Scholar 

  58. Sun, C., Li, X., Wen, J., Wang, X., Han, Z., Leung, V.C.: Federated deep reinforcement learning for recommendation-enabled edge caching in mobile edge-cloud computing networks. IEEE J. Sel. Areas Commun. 41(3), 690–705 (2023)

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China, 62172442 and The Hunan Province Natural Science Foundation of China, 2020JJ5775

Author information

Authors and Affiliations

  1. School of Computer Science, Central South University, 932 South Lu Shan Road, Changsha, 410083, Hunan, China

    Xinyu Zhang, Zhigang Hu, Yang Liang, Hui Xiao, Aikun Xu & Meiguang Zheng

  2. School of Informatics, Hunan University of Chinese Medicine, 300 Scholars Road, Changsha, 410083, Hunan, China

    Yang Liang

  3. School of Big Data and Software Engineering, Chongqing University, 174 Shazheng Street, Chongqing, 400044, China

    Chuan Sun

Authors
  1. Xinyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhigang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aikun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Meiguang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chuan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Xinyu Zhang: Responsible for highlight, experiments, full-text writing, and submissions. Zhigang Hu: Responsible for checking experimental results, checking thesis writing and providing research grants, Final approval of the version to be published. Yang Liang: Revising it critically for important intellectual content, This includes background research, literature recommendations, confirmation of motivation for the article, etc. Hui Xiao: Integrity of any part of the work are appropriately investigated and resolved. Including corrections to mathematical equations, corrections to algorithms, corrections to research ideas Aikun Xu: Revising it critically for important intellectual content, includes help with coding, help with drawing of article diagrams, etc. Meiguang Zheng: Revising it critically for important intellectual content, and provide fund support. Chuan Sun: Revising it critically for important intellectual content, and providing coding assistance.

Corresponding author

Correspondence to Zhigang Hu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Hu, Z., Liang, Y. et al. A Federated Deep Reinforcement Learning-based Low-power Caching Strategy for Cloud-edge Collaboration. J Grid Computing 22, 21 (2024). https://doi.org/10.1007/s10723-023-09730-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10723-023-09730-6

Keywords

Search

Navigation