{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,4,8]],"date-time":"2024-04-08T05:57:17Z","timestamp":1712555837622},"reference-count":44,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2023,9,13]],"date-time":"2023-09-13T00:00:00Z","timestamp":1694563200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"To accommodate the requirements of extensive coverage and ubiquitous connectivity in 6G communications, satellite plays a more significant role in it. As users and devices explosively grow, new multiple access technologies are called for. Among the new candidates, rate splitting multiple access (RSMA) shows great potential. Since satellites are power-limited, we investigate the energy-efficient resource allocation in the integrated satellite terrestrial network (ISTN)-adopting RSMA scheme in this paper. However, this non-convex problem is challenging to solve using conventional model-based methods. Because this optimization task has a quality of service (QoS) requirement and continuous action\/state space, we propose to use constrained soft actor-critic (SAC) to tackle it. This policy-gradient algorithm incorporates the Lagrangian relaxation technique to convert the original constrained problem into a penalized unconstrained one. The reward is maximized while the requirements are satisfied. Moreover, the learning process is time-consuming and unnecessary when little changes in the network. So, an on\u2013off mechanism is introduced to avoid this situation. By calculating the difference between the current state and the last one, the system will decide to learn a new action or take the last one. The simulation results show that the proposed algorithm can outperform other benchmark algorithms in terms of energy efficiency while satisfying the QoS constraint. In addition, the time consumption is lowered because of the on\u2013off design.<\/jats:p>","DOI":"10.3390\/s23187859","type":"journal-article","created":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T14:09:22Z","timestamp":1694700562000},"page":"7859","source":"Crossref","is-referenced-by-count":1,"title":["Constrained DRL for Energy Efficiency Optimization in RSMA-Based Integrated Satellite Terrestrial Network"],"prefix":"10.3390","volume":"23","author":[{"given":"Qingmiao","family":"Zhang","sequence":"first","affiliation":[{"name":"National Key Laboratory of Science and Technology on Communications, University of Electronic Science and Technology of China, Chengdu 611731, China"}]},{"given":"Lidong","family":"Zhu","sequence":"additional","affiliation":[{"name":"National Key Laboratory of Science and Technology on Communications, University of Electronic Science and Technology of China, Chengdu 611731, China"}]},{"given":"Yanyan","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer Science and Information Engineering, Xiamen Institute of Technology, Xiamen 361021, China"}]},{"given":"Shan","family":"Jiang","sequence":"additional","affiliation":[{"name":"China Mobile (Jiangxi) Communications Group Co., Ltd., Yichun 336000, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1109\/COMST.2023.3249835","article-title":"On the Road to 6G: Visions, Requirements, Key Technologies, and Testbeds","volume":"25","author":"Wang","year":"2023","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Xiao, Z., and Yang, J. (2022). LEO Satellite Access Network (LEO-SAN) towards 6G: Challenges and Approaches. IEEE Wirel. Commun. Lett., 1\u20138.","DOI":"10.1109\/MWC.011.2200310"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1037","DOI":"10.1109\/JSAC.2022.3145234","article-title":"Evolution of NOMA Toward Next Generation Multiple Access (NGMA) for 6G","volume":"40","author":"Liu","year":"2022","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Yang, Z., Shi, J., Li, Z., Chen, M., and Xu, W. (2020, January 7\u201311). Energy Efficient Rate Splitting Multiple Access (RSMA) with Reconfigurable Intelligent Surface. Proceedings of the 2020 IEEE International Conference on Communications Workshops (ICC Workshops), Dublin, Ireland.","DOI":"10.1109\/ICCWorkshops49005.2020.9145189"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"5988","DOI":"10.1109\/TCOMM.2021.3091133","article-title":"Optimization of Rate Allocation and Power Control for Rate Splitting Multiple Access (RSMA)","volume":"9","author":"Yang","year":"2021","journal-title":"IEEE Trans. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/TIT.1981.1056307","article-title":"A New Achievable Rate Region for the Interference Channel","volume":"27","author":"Han","year":"1981","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6227","DOI":"10.1109\/TSP.2016.2591501","article-title":"Robust Transmission in Downlink Multiuser MISO Systems: A Rate-Splitting Approach","volume":"64","author":"Joudeh","year":"2016","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_8","first-page":"4611","article-title":"A Rate Splitting Strategy for Massive MIMO With Imperfect CSIT","volume":"15","author":"Dai","year":"2016","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1109\/LWC.2019.2954518","article-title":"Rate-Splitting Unifying SDMA, OMA, NOMA, and Multicasting in MISO Broadcast Channel: A Simple Two-User Rate Analysis","volume":"9","author":"Clerckx","year":"2020","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"100","DOI":"10.23919\/JCC.2022.07.009","article-title":"Deep Unfolding for Cooperative Rate Splitting Multiple Access in Hybrid Satellite Terrestrial Networks","volume":"19","author":"Zhang","year":"2022","journal-title":"China Commun."},{"key":"ref_11","first-page":"100","article-title":"Rate Splitting Multiple Access for Sum-Rate Maximization in IRS Aided Uplink Communications","volume":"22","author":"Katwe","year":"2023","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1355","DOI":"10.1109\/TVT.2022.3206808","article-title":"Max-Min Fairness Precoder Design for Rate-Splitting Multiple Access: Impact of Imperfect Channel Knowledge","volume":"72","author":"Lee","year":"2023","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1109\/LWC.2022.3221526","article-title":"Max\u2013Min Fairness Beamforming With Rate-Splitting Multiple Access: Optimization Without a Toolbox","volume":"12","author":"Kim","year":"2023","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4379","DOI":"10.1109\/TWC.2021.3129881","article-title":"Full-Duplex Cooperative Rate-Splitting for Multigroup Multicast With SWIPT","volume":"21","author":"Li","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1706","DOI":"10.1109\/JSAC.2022.3145818","article-title":"Energy Efficiency Optimization for Rate-Splitting Multiple Access-Based Indoor Visible Light Communication Networks","volume":"40","author":"Xing","year":"2022","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Katwe, M., and Singh, K. (2022, January 4\u20138). Rate Splitting Multiple Access for Energy Efficient RIS-aided Multi-user Short-Packet Communications. Proceedings of the 2022 IEEE Globecom Workshops (GC Wkshps), Janeiro, Brazil.","DOI":"10.1109\/GCWkshps56602.2022.10008582"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"129358","DOI":"10.1109\/ACCESS.2021.3113501","article-title":"Deep Reinforcement Learning for Energy-Efficient Multi-Channel Transmissions in 5G Cognitive HetNets: Centralized, Decentralized and Transfer Learning Based Solutions","volume":"9","author":"Giannopoulos","year":"2021","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"5417","DOI":"10.1109\/ACCESS.2023.3236980","article-title":"Using Reinforcement Learning to Reduce Energy Consumption of Ultra-Dense Networks With 5G Use Cases Requirements","volume":"11","author":"Malta","year":"2023","journal-title":"IEEE Access"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"9060","DOI":"10.1109\/JIOT.2021.3118322","article-title":"Reinforcement Learning for Energy-Efficient Trajectory Design of UAVs","volume":"9","author":"Arani","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"856","DOI":"10.1109\/TVT.2021.3128513","article-title":"Energy-Efficient Deep Reinforcement Learning Assisted Resource Allocation for 5G-RAN Slicing","volume":"71","author":"Azimi","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Thantharate, A., Tondwalkar, A.V., Beard, C., and Kwasinski, A. (2022). ECO6G: Energy and Cost Analysis for Network Slicing Deployment in Beyond 5G Networks. Sensors, 22.","DOI":"10.3390\/s22228614"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"El-Fouly, F.H., Khedr, A.Y., Sharif, M.H., Alreshidi, E.J., Yadav, K., Kusetogullari, H., and Ramadan, R.A. (2022). ERCP: Energy-Efficient and Reliable-Aware Clustering Protocol for Wireless Sensor Networks. Sensors, 22.","DOI":"10.3390\/s22228950"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"68200","DOI":"10.1109\/ACCESS.2023.3291915","article-title":"Pizzza: A Joint Sector Shape and Minimum Spanning Tree-Based Clustering Scheme for Energy Efficient Routing in Wireless Sensor Networks","volume":"11","author":"Nasirian","year":"2023","journal-title":"IEEE Access"},{"key":"ref_24","unstructured":"Tessler, C., and Mankowitz, D.J. (2018). Reward Constrained Policy Optimization. arXiv."},{"key":"ref_25","first-page":"8378","article-title":"Natural Policy Gadient Primal-dual Method for Constrained Markov Decision Processes","volume":"33","author":"Ding","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_26","unstructured":"Achiam, J., and Held, D. (2017, January 6\u201311). Constrained policy optimization. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, NSW, Australia."},{"key":"ref_27","unstructured":"Yang, T.Y., and Rosca, J. (2020). Projection-based Constrained Policy Optimization. arXiv."},{"key":"ref_28","unstructured":"Liu, Y., and Ding, J. (March, January 22). IPO: Interior-point Policy Optimization under Constraints. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Ding, J. (2021, January 15\u201318). CLARA: A Constrained Reinforcement Learning Based Resource Allocation Framework for Network Slicing. Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA.","DOI":"10.1109\/BigData52589.2021.9671840"},{"key":"ref_30","first-page":"1414","article-title":"Constrained Soft Actor-Critic for Energy-Aware Trajectory Design in UAV-Aided IoT Networks","volume":"11","author":"Zhou","year":"2022","journal-title":"Electronics"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Qi, Q., and Lin, W. (2022). Augmented Lagrangian-Based Reinforcement Learning for Network Slicing in IIoT. Electronics, 11.","DOI":"10.3390\/electronics11203385"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3270","DOI":"10.1109\/TPWRS.2020.2987292","article-title":"Real-Time Optimal Power Flow: A Lagrangian Based Deep Reinforcement Learning Approach","volume":"35","author":"Yan","year":"2020","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2707","DOI":"10.1109\/TCOMM.2022.3142290","article-title":"Performance Analysis of Satellite Communication System Under the Shadowed-Rician Fading: A Stochastic Geometry Approach","volume":"70","author":"Jung","year":"2022","journal-title":"IEEE Trans. Commun."},{"key":"ref_34","unstructured":"Gradshteyn, S., and Ryzhik, I.M. (2007). Table of Integrals, Series and Products, Academic Press. [7th ed.]."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1016\/j.ymssp.2019.01.057","article-title":"Sensor placement algorithm for structural health monitoring with redundancy elimination model based on sub-clustering strategy","volume":"124","author":"Yang","year":"2019","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"108386","DOI":"10.1016\/j.ymssp.2021.108386","article-title":"A novel load-dependent sensor placement method for model updating based on time-dependent reliability optimization considering multi-source uncertainties","volume":"165","author":"Yang","year":"2022","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"109173","DOI":"10.1016\/j.ymssp.2022.109173","article-title":"A novel two-step strategy of non-probabilistic multi-objective optimization for load-dependent sensor placement with interval uncertainties","volume":"176","author":"Yang","year":"2022","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1835","DOI":"10.1109\/LWC.2021.3083267","article-title":"Deep learning-based channel prediction for LEO satellite massive MIMO communication system","volume":"10","author":"Zhang","year":"2021","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"0951","DOI":"10.1016\/j.ress.2023.109382","article-title":"Reliability-constrained optimal attitude-vibration control for rigid-flexible coupling satellite using interval dimension-wise analysis","volume":"237","author":"Yang","year":"2023","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1270","DOI":"10.1016\/j.ast.2023.108406","article-title":"Uncertain optimal attitude control for space power satellite based on interval Riccati equation with non-probabilistic time-dependent reliability","volume":"139","author":"Yang","year":"2023","journal-title":"Aerosp. Sci. Technol."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Li, J., and Fridovich-Keil, D. (2021, January 14\u201317). Augmented Lagrangian Method for Instantaneously Constrained Reinforcement Learning Problems. Proceedings of the 2021 60th IEEE Conference on Decision and Control (CDC), Austin, TX, USA.","DOI":"10.1109\/CDC45484.2021.9683088"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1286","DOI":"10.1137\/060654797","article-title":"On Augmented Lagrangian Methods with General Lower-level Constraints","volume":"18","author":"Andreani","year":"2008","journal-title":"SIAM J. Optim."},{"key":"ref_43","unstructured":"Haarnoja, T., and Zhou, A. (2018, January 10\u201315). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2820","DOI":"10.1109\/LWC.2021.3118441","article-title":"Optimal Power Allocation for Rate Splitting Communications With Deep Reinforcement Learning","volume":"10","author":"Hieu","year":"2021","journal-title":"IEEE Wirel. Commun. Lett."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7859\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T14:22:11Z","timestamp":1694701331000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/18\/7859"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,13]]},"references-count":44,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23187859"],"URL":"https:\/\/doi.org\/10.3390\/s23187859","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,13]]}}}
