{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,10,2]],"date-time":"2024-10-02T04:01:10Z","timestamp":1727841670641},"reference-count":49,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,8,9]],"date-time":"2022-08-09T00:00:00Z","timestamp":1660003200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JSAN"],"abstract":"This paper studies the gradient-based adversarial attacks on cluster-based, heterogeneous, multi-agent, deep reinforcement learning (MADRL) systems with time-delayed data transmission. The structure of the MADRL system consists of various clusters of agents. The deep Q-network (DQN) architecture presents the first cluster\u2019s agent structure. The other clusters are considered as the environment of the first cluster\u2019s DQN agent. We introduce two novel observations in data transmission, termed on-time and time-delay observations. The proposed observations are considered when the data transmission channel is idle, and the data is transmitted on time or delayed. By considering the distance between the neighboring agents, we present a novel immediate reward function by appending a distance-based reward to the previously utilized reward to improve the MADRL system performance. We consider three types of gradient-based attacks to investigate the robustness of the proposed system data transmission. Two defense methods are proposed to reduce the effects of the discussed malicious attacks. We have rigorously shown the system performance based on the DQN loss and the team reward for the entire team of agents. Moreover, the effects of the various attacks before and after using defense algorithms are demonstrated. The theoretical results are illustrated and verified with simulation examples.<\/jats:p>","DOI":"10.3390\/jsan11030045","type":"journal-article","created":{"date-parts":[[2022,8,10]],"date-time":"2022-08-10T06:42:53Z","timestamp":1660113773000},"page":"45","source":"Crossref","is-referenced-by-count":3,"title":["Adversarial Attacks on Heterogeneous Multi-Agent Deep Reinforcement Learning System with Time-Delayed Data Transmission"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-7093-2693","authenticated-orcid":false,"given":"Neshat","family":"Elhami Fard","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Concordia University, Montreal, QC H3G 1M8, Canada"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-9345-8077","authenticated-orcid":false,"given":"Rastko R.","family":"Selmic","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Concordia University, Montreal, QC H3G 1M8, Canada"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,9]]},"reference":[{"key":"ref_1","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Canese, L., Cardarilli, G.C., Di Nunzio, L., Fazzolari, R., Giardino, D., Re, M., and Span\u00f2, S. (2021). Multi-agent reinforcement learning: A review of challenges and applications. Appl. Sci., 11.","DOI":"10.3390\/app11114948"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mousavi, S.S., Schukat, M., and Howley, E. (2016, January 21\u201322). Deep reinforcement learning: An overview. Proceedings of the SAI Intelligent Systems Conference, London, UK.","DOI":"10.1007\/978-3-319-56991-8_32"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","article-title":"Deep reinforcement learning: A brief survey","volume":"34","author":"Arulkumaran","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Fran\u00e7ois-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., and Pineau, J. (2018). An introduction to deep reinforcement learning. arXiv.","DOI":"10.1561\/9781680835397"},{"key":"ref_6","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_8","unstructured":"Chen, Z., Zhang, S., Doan, T.T., Maguluri, S.T., and Clarke, J.P. (2019). Performance of Q-learning with linear function approximation: Stability and finite-time analysis. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gao, Z., Gao, Y., Hu, Y., Jiang, Z., and Su, J. (2020, January 8\u201311). Application of deep q-network in portfolio management. Proceedings of the IEEE International Conference on Big Data Analytics (ICBDA), Xiamen, China.","DOI":"10.1109\/ICBDA49040.2020.9101333"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"113820","DOI":"10.1016\/j.eswa.2020.113820","article-title":"Multi-DQN: An ensemble of Deep Q-learning agents for stock market forecasting","volume":"164","author":"Carta","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"133653","DOI":"10.1109\/ACCESS.2019.2941229","article-title":"Q-learning algorithms: A comprehensive classification and applications","volume":"7","author":"Jang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2227","DOI":"10.1109\/TNNLS.2018.2806087","article-title":"Multisource transfer double DQN based on actor learning","volume":"29","author":"Pan","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.isatra.2020.11.030","article-title":"Secure consensus of multi-agent systems with redundant signal and communication interference via distributed dynamic event-triggered control","volume":"112","author":"Zhao","year":"2021","journal-title":"ISA Trans."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"109223","DOI":"10.1016\/j.automatica.2020.109223","article-title":"Periodic event-triggered output regulation for linear multi-agent systems","volume":"122","author":"Zheng","year":"2020","journal-title":"Automatica"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neucom.2021.02.040","article-title":"Adaptive event-triggered consensus control of linear multi-agent systems with cyber attacks","volume":"442","author":"Yuan","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2580","DOI":"10.1109\/TFUZZ.2020.3004009","article-title":"Dynamic event-triggered asynchronous control for nonlinear multi-agent systems based on TS fuzzy models","volume":"29","author":"Chen","year":"2020","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"6217","DOI":"10.1016\/j.jfranklin.2021.06.014","article-title":"Leaderless Consensus Control of Nonlinear Multi-agent Systems under Directed Topologies subject to Input Saturation using Adaptive Event-Triggered Mechanism","volume":"358","author":"Rehan","year":"2021","journal-title":"J. Frankl. Inst."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhou, B., Yang, Y., Li, L., and Hao, R. (2021). Leaderless and leader-following consensus of heterogeneous second-order multi-agent systems on time scales: An asynchronous impulsive approach. Int. J. Control, 1\u201311.","DOI":"10.1080\/00207179.2021.1917777"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2848","DOI":"10.1109\/TMC.2020.2990399","article-title":"Non-Uniform Time-Step Deep Q-Network for Carrier-Sense Multiple Access in Heterogeneous Wireless Networks","volume":"20","author":"Yu","year":"2021","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_21","unstructured":"Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., and Mukhopadhyay, D. (2018). Adversarial attacks and defences: A survey. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"14410","DOI":"10.1109\/ACCESS.2018.2807385","article-title":"Threat of adversarial attacks on deep learning in computer vision: A survey","volume":"6","author":"Akhtar","year":"2018","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1145\/3134599","article-title":"Making machine learning robust against adversarial inputs","volume":"61","author":"Goodfellow","year":"2018","journal-title":"Commun. ACM"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jpdc.2019.03.003","article-title":"The security of machine learning in an adversarial setting: A survey","volume":"130","author":"Wang","year":"2019","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"100199","DOI":"10.1016\/j.cosrev.2019.100199","article-title":"A taxonomy and survey of attacks against machine learning","volume":"34","author":"Pitropakis","year":"2019","journal-title":"Comput. Sci. Rev."},{"key":"ref_26","unstructured":"Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv."},{"key":"ref_27","unstructured":"Kurakin, A., Goodfellow, I., and Bengio, S. (2016). Adversarial machine learning at scale. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., and Li, J. (2018, January 18\u201322). Boosting adversarial attacks with momentum. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00957"},{"key":"ref_29","unstructured":"Kurakin, A., Goodfellow, I.J., and Bengio, S. (2016). Adversarial examples in the physical world. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1109\/OJITS.2021.3118972","article-title":"Adversarial Attacks and Defense in Deep Reinforcement Learning (DRL)-Based Traffic Signal Controllers","volume":"2","author":"Haydari","year":"2021","journal-title":"IEEE Open J. Intell. Transp. Syst."},{"key":"ref_31","unstructured":"Hussenot, L., Geist, M., and Pietquin, O. (2019, January 14). Manipulating Neural Policies with Adversarial Observations. Proceedings of the Real-World Sequential Decision Making Workshop, ICML, Long Beach, CA, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yuan, Z., Zhang, J., Jia, Y., Tan, C., Xue, T., and Shan, S. (2021, January 11\u201317). Meta gradient adversarial attack. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00765"},{"key":"ref_33","unstructured":"Metzen, J.H., Genewein, T., Fischer, V., and Bischoff, B. (2017). On detecting adversarial perturbations. arXiv."},{"key":"ref_34","unstructured":"Dong, Y., Su, H., Zhu, J., and Bao, F. (2017). Towards interpretable deep neural networks by leveraging adversarial examples. arXiv."},{"key":"ref_35","unstructured":"Tram\u00e8r, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., and McDaniel, P. (2017). Ensemble adversarial training: Attacks and defenses. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. (2016, January 22\u201326). Distillation as a defense to adversarial perturbations against deep neural networks. Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA.","DOI":"10.1109\/SP.2016.41"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Elhami Fard, N., and Selmic, R.R. (July, January 28). Time-delayed Data Transmission in Heterogeneous Multi-agent Deep Reinforcement Learning System. Proceedings of the 2022 30th Mediterranean Conference on Control and Automation (MED), Athens, Greece.","DOI":"10.1109\/MED54222.2022.9837194"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Mesbahi, M., and Egerstedt, M. (2010). Graph Theoretic Methods in Multiagent Networks, Princeton University Press.","DOI":"10.1515\/9781400835355"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wang, S., Diao, R., Lan, T., Wang, Z., Shi, D., Li, H., and Lu, X. (2020, January 3\u20136). A DRL-aided multi-layer stability model calibration platform considering multiple events. Proceedings of the 2020 IEEE Power & Energy Society General Meeting (PESGM), Virtual.","DOI":"10.1109\/PESGM41954.2020.9282022"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1711","DOI":"10.1007\/s40747-020-00138-3","article-title":"Improving ant colony optimization algorithm with epsilon greedy and Levy flight","volume":"7","author":"Liu","year":"2021","journal-title":"Complex Intell. Syst."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"115","DOI":"10.18196\/jrc.v3i2.13082","article-title":"Consensus of Multi-agent Reinforcement Learning Systems: The Effect of Immediate Rewards","volume":"3","author":"Selmic","year":"2022","journal-title":"J. Robot. Control (JRC)"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"103","DOI":"10.3389\/fnbot.2019.00103","article-title":"Constrained deep q-learning gradually approaching ordinary q-learning","volume":"13","author":"Ohnishi","year":"2019","journal-title":"Front. Neurorobot."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1109\/JSAC.2019.2904329","article-title":"Deep-reinforcement learning multiple access for heterogeneous wireless networks","volume":"37","author":"Yu","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_44","unstructured":"Ammouri, K. (2021, September 06). Deep Reinforcement Learning for Temperature Control in Buildings and Adversarial Attacks. Available online: https:\/\/www.diva-portal.org\/smash\/record.jsf?pid=diva2%3A1590898&dswid=-7284."},{"key":"ref_45","unstructured":"Yu, Y. (2019, August 26). CS-DLMA. Available online: https:\/\/github.com\/YidingYu\/CS-DLMA."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Abramson, N. (1970, January 17\u201319). The ALOHA system: Another alternative for computer communications. Proceedings of the Fall Joint Computer Conference, Houston, TX, USA.","DOI":"10.1145\/1478462.1478502"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1145\/205447.205451","article-title":"The ALOHA system","volume":"25","author":"Kuo","year":"1995","journal-title":"ACM SIGCOMM Comput. Commun. Rev."},{"key":"ref_48","unstructured":"Kuo, F.F. (1981). Computer Networks\u2013The ALOHA System, Hawaii University at Manoa Honolulu Department of Electrical Engineering. Technical report."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Jung, P. (2003). Time Division Multiple Access (TDMA). Wiley Encyclopedia of Telecommunications, John Wiley & Sons.","DOI":"10.1002\/0471219282.eot135"}],"container-title":["Journal of Sensor and Actuator Networks"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2224-2708\/11\/3\/45\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,1]],"date-time":"2024-10-01T03:09:42Z","timestamp":1727752182000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2224-2708\/11\/3\/45"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,9]]},"references-count":49,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["jsan11030045"],"URL":"http:\/\/dx.doi.org\/10.3390\/jsan11030045","relation":{},"ISSN":["2224-2708"],"issn-type":[{"type":"electronic","value":"2224-2708"}],"subject":[],"published":{"date-parts":[[2022,8,9]]}}}
