Event-Enhanced Multi-Modal Spiking Neural Network for Dynamic Obstacle Avoidance
Authors: 
Yang Wang, Bo Dong, Yuji Zhang, Yunduo Zhou, Haiyang Mei, Ziqi Wei, Xin YangAuthors Info & Claims
Yang Wang, Bo Dong, Yuji Zhang, Yunduo Zhou, Haiyang Mei, Ziqi Wei, Xin YangAuthors Info & ClaimsPages 3138 - 3148
Abstract
Autonomous obstacle avoidance is of vital importance for an intelligent agent such as a mobile robot to navigate in its environment. Existing state-of-the-art methods train a spiking neural network (SNN) with deep reinforcement learning (DRL) to achieve energy-efficient and fast inference speed in complex/unknown scenes. These methods typically assume that the environment is static while the obstacles in real-world scenes are often dynamic. The movement of obstacles increases the complexity of the environment and poses a great challenge to the existing methods. In this work, we approach robust dynamic obstacle avoidance twofold. First, we introduce the neuromorphic vision sensor (i.e., event camera) to provide motion cues complementary to the traditional Laser depth data for handling dynamic obstacles. Second, we develop an DRL-based event-enhanced multimodal spiking actor network (EEM-SAN) that extracts information from motion events data via unsupervised representation learning and fuses Laser and event camera data with learnable thresholding. Experiments demonstrate that our EEM-SAN outperforms state-of-the-art obstacle avoidance methods by a significant margin, especially for dynamic obstacle avoidance.
References
[1]
Ignacio Abad'ia, Francisco Naveros, Eduardo Ros, Richard R Carrillo, and Niceto R Luque. 2021. A cerebellar-based solution to the nondeterministic time delay problem in robotic control. Science Robotics, Vol. 6, 58 (2021), eabf2756.
[2]
Riku Arakawa and Shintaro Shiba. 2020. Exploration of reinforcement learning for event camera using car-like robots. arXiv preprint arXiv:2004.00801 (2020).
[3]
Hermann Blum, Alexander Dietmüller, Moritz Milde, Jörg Conradt, Giacomo Indiveri, and Yulia Sandamirskaya. 2017. A neuromorphic controller for a robotic vehicle equipped with a dynamic vision sensor. Robotics Science and Systems, RSS 2017 (2017).
[4]
Sander M Bohte, Joost N Kok, and Han La Poutre. 2002. Error-backpropagation in temporally encoded networks of spiking neurons. Neurocomputing (2002).
[5]
Xiang Cheng, Tielin Zhang, Shuncheng Jia, and Bo Xu. 2020. meta-dynamic neurons in spiking neural networks for spatio-temporal learning. arXiv preprint arXiv:2010.03140 (2020).
[6]
Travis DeWolf. 2021. Spiking neural networks take control. Science Robotics, Vol. 6, 58 (2021), eabk3268.
[7]
Jianchuan Ding, Bo Dong, Felix Heide, Yufei Ding, Yunduo Zhou, Baocai Yin, and Xin Yang. 2022. Biologically Inspired Dynamic Thresholds for Spiking Neural Networks. In Advances in Neural Information Processing Systems.
[8]
Davide Falanga, Suseong Kim, and Davide Scaramuzza. 2019. How fast is too fast? the role of perception latency in high-speed sense and avoid. IEEE Robotics and Automation Letters, Vol. 4, 2 (2019), 1884--1891.
[9]
C. Florensa, D. Held, M. Wulfmeier, and P. Abbeel. 2017. Reverse Curriculum Generation for Reinforcement Learning. (2017).
[10]
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. 2020. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence (2020).
[11]
Wulfram Gerstner. 1995. Time structure of the activity in neural network models. Physical review E (1995).
[12]
Wulfram Gerstner and Werner M Kistler. 2002. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press.
[13]
Shuncheng Jia, Ruichen Zuo, Tielin Zhang, Hongxing Liu, and Bo Xu. 2022. Motif-Topology and Reward-Learning Improved Spiking Neural Network for Efficient Multi-Sensory Integration. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 8917--8921.
[14]
Hiromichi Kamata, Yusuke Mukuta, and Tatsuya Harada. 2022. Fully spiking variational autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence.
[15]
S. Khattak, H. D. Nguyen, F. Mascarich, T. Dang, and K. Alexis. 2020. Complementary Multi-Modal Sensor Fusion for Resilient Robot Pose Estimation in Subterranean Environments. In IEEE International Conference on Unmanned Aircraft Systems 2020.
[16]
Taeyoon Kim, Suman Hu, Jaewook Kim, Joon Young Kwak, Jongkil Park, Suyoun Lee, Inho Kim, Jong-Keuk Park, and YeonJoo Jeong. 2021. Spiking neural network (snn) with memristor synapses having non-linear weight update. Frontiers in computational neuroscience, Vol. 15 (2021), 646125.
[17]
Youngeun Kim, Hyoungseob Park, Abhishek Moitra, Abhiroop Bhattacharjee, Yeshwanth Venkatesha, and Priyadarshini Panda. 2022. Rate Coding Or Direct Coding: Which One Is Better For Accurate, Robust, And Energy-Efficient Spiking Neural Networks?. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 71--75.
[18]
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
[19]
Nathan Koenig and Andrew Howard. 2004. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), Vol. 3. IEEE, 2149--2154.
[20]
P. Lichtsteiner, C. Posch, and T. Delbruck. 2008. A 128 X 128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor. IEEE Journal of Solid-State Circuits, Vol. 43, 2 (2008), 566--576.
[21]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
[22]
Qianhui Liu, Dong Xing, Lang Feng, Huajin Tang, and Gang Pan. 2022. Event-Based Multimodal Spiking Neural Network with Attention Mechanism. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).8922--8926.
[23]
Amarnath Mahadevuni and Peng Li. 2017. Navigating mobile robots to target in near shortest time using reinforcement learning with spiking neural networks. In International Joint Conference on Neural Networks (IJCNN). 2243--2250.
[24]
David Marr and Tomaso Poggio. 1979. A computational theory of human stereo vision. Proceedings of the Royal Society of London. Series B. Biological Sciences, Vol. 204, 1156 (1979), 301--328.
[25]
Haiyang Mei, Zuowen Wang, Xin Yang, Xiaopeng Wei, and Tobi Delbruck. 2023. Deep polarization reconstruction with PDAVIS events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[26]
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. 2022. Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12444--12453.
[27]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature, Vol. 518, 7540 (2015), 529--533.
[28]
Michael Montemerlo, Sebastian Thrun, Daphne Koller, Ben Wegbreit, et al. 2002. FastSLAM: A factored solution to the simultaneous localization and mapping problem. Aaai/iaai, Vol. 593598 (2002).
[29]
Elias Mueggler, Nathan Baumli, Flavio Fontana, and Davide Scaramuzza. 2015. Towards evasive maneuvers with quadrotors using dynamic vision sensors. In 2015 European Conference on Mobile Robots (ECMR). IEEE, 1--8.
[30]
Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, and Juan Nieto. 2018. Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters, Vol. 3, 4 (2018), 4423--4430.
[31]
Nitin Rathi and Kaushik Roy. 2018. STDP based unsupervised multimodal learning with cross-modal processing in spiking neural networks. IEEE Transactions on Emerging Topics in Computational Intelligence, Vol. 5, 1 (2018), 143--153.
[32]
Nitin Rathi and Kaushik Roy. 2020. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658 (2020).
[33]
Nikolaus Salvatore, Sami Mian, Collin Abidi, and Alan George. 2020. A Neuro-inspired Approach to Intelligent Collision Avoidance and Navigation. In IEEE/AIAA 39th Digital Avionics Systems Conference.
[34]
Nitin J Sanket, Chethan M Parameshwara, Chahat Deep Singh, Ashwin V Kuruttukulam, Cornelia Fermüller, Davide Scaramuzza, and Yiannis Aloimonos. 2020. Evdodgenet: Deep dynamic obstacle dodging with event cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 10651--10657.
[35]
Thorben Schoepe, Ella Janotte, Moritz B Milde, Olivier JN Bertrand, Martin Egelhaaf, and Elisabetta Chicca. 2021. Finding the gap: neuromorphic motion vision in cluttered environments. arXiv preprint arXiv:2102.08417 (2021).
[36]
Nicolas Skatchkovsky, Osvaldo Simeone, and Hyeryung Jang. 2021. Learning to time-decode in spiking neural networks through the information bottleneck. arXiv preprint arXiv:2106.01177 (2021).
[37]
Ioana Sporea and André Grüning. 2013. Supervised learning in multilayer spiking neural networks. Neural computation, Vol. 25, 2 (2013), 473--509.
[38]
Kenneth Stewart, Andreea Danielescu, Lazar Supic, Timothy Shea, and Emre Neftci. 2021. Gesture similarity analysis on event data using a hybrid guided variational auto encoder. arXiv preprint arXiv:2104.00165 (2021).
[39]
Wensheng Sun and Dennis L Barbour. 2017. Rate, not selectivity, determines neuronal population coding accuracy in auditory cortex. PLoS biology, Vol. 15, 11 (2017), e2002459.
[40]
T. Takahashi, I. Takeuchi, T. Koto, S. Tadokoro, and I. Noda. 2000. RoboCup-Rescue Disaster Simulator Architecture. In Robot Soccer World Cup.
[41]
Doron Tal and Eric L Schwartz. 1997. Computing with the leaky integrate-and-fire neuron: logarithmic computation and multiplication. Neural computation, Vol. 9, 2 (1997), 305--318.
[42]
Sameerah Talafha, Banafsheh Rekabdar, Christos Mousas, and Chinwe Ekenna. 2020. Biologically inspired sleep algorithm for variational auto-encoders. In International Symposium on Visual Computing. Springer, 54--67.
[43]
Guangzhi Tang, Neelesh Kumar, and Konstantinos P Michmizos. 2020a. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 6090--6097.
[44]
Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos P. Michmizos. 2020b. Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control. In Conference on Robot Learning.
[45]
Siqi Wang, Tee Hiang Cheng, and Meng-Hiot Lim. 2022. LTMD: Learning Improvement of Spiking Neural Networks with Learnable Thresholding Neurons and Moderate Dropout. In Advances in Neural Information Processing Systems.
[46]
Yuanda Wang, Haibo He, and Changyin Sun. 2018. Learning to navigate through complex dynamic environment with modular deep reinforcement learning. IEEE Transactions on Games, Vol. 10, 4 (2018), 400--412.
[47]
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. 2018. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, Vol. 12 (2018), 331.
[48]
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. 2022b. Attention spiking neural networks. arXiv preprint arXiv:2209.13929 (2022).
[49]
Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. 2022a. GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks. In Advances in Neural Information Processing Systems.
[50]
Yi Zeng, Tielin Zhang, and Bo Xu. 2017. Improving multi-layer spiking neural networks by incorporating brain-inspired rules. Science China Information Sciences, Vol. 60, 5 (2017), 1--11.
[51]
Duzhen Zhang, Tielin Zhang, Shuncheng Jia, and Bo Xu. 2022b. Multiscale Dynamic Coding improved Spiking Actor Network for Reinforcement Learning. AAAI (2022).
[52]
Haiwei Zhang, Jiqing Zhang, Bo Dong, Pieter Peers, Wenwei Wu, Xiaopeng Wei, Felix Heide, and Xin Yang. 2023. In the blink of an eye: Event-based emotion recognition. In ACM SIGGRAPH 2023 Conference Proceedings. 1--11.
[53]
Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang. 2022a. Spiking Transformers for Event-Based Single Object Tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[54]
Wenqi Zhang, Kai Zhao, Peng Li, Xiao Zhu, Yongliang Shen, Yanna Ma, Yingfeng Chen, and Weiming Lu. 2022c. A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigation. arXiv preprint arXiv:2207.11901 (2022).
[55]
Wenqi Zhang, Kai Zhao, Peng Li, Xiao Zhu, Faping Ye, Weijie Jiang, Huiqiao Fu, and Tao Wang. 2021. Learning to Navigate in a VUCA Environment: Hierarchical Multi-expert Approach. In IEEE/RSJ International Conference on Intelligent Robots and Systems. 9254--9261.
Index Terms
- Event-Enhanced Multi-Modal Spiking Neural Network for Dynamic Obstacle Avoidance
Recommendations
Real-time Trajectory Replanning for Dynamic Obstacles Avoidance for Robotics Manipulators
CompSysTech '22: Proceedings of the 23rd International Conference on Computer Systems and TechnologiesCollaborative robots are being used in various industrial tasks and need to collaborate with other robots or humans. An unforeseen obstacle may appear in their workspace at any time. Therefore, they must execute their motions without colliding with ...
Dynamic Obstacle Avoidance for Unmanned Aerial Vehicle Using Dynamic Vision Sensor
Artificial Neural Networks and Machine Learning – ICANN 2023AbstractObstacle avoidance in dynamic environments is a critical issue in unmanned aerial vehicle (UAV) applications. Current solutions rely on deep reinforcement learning (DRL), which requires significant computing power and energy and limits UAVs with ...
Obstacle avoidance in a dynamic environment: a collision cone approach
A novel collision cone approach is proposed as an aid to collision detection and avoidance between irregularly shaped moving objects with unknown trajectories. It is shown that the collision cone can be effectively used to determine whether collision ...
Comments
Information & Contributors
Information
Published In
October 2023
9913 pages
ISBN:9798400701085
DOI:10.1145/3581783
- General Chairs:
- Abdulmotaleb El Saddik,
- Tao Mei,
- Rita Cucchiara,
- Program Chairs:
- Marco Bertini,
- Diana Patricia Tobon Vallejo,
- Pradeep K. Atrey,
- M. Shamim Hossain
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Distinguished Young Scholars Funding of Dalian
- National Natural Science Foundation of China
- National Key Research and Development Program of China
Conference
MM '23
Sponsor:
MM '23: The 31st ACM International Conference on Multimedia
October 29 - November 3, 2023
Ottawa ON, Canada
Acceptance Rates
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 287Total Downloads
- Downloads (Last 12 months)237
- Downloads (Last 6 weeks)39
Reflects downloads up to 10 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in