iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://doi.org/10.1007/s00422-023-00974-9
Stakes of neuromorphic foveation: a promising future for embedded event cameras | Biological Cybernetics Skip to main content
Log in

Stakes of neuromorphic foveation: a promising future for embedded event cameras

  • Original Article
  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

Foveation can be defined as the organic action of directing the gaze towards a visual region of interest to acquire relevant information selectively. With the recent advent of event cameras, we believe that taking advantage of this visual neuroscience mechanism would greatly improve the efficiency of event data processing. Indeed, applying foveation to event data would allow to comprehend the visual scene while significantly reducing the amount of raw data to handle. In this respect, we demonstrate the stakes of neuromorphic foveation theoretically and empirically across several computer vision tasks, namely semantic segmentation and classification. We show that foveated event data have a significantly better trade-off between quantity and quality of the information conveyed than high- or low-resolution event data. Furthermore, this compromise extends even over fragmented datasets. Our code is publicly available online at: https://github.com/amygruel/FoveationStakes_DVS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Code Availability

The code is publicly available online at: https://github.com/amygruel/FoveationStakes_DVS.

Notes

  1. https://github.com/SensorsINI/ddd20-utils.

References

  • Albada S, Rowley A, Senk J, Hopkins M, Schmidt M, Stokes A, Lester D, Diesmann M, Furber S (2018) Performance comparison of the digital neuromorphic hardware spinnaker and the neural network simulation software nest for a full-scale cortical microcircuit model. Front Neurosci 12:291

    Article  PubMed  PubMed Central  Google Scholar 

  • Alonso I, Murillo A (2019) EV-SegNet: semantic segmentation for event-based Cameras. In: 2019 IEEE, CVF conference on computer vision and pattern recognition workshops (CVPRW)

  • Amir A, Taba B, Berg D, Melano, T, McKinstry J, Di Nolfo C, Nayak T, Andreopoulos A, Garreau G, Mendoza M et al (2017) A low power, fully event-based gesture recognition system. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7243–7252

  • Araujo H, Dias J (1997) An introduction to the log-polar mapping. In: Proceedings II workshop on cybernetic vision, vol 1, pp 139–144

  • Bear M et al (2007) The human eye. Neurosciences, exploring the brain. Wolters Kluwer, Philadelphia

    Google Scholar 

  • Binas J, Neil D, Liu S-C, Delbruck T (2017) DDD17: end-to-end Davis driving dataset. arXiv:1711.01458 [cs]

  • Dampfhoffer M, Mesquida T, Valentian A, Anghel L (2022) Are SNNs really more energy-efficient than ANNs? An in-depth hardware-aware study. IEEE Trans Emerg Top Comput Intell 2022:1–11

    Google Scholar 

  • D’Angelo G, Janotte E, Schoepe T, O’Keeffe J, Milde M, Chicca E, Bartolozzi C (2020) Event-based eccentric motion detection exploiting time difference encoding. Front Neurosci 14:451

    Article  PubMed  PubMed Central  Google Scholar 

  • Daucé E, Albiges P, Perrinet LU (2020) A dual foveal-peripheral visual processing model implements efficient saccade selection. J Vis 20(8):22–22

    Article  PubMed Central  Google Scholar 

  • Daucé E, Perrinet L (2020) Visual search as active inference. In: Verbelen T, Lanillos P, Buckley CL, De Boom C (eds) Active inference, communications in computer and information science. Springer International Publishing, Berlin, pp 165–178

    Google Scholar 

  • Davies M et al (2018) Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38(1):82–99

    Article  Google Scholar 

  • Davison AP, Brüderle D, Eppler J, Kremkow J, Muller E, Pecevski D, Perrinet L, Yger P (2009) PyNN: a common interface for neuronal network simulators. Front Neuroinform 2:388

    Google Scholar 

  • Delbrück T, Graca R, Paluch M (2021) Feedback control of event cameras. CoRRarXiv:2105.00409

  • Finateu T, Niwa A, Matolin D, Tsuchimoto K, Mascheroni A, Reynaud E, Mostafalu P, Brady FT, Chotard L, Legoff F, Takahashi H, Wakabayashi H, Oike Y, Posch C (2020) 5.10 a 1280 \(\times \) 720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86 \(\mu \) m pixels, 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline. In: 2020 IEEE international solid- state circuits conference—(ISSCC), pp 112–114

  • Furber S, Bogdan P (2020) Spinnaker–a spiking neural network architecture. NOW Publishers INC, Hanover

    Book  Google Scholar 

  • Gehrig D, Scaramuzza D (2022) Are high-resolution cameras really needed?. arXiv

  • Geisler WS, Perry JS (1998) Real-time foveated multiresolution system for low-bandwidth video communication. In: Rogowitz BE, Pappas TN (eds) Human vision and electronic imaging III, vol 3299. International Society for Optics and Photonics, SPIE, pp 294–305

  • Gewaltig M-O, Diesmann M (2007) Nest (neural simulation tool)

  • Ghosh R, Gupta A, Silva AN, Soares A, Thakor NV (2019) Spatiotemporal filtering for event-based action recognition. CoRRarXiv:1903.07067

  • Grimaldi A, Boutin V, Ieng S-H, Benosman R, Perrinet L (2022) A robust event-driven approach to always-on object recognition

  • Gruel A, Hareb D, Martinet J, Linares-Barranco B, Serrano-Gotarredona T (2022a) Neuromorphic foveation applied to semantic segmentation. In: NeuroVision: what can computer vision learn from visual neuroscience? A CVPR 2022 Workshop, New Orleans, United States

  • Gruel A, Martinet J (2021) Bio-inspired visual attention for silicon retinas based on spiking neural networks applied to pattern classification. In: 2021 international conference on content-based multimedia indexing (CBMI)

  • Gruel A, Martinet J, Magno M (2023) Simultaneous neuromorphic selection of multiple salient objects for event vision. In: 2023 international joint conference on neural networks (IJCNN)

  • Gruel A, Martinet J, Serrano-Gotarredona T, Linares-Barranco B (2022b) Event data downscaling for embedded computer vision. In: Proceedings of the 17th international joint conference on computer vision, imaging and computer graphics theory and applications (VISAPP)

  • Gruel A, Vitale A, Martinet J, Magno M (2022c) Neuromorphic event-based spatio-temporal attention using adaptive mechanisms. In: 2022 IEEE 4th international conference on artificial intelligence circuits and systems (AICAS)

  • Guo M, Huang J, Chen S (2017) Live demonstration: a 768 \(\times \) 640 pixels 200meps dynamic vision sensor. In: 2017 IEEE international symposium on circuits and systems (ISCAS)

  • Hao Q, Tao Y, Cao J, Tang M, Cheng Y, Zhou D, Ning Y, Bao C, Cui H (2021) Retina-like imaging and its applications: a brief review. Appl Sci 11(15):7058

    Article  CAS  Google Scholar 

  • Kubendran R, Paul A, Cauwenberghs G (2021) A 256 \(\times \) 256 6.3pj/pixel-event query-driven dynamic vision sensor with energy-conserving row-parallel event scanning. In: 2021 IEEE custom integrated circuits conference (CICC), pp 1–2

  • Lagorce X, Orchard G, Galluppi F, Shi BE, Benosman RB (2016) Hots: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans Pattern Anal Mach Intell 39(7):1346–1359

    Article  Google Scholar 

  • Land MF (2018) Eyes to see: the astonishing variety of vision in nature. Oxford University Press, Oxford

    Google Scholar 

  • Li C, Longinotti L, Corradi F, Delbruck T (2019) A 132 by 104 10 micro m-pixel 250 micro w 1kefps dynamic vision sensor with pixel-parallel noise and spatial redundancy suppression. In: 2019 symposium on VLSI circuits, pp C216–C217

  • Lichtsteiner P, Posch C, Delbruck T (2008) A 128 \(\times \) 128 120 dB 15 \(\mu \)s latency asynchronous temporal contrast vision sensor. IEEE J Solid State Circuit 43(2):566–576

  • Maass W (1997) Networks of spiking neurons. Neural Netw 10(9):1659–1671

    Article  Google Scholar 

  • Markram H, Wang Y, Tsodyks M (1998) Differential signaling via the same axon of neocortical pyramidal neurons. Proc Natl Acad Sci USA 95:5323–5328

  • Martinet J, Lablack A, Lew S, Djeraba C (2009) Gaze based quality assessment of visual media understanding. In: IEEE Pacific-Rim symposium on image and video technology-CVIM’09

  • Paugam-Moisy H, Bohte SM (2012) Computing with spiking neuron networks. Handbook of natural computing. Springer-Verlag, Berlin

    Google Scholar 

  • Pehle C, Pedersen JE (2021) Norse—a deep learning library for spiking neural networks. Documentation: https://norse.ai/docs/

  • Posch C, Serrano-Gotarredona T, Linares-Barranco B, Delbruck T (2014) Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE 102(10):1470–1484

  • Pramod R, Katti H, Arun S (2022) Human peripheral blur is optimal for object recognition. Vis Res 200:108083

    Article  CAS  PubMed  Google Scholar 

  • Rizzo C, Schuman CD, Plank JS (2023) Neuromorphic downsampling of event-based camera output. In: Kudithipudi D, Frenkel C, Cardwell S, Aimone JB (eds) Neuro-inspired computational elements conference, NICE2023, San Antonio, TX, USA, 11–14 Apr, 2023. ACM, pp 26–34

  • Sarvaiya JN, Patnaik S, Bombaywala S (2009) Image registration using log-polar transform and phase correlation. In: IEEE region 10 annual international conference, proceedings/TENCON, pp 1–5

  • Serrano-Gotarredona T, Faramarzi F, Linares-Barranco B (2022) Electronically foveated dynamic vision sensor. In: 2022 IEEE international conference on omni-layer intelligent systems (COINS), pp 1–5

  • Suh Y, Choi S, Ito M, Kim J, Lee Y, Seo J, Jung H, Yeo D-H, Namgung S, Bong J, Yoo S, Shin S-H, Kwon D, Kang P, Kim S, Na H, Hwang K, Shin C, Kim J-S, Park P KJ, Kim J, Ryu H, Park Y (2020) A 1280 \(\times \) 960 dynamic vision sensor with a 4.95-micro m pixel pitch and motion artifact minimization. In: 2020 IEEE international symposium on circuits and systems (ISCAS), pp. 1–5

  • Traver VJ, Pla F (2003) Designing the lattice for log-polar images. Discrete geometry for computer imagery. Springer, Berlin, pp 164–173

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was supported by the European Union’s ERA-NET CHIST-ERA 2018 research and innovation programme under grant agreement ANR-19-CHR3-0008. The authors are grateful to the OPAL infrastructure from Université Côte d’Azur for providing resources and support.

Funding

This work was supported by the European Union’s ERA-NET CHIST-ERA 2018 research and innovation programme under Grant Agreement ANR-19-CHR3-0008.

Author information

Authors and Affiliations

Authors

Contributions

The authors TS-G, JM, AG and BL-B contributed to the conceptualisation and methodology design of the study. The project coordination and administration were handled by AG. JM and Laurent Perrinet carried out the funding acquisition and supervision. Formal analysis and investigation were performed by AG, DH and AG. Results visualisation and presentation were realised by AG. The first draft of the manuscript was written by AG, DH and JM; AG, LP and TS-G added to a second draft by reviewing and editing. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Amélie Gruel.

Ethics declarations

Conflict of interest

The authors declares that they have no conflict of interest.

Consent for publication

Not applicable.

Ethical approval

Not applicable.

Additional information

Communicated by Benjamin Lindner

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is published as part of the Special Issue on “What can Computer Vision learn from Visual Neuroscience?".

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gruel, A., Hareb, D., Grimaldi, A. et al. Stakes of neuromorphic foveation: a promising future for embedded event cameras. Biol Cybern 117, 389–406 (2023). https://doi.org/10.1007/s00422-023-00974-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-023-00974-9

Keywords

Navigation