{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T16:40:17Z","timestamp":1721925617138},"reference-count":38,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T00:00:00Z","timestamp":1636070400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006701","name":"Umm Al-Qura University","doi-asserted-by":"crossref","award":["19- ENG-1-01-0015"],"id":[{"id":"10.13039\/501100006701","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"Image memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learning architecture called ResMem-Net that is a hybrid of LSTM and CNN that uses information from the hidden layers of the CNN to compute the memorability score of an image. The intermediate layers are important for predicting the output because they contain information about the intrinsic properties of the image. The proposed architecture automatically learns visual emotions and saliency, shown by the heatmaps generated using the GradRAM technique. We have also used the heatmaps and results to analyze and answer one of the most important questions in image memorability: \u201cWhat makes an image memorable?\u201d. The model is trained and evaluated using the publicly available Large-scale Image Memorability dataset (LaMem) from MIT. The results show that the model achieves a rank correlation of 0.679 and a mean squared error of 0.011, which is better than the current state-of-the-art models and is close to human consistency (p<\/jats:italic>\u00a0=\u00a00.68). The proposed architecture also has a significantly low number of parameters compared to the state-of-the-art architecture, making it memory efficient and suitable for production.<\/jats:p>","DOI":"10.7717\/peerj-cs.767","type":"journal-article","created":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T10:26:10Z","timestamp":1636107970000},"page":"e767","source":"Crossref","is-referenced-by-count":4,"title":["ResMem-Net: memory based deep CNN for image memorability estimation"],"prefix":"10.7717","volume":"7","author":[{"given":"Arockia","family":"Praveen","sequence":"first","affiliation":[{"name":"Phosphene AI, Madurai, India"}]},{"given":"Abdulfattah","family":"Noorwali","sequence":"additional","affiliation":[{"name":"Umm Al-Qura University, Makkah, Saudi Arabia"}]},{"given":"Duraimurugan","family":"Samiayya","sequence":"additional","affiliation":[{"name":"Optisol Business Solutions, Chennai, India"}]},{"given":"Mohammad","family":"Zubair Khan","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Taibah University, Medina, Saudi Arabia"}]},{"given":"Durai Raj","family":"Vincent P M","sequence":"additional","affiliation":[{"name":"School of Information Technology and Engineering, Vellore Institute of Technology, Vellore, Tamilnadu, India"}]},{"given":"Ali Kashif","family":"Bashir","sequence":"additional","affiliation":[{"name":"The Manchester Metropolitan University, Manchester, United Kingdom"}]},{"given":"Vinoth","family":"Alagupandi","sequence":"additional","affiliation":[{"name":"Optisol Business Solutions, Chennai, India"}]}],"member":"4443","published-online":{"date-parts":[[2021,11,5]]},"reference":[{"issue":"3","key":"10.7717\/peerj-cs.767\/ref-1","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1037\/pspa0000193","article-title":"Attitudes from mere co-occurrences are guided by differentiation","volume":"119","author":"Alves","year":"2020","journal-title":"Journal of Personality and Social Psychology"},{"key":"10.7717\/peerj-cs.767\/ref-2","doi-asserted-by":"publisher","first-page":"107408","DOI":"10.1016\/j.neuropsychologia.2020.107408","article-title":"The resiliency of image memorability: a predictor of memory separate from attention and priming","volume":"141","author":"Bainbridge","year":"2020","journal-title":"Neuropsychologia"},{"key":"10.7717\/peerj-cs.767\/ref-3","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/j.neuroimage.2017.01.063","article-title":"Memorability: a stimulus-driven perceptual neural signature distinctive from memory","volume":"149","author":"Bainbridge","year":"2017","journal-title":"NeuroImage"},{"issue":"24","key":"10.7717\/peerj-cs.767\/ref-4","doi-asserted-by":"publisher","first-page":"35511","DOI":"10.1007\/s11042-019-08202-y","article-title":"Multiple instance learning based deep CNN for image memorability prediction","volume":"78","author":"Basavaraju","year":"2019","journal-title":"Multimedia Tools and Applications"},{"key":"10.7717\/peerj-cs.767\/ref-5","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1145\/2964284.2967269","article-title":"Deep learning for image memorability prediction: the emotional bias","author":"Baveye","year":"2016"},{"key":"10.7717\/peerj-cs.767\/ref-6","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1186\/s40537-020-00335-4","article-title":"Exploring the efficacy of transfer learning in mining image-based software artifacts","volume":"7","author":"Best","year":"2020","journal-title":"Journal of Big Data"},{"key":"10.7717\/peerj-cs.767\/ref-7","doi-asserted-by":"publisher","first-page":"998","DOI":"10.3758\/s13421-020-01105-6","article-title":"The effect of intrinsic image memorability on recollection and familiarity","volume":"49","author":"Broers","year":"2021","journal-title":"Memory & Cognition"},{"key":"10.7717\/peerj-cs.767\/ref-8","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.cageo.2019.04.006","article-title":"Towards a robust parameterization for conditioning facies models using deep variational autoencoders and ensemble smoother","volume":"128","author":"Canchumuni","year":"2019","journal-title":"Computers & Geosciences"},{"key":"10.7717\/peerj-cs.767\/ref-9","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1016\/j.imavis.2015.07.001","article-title":"Predicting memorability of images using attention-driven spatial pooling and image semantics","volume":"42(C)","author":"Celikkale","year":"2015","journal-title":"Image and Vision Computing"},{"key":"10.7717\/peerj-cs.767\/ref-10","author":"Cho","year":"2014","journal-title":"Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation"},{"key":"10.7717\/peerj-cs.767\/ref-11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/QoMEX48832.2020.9123102","article-title":"Can visual scanpath reveal personal image memorability? Investigation of HMM tools for gaze patterns analysis","author":"Ellahi","year":"2020"},{"key":"10.7717\/peerj-cs.767\/ref-12","first-page":"6363","article-title":"Amnet: memorability estimation with attention","author":"Fajtl","year":"2018"},{"key":"10.7717\/peerj-cs.767\/ref-13","doi-asserted-by":"publisher","first-page":"1481","DOI":"10.3758\/s13421-019-00954-0","article-title":"Visual short-term memory capacity predicts the \u201cbandwidth\u201d of visual long-term memory encoding","volume":"47","author":"Fukuda","year":"2019","journal-title":"Memory & Cognition"},{"issue":"10","key":"10.7717\/peerj-cs.767\/ref-14","doi-asserted-by":"publisher","first-page":"2222","DOI":"10.1109\/TNNLS.2016.2582924","article-title":"LSTM: A search space odyssey","volume":"28","author":"Greff","year":"2016","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"10.7717\/peerj-cs.767\/ref-15","doi-asserted-by":"publisher","first-page":"984","DOI":"10.1080\/09658211.2020.1803916","article-title":"Multiple species of distinctiveness in memory? Comparing encoding versus statistical distinctiveness on recognition","volume":"28","author":"Gretz","year":"2020","journal-title":"Memory"},{"key":"10.7717\/peerj-cs.767\/ref-16","doi-asserted-by":"publisher","first-page":"1026","DOI":"10.1109\/ICCV.2015.123","article-title":"Delving deep into rectifiers: surpassing human-level performance on imagenet classification","author":"He","year":"2015"},{"key":"10.7717\/peerj-cs.767\/ref-17","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.sysarc.2019.02.008","article-title":"Filter-based deep-compression with global average pooling for convolutional networks","volume":"95","author":"Hsiao","year":"2019","journal-title":"Journal of Systems Architecture"},{"key":"10.7717\/peerj-cs.767\/ref-18","doi-asserted-by":"crossref","article-title":"Understanding the intrinsic memorability of images","author":"Isola","year":"2011","DOI":"10.21236\/ADA554133"},{"key":"10.7717\/peerj-cs.767\/ref-19","doi-asserted-by":"crossref","article-title":"What makes an image memorable?","author":"Isola","year":"2011","DOI":"10.1109\/CVPR.2011.5995721"},{"issue":"5","key":"10.7717\/peerj-cs.767\/ref-20","doi-asserted-by":"publisher","first-page":"1050","DOI":"10.1109\/TMM.2016.2644866","article-title":"Predicting image memorability through adaptive transfer learning from external sources","volume":"19","author":"Jing","year":"2016","journal-title":"IEEE Transactions on Multimedia"},{"key":"10.7717\/peerj-cs.767\/ref-21","doi-asserted-by":"publisher","article-title":"Understanding and predicting image memorability at a large scale","author":"Khosla","year":"2015","DOI":"10.1109\/ICCV.2015.275"},{"key":"10.7717\/peerj-cs.767\/ref-22","first-page":"296","article-title":"Memorability of image regions","volume":"25","author":"Khosla","year":"2012","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10.7717\/peerj-cs.767\/ref-23","doi-asserted-by":"publisher","article-title":"Photo aesthetics ranking network with attributes and content adaptation","author":"Kong","year":"2016","DOI":"10.1007\/978-3-319-46448-0_40"},{"key":"10.7717\/peerj-cs.767\/ref-24","doi-asserted-by":"publisher","first-page":"2488","DOI":"10.1109\/ICIP42928.2021.9506411","article-title":"Video memorability prediction via late fusion of deep multi-modal features","author":"Leyva","year":"2021"},{"key":"10.7717\/peerj-cs.767\/ref-25","doi-asserted-by":"publisher","first-page":"4927","DOI":"10.1109\/TIP.2020.2975957","article-title":"Understanding and predicting the memorability of outdoor natural scenes","volume":"29","author":"Lu","year":"2020","journal-title":"IEEE Transactions on Image Processing"},{"key":"10.7717\/peerj-cs.767\/ref-26","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1109\/ICIP.2013.6738041","article-title":"Memorability of natural scenes: the role of attention","author":"Mancas","year":"2013"},{"key":"10.7717\/peerj-cs.767\/ref-27","doi-asserted-by":"publisher","article-title":"Predicting image memorability by multi-view adaptive regression","author":"Peng","year":"2015","DOI":"10.1145\/2733373.2806303"},{"key":"10.7717\/peerj-cs.767\/ref-28","doi-asserted-by":"crossref","article-title":"Is image memorability prediction solved?","author":"Perera","year":"2019","DOI":"10.1109\/CVPRW.2019.00108"},{"key":"10.7717\/peerj-cs.767\/ref-29","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1016\/j.tics.2020.04.001","article-title":"Understanding image memorability","volume":"24","author":"Rust","year":"2020","journal-title":"Trends in Cognitive Sciences"},{"key":"10.7717\/peerj-cs.767\/ref-30","author":"Rusu","year":"2016","journal-title":"Progressive neural networks"},{"key":"10.7717\/peerj-cs.767\/ref-31","doi-asserted-by":"publisher","first-page":"1035","DOI":"10.3758\/s13414-018-1522-y","article-title":"Visual memory, the long and the short of it: a review of visual working memory and long-term memory","volume":"80","author":"Schurgin","year":"2018","journal-title":"Attention, Perception, & Psychophysics"},{"key":"10.7717\/peerj-cs.767\/ref-32","doi-asserted-by":"crossref","article-title":"Grad-cam: visual explanations from deep networks via gradient-based localization","author":"Selvaraju","year":"2017","DOI":"10.1109\/ICCV.2017.74"},{"key":"10.7717\/peerj-cs.767\/ref-33","author":"Smith","year":"2018","journal-title":"A disciplined approach to neural network hyper-parameters: part 1\u2013learning rate, batch size, momentum, and weight decay"},{"key":"10.7717\/peerj-cs.767\/ref-34","doi-asserted-by":"publisher","first-page":"100926","DOI":"10.1016\/j.dcn.2021.100926","article-title":"Memory specificity is linked to repetition effects in event-related potentials across the lifespan","volume":"48","author":"Sommer","year":"2021","journal-title":"Developmental Cognitive Neuroscience"},{"key":"10.7717\/peerj-cs.767\/ref-35","doi-asserted-by":"publisher","first-page":"2371","DOI":"10.1109\/ICASSP.2018.8462292","article-title":"Deep learning for predicting image memorability","author":"Squalli-Houssaini","year":"2018"},{"issue":"3","key":"10.7717\/peerj-cs.767\/ref-36","doi-asserted-by":"publisher","first-page":"1073","DOI":"10.3390\/app10031073","article-title":"An effective optimization method for machine learning based on ADAM","volume":"10","author":"Yi","year":"2020","journal-title":"Applied Sciences"},{"key":"10.7717\/peerj-cs.767\/ref-37","author":"Zhao","year":"2021","journal-title":"Multi -modal ensemble models for predicting video memorability"},{"key":"10.7717\/peerj-cs.767\/ref-38","doi-asserted-by":"publisher","first-page":"360","DOI":"10.1109\/MIPR49039.2020.00079","article-title":"Aesthetics-assisted multi-task learning with attention for image memorability prediction","author":"Zhu","year":"2020"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-767.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-767.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-767.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-767.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T10:26:19Z","timestamp":1636107979000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,5]]},"references-count":38,"alternative-id":["10.7717\/peerj-cs.767"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.767","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{"has-review":[{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.3\/reviews\/5","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.3\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.1\/reviews\/5","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.1\/reviews\/4","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.1\/reviews\/3","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.1\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.1\/reviews\/1","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.2\/reviews\/5","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.2\/reviews\/4","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.2\/reviews\/3","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.2\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.767v0.2\/reviews\/1","asserted-by":"object"}]},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,5]]},"article-number":"e767"}}
