{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,2]],"date-time":"2024-09-02T17:52:05Z","timestamp":1725299525835},"reference-count":48,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,11,6]],"date-time":"2022-11-06T00:00:00Z","timestamp":1667692800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Strategic Academic Leadership Program of the Southern Federal University","award":["Priority 2030"]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"In the field of intelligent surface inspection systems, particular attention is paid to decision making problems, based on data from different sensors. The combination of such data helps to make an intelligent decision. In this research, an approach to intelligent decision making based on a data integration strategy to raise awareness of a controlled object is used. In the following article, this approach is considered in the context of reasonable decisions when detecting defects on the surface of welds that arise after the metal pipe welding processes. The main data types were RGB, RGB-D images, and acoustic emission signals. The fusion of such multimodality data, which mimics the eyes and ears of an experienced person through computer vision and digital signal processing, provides more concrete and meaningful information for intelligent decision making. The main results of this study include an overview of the architecture of the system with a detailed description of its parts, methods for acquiring data from various sensors, pseudocodes for data processing algorithms, and an approach to data fusion meant to improve the efficiency of decision making in detecting defects on the surface of various materials.<\/jats:p>","DOI":"10.3390\/s22218554","type":"journal-article","created":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T08:02:22Z","timestamp":1667808142000},"page":"8554","source":"Crossref","is-referenced-by-count":3,"title":["Integrated Video and Acoustic Emission Data Fusion for Intelligent Decision Making in Material Surface Inspection System"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-2922-330X","authenticated-orcid":false,"given":"Andrey V.","family":"Chernov","sequence":"first","affiliation":[{"name":"The Smart Materials Research Institute, Southern Federal University, 178\/24 Sladkova, 344090 Rostov-on-Don, Russia"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-2060-0911","authenticated-orcid":false,"given":"Ilias K.","family":"Savvas","sequence":"additional","affiliation":[{"name":"School of Technology, University of Thessaly, Larissa-Trikala Ring-Road, 415000 Larissa, Greece"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-7918-453X","authenticated-orcid":false,"given":"Alexander A.","family":"Alexandrov","sequence":"additional","affiliation":[{"name":"The Smart Materials Research Institute, Southern Federal University, 178\/24 Sladkova, 344090 Rostov-on-Don, Russia"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-1260-8676","authenticated-orcid":false,"given":"Oleg O.","family":"Kartashov","sequence":"additional","affiliation":[{"name":"The Smart Materials Research Institute, Southern Federal University, 178\/24 Sladkova, 344090 Rostov-on-Don, Russia"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-7925-6302","authenticated-orcid":false,"given":"Dmitry S.","family":"Polyanichenko","sequence":"additional","affiliation":[{"name":"The Smart Materials Research Institute, Southern Federal University, 178\/24 Sladkova, 344090 Rostov-on-Don, Russia"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-6167-7942","authenticated-orcid":false,"given":"Maria A.","family":"Butakova","sequence":"additional","affiliation":[{"name":"The Smart Materials Research Institute, Southern Federal University, 178\/24 Sladkova, 344090 Rostov-on-Don, Russia"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-8411-0546","authenticated-orcid":false,"given":"Alexander V.","family":"Soldatov","sequence":"additional","affiliation":[{"name":"The Smart Materials Research Institute, Southern Federal University, 178\/24 Sladkova, 344090 Rostov-on-Don, Russia"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"H\u00e4rtel, S., Adams, T.-E., Hoefer, K., Awiszus, B., and Mayr, P. (2020). A Novel Method for Improving Weld Seam Properties through Inline Coupling of Welding and Forming. Materials, 13.","DOI":"10.3390\/ma13020271"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1007\/s40194-021-01229-6","article-title":"In Situ Detection of Welding Defects: A Review","volume":"66","author":"Madhvacharyula","year":"2022","journal-title":"Weld World"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"62","DOI":"10.3934\/matersci.2021005","article-title":"A Systematic Review on Recent Progress in Advanced Joining Techniques of the Lightweight Materials","volume":"8","author":"Bahl","year":"2021","journal-title":"AIMS Mater. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Shaloo, M., Schnall, M., Klein, T., Huber, N., and Reitinger, B. (2022). A Review of Non-Destructive Testing (NDT) Techniques for Defect Detection: Application to Fusion Welding and Future Wire Arc Additive Manufacturing Processes. Materials, 15.","DOI":"10.3390\/ma15103697"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3732","DOI":"10.1016\/j.matpr.2020.11.578","article-title":"Non-Destructive Testing (NDT) Techniques for Low Carbon Steel Welded Joints: A Review and Experimental Study","volume":"44","author":"Deepak","year":"2021","journal-title":"Mater. Today Proc."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"123","DOI":"10.3103\/S1068798X21020039","article-title":"Ultrasonic Testing of Welded Joints in Polyethylene Pipe","volume":"41","author":"Aleshin","year":"2021","journal-title":"Russ. Engin. Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Stephanou, P., and Xu, D. (2020, January 25\u201328). Ultrasound-Based Sensor for Non-Invasively Detecting Obstructions Within Natural Gas Pipeline. Proceedings of the 2020 IEEE SENSORS, Rotterdam, The Netherlands.","DOI":"10.1109\/SENSORS47125.2020.9278919"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wu, H.F., Gyekenyesi, A.L., Shull, P.J., and Yu, T.-Y. (2017). Ultrasonic Velocity Testing of Steel Pipeline Welded Joints. Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, and Civil Infrastructure 2017, SPIE.","DOI":"10.1117\/12.2249684"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Lim, S.J., Kim, Y.L., Cho, S., and Park, I.K. (2022). Ultrasonic Inspection for Welds with Irregular Curvature Geometry Using Flexible Phased Array Probes and Semi-Auto Scanners: A Feasibility Study. Appl. Sci., 12.","DOI":"10.3390\/app12020748"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4736","DOI":"10.1016\/j.jmrt.2019.08.020","article-title":"Phased Array Inspection of Glass Fiber Reinforced Polymers Pipeline Joints","volume":"8","author":"Pereira","year":"2019","journal-title":"J. Mater. Res. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hou, W., Zhang, D., Wei, Y., Guo, J., and Zhang, X. (2020). Review on Computer Aided Weld Defect Detection from Radiography Images. Appl. Sci., 10.","DOI":"10.3390\/app10051878"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1909","DOI":"10.1016\/j.matpr.2022.01.160","article-title":"Examination of Various Weld Process Parameters in MIG Welding of Carbon Steel on Weld Quality Using Radiography & Magnetic Particle Testing","volume":"62","author":"Pydi","year":"2022","journal-title":"Mater. Today Proc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1784\/insi.2021.63.2.75","article-title":"Development and Optimisation of Low-Power Magnetic Flux Leakage Inspection Parameters for Mild Steel Welds","volume":"63","author":"Watson","year":"2021","journal-title":"Insight"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1007\/s10556-020-00809-5","article-title":"Assessment of Conditions for Detecting Corrosion Cracking of Welded Joints by Acoustic Emission Method","volume":"56","author":"Rastegaev","year":"2020","journal-title":"Chem Pet. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"710","DOI":"10.1177\/14759217211008969","article-title":"Structural Damage Monitoring for Metallic Panels Based on Acoustic Emission and Adaptive Improvement Variational Mode Decomposition\u2013Wavelet Packet Transform","volume":"21","author":"Li","year":"2022","journal-title":"Struct. Health Monit."},{"key":"ref_16","first-page":"817","article-title":"Use of Acoustic Emission in the Evaluation of Corrosion Resistance of CMT Welds","volume":"20","author":"Rozlivka","year":"2020","journal-title":"Manuf. Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"112069","DOI":"10.1016\/j.matchar.2022.112069","article-title":"Analyzing Defects and Their Effects on the Strength of a Three-Layer FSW Joint by Using X-Ray Microtomography, Localized Spectrum Analysis, and Acoustic Emission","volume":"190","author":"Nitu","year":"2022","journal-title":"Mater. Charact."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kartashov, O.O., Chernov, A.V., Alexandrov, A.A., Polyanichenko, D.S., Ierusalimov, V.S., Petrov, S.A., and Butakova, M.A. (2022). Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection. Sensors, 22.","DOI":"10.3390\/s22166201"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1007\/s13198-021-01282-w","article-title":"The State-of-the-Art Methodologies for Quality Analysis of Arc Welding Process Using Weld Data Acquisition and Analysis Techniques","volume":"13","author":"Kumar","year":"2022","journal-title":"Int. J. Syst. Assur. Eng. Manag."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"101687","DOI":"10.1016\/j.aei.2022.101687","article-title":"Review on Automated Condition Assessment of Pipelines with Machine Learning","volume":"53","author":"Liu","year":"2022","journal-title":"Adv. Eng. Inform."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"5018911","DOI":"10.1109\/TIM.2021.3124053","article-title":"A Deep Learning-Based Weld Defect Classification Method Using Radiographic Images With a Cylindrical Projection","volume":"70","author":"Chang","year":"2021","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Naddaf-Sh, M.-M., Naddaf-Sh, S., Zargarzadeh, H., Zahiri, S.M., Dalton, M., Elpers, G., and Kashani, A.R. (2021). Defect Detection and Classification in Welding Using Deep Learning and Digital Radiography. Fault Diagnosis and Prognosis Techniques for Complex Engineering Systems, Elsevier.","DOI":"10.1016\/B978-0-12-822473-1.00007-0"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1007\/s10921-015-0315-7","article-title":"GDXray: The Database of X-Ray Images for Nondestructive Testing","volume":"34","author":"Mery","year":"2015","journal-title":"J. Nondestruct. Eval."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Guo, W., Qu, H., and Liang, L. (2018, January 28\u201330). WDXI: The Dataset of X-Ray Image for Weld Defects. Proceedings of the 2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Huangshan, China.","DOI":"10.1109\/FSKD.2018.8686975"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"164952","DOI":"10.1109\/ACCESS.2019.2953313","article-title":"An Automatic Detection and Identification Method of Welded Joints Based on Deep Neural Network","volume":"7","author":"Yang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1007\/s10845-020-01581-2","article-title":"Weld Defect Classification in Radiographic Images Using Unified Deep Neural Network with Multi-Level Features","volume":"32","author":"Yang","year":"2021","journal-title":"J. Intell. Manuf."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhu, H., Ge, W., and Liu, Z. (2019). Deep Learning-Based Classification of Weld Surface Defects. Appl. Sci., 9.","DOI":"10.3390\/app9163312"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1574350","DOI":"10.1155\/2020\/1574350","article-title":"Deep Learning Technology for Weld Defects Classification Based on Transfer Learning and Activation Features","volume":"2020","author":"Ajmi","year":"2020","journal-title":"Adv. Mater. Sci. Eng."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Apasov, A.M., and Apasov, A.A. (2012, January 18\u201321). Acoustic Emission Diagnostics of Faulty Fusion in Welding. Proceedings of the 2012 7th International Forum on Strategic Technology (IFOST), Tomsk, Russia.","DOI":"10.1109\/IFOST.2012.6357718"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"7","DOI":"10.4028\/www.scientific.net\/AMR.1025-1026.7","article-title":"Screening for Welding Defects Using Acoustic Emission Technique","volume":"1025\u20131026","author":"Aboali","year":"2014","journal-title":"AMR"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Barat, V., Marchenkov, A., Bardakov, V., Zhgut, D., Karpova, M., Balandin, T., and Elizarov, S. (2022). Assessment of the Structural State of Dissimilar Welded Joints by the Acoustic Emission Method. Appl. Sci., 12.","DOI":"10.3390\/app12147213"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.jcsr.2017.03.012","article-title":"Acoustic Emission Method for Defect Detection and Identification in Carbon Steel Welded Joints","volume":"134","author":"Droubi","year":"2017","journal-title":"J. Constr. Steel Res."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1016\/j.procs.2015.04.042","article-title":"Use of Machine Learning Algorithms for Weld Quality Monitoring Using Acoustic Signature","volume":"50","author":"Sumesh","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"109450","DOI":"10.1016\/j.compositesb.2021.109450","article-title":"Acoustic Emission Data Based Deep Learning Approach for Classification and Detection of Damage-Sources in a Composite Panel","volume":"228","author":"Sikdar","year":"2022","journal-title":"Compos. Part. B Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/5.554205","article-title":"An Introduction to Multisensor Data Fusion","volume":"85","author":"Hall","year":"1997","journal-title":"Proc. IEEE"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.inffus.2016.09.005","article-title":"Multi-Sensor Fusion in Body Sensor Networks: State-of-the-Art and Research Challenges","volume":"35","author":"Gravina","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1456650.1456651","article-title":"Data Fusion","volume":"41","author":"Bleiholder","year":"2009","journal-title":"ACM Comput. Surv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"7771","DOI":"10.3390\/s91007771","article-title":"Advances in Multi-Sensor Data Fusion: Algorithms and Applications","volume":"9","author":"Dong","year":"2009","journal-title":"Sensors"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Becerra, M.A., Tob\u00f3n, C., Castro-Ospina, A.E., and Peluffo-Ord\u00f3\u00f1ez, D.H. (2021). Information Quality Assessment for Data Fusion Systems. Data, 6.","DOI":"10.3390\/data6060060"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"100366","DOI":"10.1016\/j.cosrev.2021.100366","article-title":"A Survey of Data Center Consolidation in Cloud Computing Systems","volume":"39","author":"Helali","year":"2021","journal-title":"Comput. Sci. Rev."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Schreiber-Ehle, S., and Koch, W. (2012, January 4\u20136). The JDL Model of Data Fusion Applied to Cyber-Defence\u2014A Review Paper. Proceedings of the 2012 Workshop on Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany.","DOI":"10.1109\/SDF.2012.6327919"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lambert, D.A. (2007, January 9\u201312). STDF Model Based Maritime Situation Assessments. Proceedings of the 2007 10th International Conference on Information Fusion, Quebec City, QC, Canada.","DOI":"10.1109\/ICIF.2007.4408055"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s00138-021-01249-8","article-title":"Early, Intermediate and Late Fusion Strategies for Robust Deep Learning-Based Multimodal Action Recognition","volume":"32","author":"Boulahia","year":"2021","journal-title":"Mach. Vis. Appl."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"117407","DOI":"10.1016\/j.eswa.2022.117407","article-title":"Automated Accurate Fire Detection System Using Ensemble Pretrained Residual Network","volume":"203","author":"Dogan","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Nicholas, P.J., To, A., Tanglay, O., Young, I.M., Sughrue, M.E., and Doyen, S. (2022). Using a ResNet-18 Network to Detect Features of Alzheimer\u2019s Disease on Functional Magnetic Resonance Imaging: A Failed Replication. Comment on Odusami et al. Analysis of Features of Alzheimer\u2019s Disease: Detection of Early Stage from Functional Brain Changes in Magnetic Resonance Images Using a Finetuned ResNet18 Network. Diagnostics 2021, 11, 1071. Diagnostics, 12.","DOI":"10.3390\/diagnostics12051094"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.neunet.2017.12.012","article-title":"Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning","volume":"107","author":"Elfwing","year":"2018","journal-title":"Neural Netw."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"14809","DOI":"10.1007\/s00521-021-06120-5","article-title":"Multi-Stack Hybrid CNN with Non-Monotonic Activation Functions for Hyperspectral Satellite Image Classification","volume":"33","author":"Bandyopadhyay","year":"2021","journal-title":"Neural Comput. Applic"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"168781402199691","DOI":"10.1177\/1687814021996915","article-title":"Sounds and Acoustic Emission-Based Early Fault Diagnosis of Induction Motor: A Review Study","volume":"13","author":"AlShorman","year":"2021","journal-title":"Adv. Mech. Eng."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8554\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T03:05:24Z","timestamp":1723172724000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8554"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,6]]},"references-count":48,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22218554"],"URL":"http:\/\/dx.doi.org\/10.3390\/s22218554","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,11,6]]}}}
