Abstract
Mesh-based image warping techniques typically represent image deformation using linear functions on triangular meshes or bilinear functions on rectangular meshes. This enables simple and efficient implementation, but in turn, restricts the representation capability of the deformation, often leading to unsatisfactory warping results. We present a novel, flexible polygonal finite element (poly-FEM) method for content-aware image warping. Image deformation is represented by high-order poly-FEMs on a content-aware polygonal mesh with a cell distribution adapted to saliency information in the source image. This allows highly adaptive meshes and smoother warping with fewer degrees of freedom, thus significantly extending the flexibility and capability of the warping representation. Benefiting from the continuous formulation of image deformation, our poly-FEM warping method is able to compute the optimal image deformation by minimizing existing or even newly designed warping energies consisting of penalty terms for specific transformations. We demonstrate the versatility of the proposed poly-FEM warping method in representing different deformations and its superiority by comparing it to other existing state-of-the-art methods.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Vaquero, D.; Turk, M.; Pulli, K.; Tico, M.; Gelfand, N. A survey of image retargeting techniques. In: Proceedings of the SPIE 7798, Applications of Digital Image Processing XXXIII, 779814, 2010.
Kiess, J.; Kopf, S.; Guthier, B.; Effelsberg, W. A survey on content-aware image and video retargeting. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 14, No. 3, Article No. 76, 2018.
Suh, B.; Ling, H.; Bederson, B. B.; Jacobs. D. W. Automatic thumbnail cropping and its effectiveness. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, 95–104, 2003.
Nishiyama, M.; Okabe, T.; Sato, Y.; Sato, I. Sensation-based photo cropping. In: Proceedings of the 17th ACM International Conference on Multimedia, 669–672, 2009.
Setlur, V.; Takagi, S.; Raskar, R.; Gleicher, M.; Gooch, B. Automatic image retargeting. In: Proceedings of the 4th International Conference on Mobile and Ubiquitous Multimedia, 59–68, 2005.
Avidan, S.; Shamir, A. Seam carving for content-aware image resizing. ACM Transactions on Graphics Vol. 26, No. 3, 10–es, 2007.
Rubinstein, M.; Shamir, A.; Avidan, S. Improved seam carving for video retargeting. ACM Transactions on Graphics Vol. 27, No. 3, 1–9, 2008.
Rubinstein, M.; Shamir, A.; Avidan, S. Multi-operator media retargeting. ACM Transactions on Graphics Vo. 28, No. 3 Article No. 23, 2009.
Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B. PatchMatch: A randomized correspondence algorithm for structural image editing. In: Proceedings of the ACM SIGGRAPH 2009 papers, Article No. 24, 2009.
Pritch, Y.; Kav-Venaki, E.; Peleg, S. Shift-map image editing. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 151–158, 2009.
Guo, Y. W.; Liu, F.; Shi, J.; Zhou, Z. H.; Gleicher, M. Image retargeting using mesh parametrization. IEEE Transactions on Multimedia Vol. 11, No. 5, 856–867, 2009.
Lau, C. P.; Yung, C. P.; Lui, L. M. Image retargeting via Beltrami representation. IEEE Transactions on Image Processing Vol. 27, No. 12, 5787–5801, 2018.
Wolf, L.; Guttmann, M.; Cohen-Or, D. Non-homogeneous content-driven video-retargeting. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–6, 2007.
Wang, Y.-S.; Tai, C.-L.; Sorkine, O.; Lee, T.-Y. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics Vol. 27, No. 5, Article No. 118, 2008.
Karni, Z.; Freedman, D.; Gotsman, C. Energy-based image deformation. Computer Graphics Forum Vol. 28, 1257–1268, 2009.
Krähenbühl, P.; Lang, M.; Hornung, A.; Gross, M. A system for retargeting of streaming video. ACM Transactions on Graphics Vol. 28, No. 5, 1–10, 2009.
Shi, M.; Yang, L.; Peng, G.; Xu, D. A content-aware image resizing method with prominent object size adjusted. In: Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology, 175–176, 2010.
Chen, R. J.; Freedman, D.; Karni, Z.; Gotsman, C.; Liu, L. G. Content-aware image resizing by quadratic programming. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition — Workshops, 1–8, 2010.
Cho, D.; Park, J.; Oh, T. H.; Tai, Y. W.; Kweon, I. S. Weakly- and self-supervised learning for content-aware deep image retargeting. In: Proceedings of the IEEE International Conference on Computer Vision, 4568–4577, 2017.
Tan, W. M.; Yan, B.; Lin, C. M.; Niu, X. J. Cycle-IR: Deep cyclic image retargeting. IEEE Transactions on Multimedia Vol. 22, No. 7, 1730–1743, 2020.
Zhou, Y.; Chen, Z. B.; Li, W. P. Weakly supervised reinforced multi-operator image retargeting. IEEE Transactions on Circuits and Systems for Video Technology Vol. 31, No. 1, 126–139, 2021.
Danon, D.; Arar, M.; Cohen-Or, D.; Shamir, A. Image resizing by reconstruction from deep features. Computational Visual Media Vol. 7, No. 4, 453–466, 2021.
Hu, S. M.; Li, C. F.; Zhang, H. Actual morphing: A physics-based approach to blending. In: Proceedings of the 9th ACM Symposium on Solid Modeling and Applications, 309–314, 2004.
Yan, H.-B.; Hu, S.-M.; Martin, R. Morphing based on strain field interpolation. Computer Animation and Virtual Worlds Vol. 15, Nos. 3–4, 443–452, 2004.
Bajaj, C.; Zhang, Y.; Xu, G. Physically-based surface texture synthesis using a coupled finite element system. In: Advances in Geometric Modeling and Processing. Lecture Notes in Computer Science, Vol. 4975. Chen, F.; Jüttler, B. Eds. Springer Berlin Heidelberg, 344–357, 2008.
Jacobson, A.; Tosun, E.; Sorkine, O.; Zorin, D. Mixed finite elements for variational surface modeling. Computer Graphics Forum Vol. 29, No. 5, 1565–1574, 2010.
Zhang, Y. J. Geometric Modeling and Mesh Generation from Scanned Images. New York: Chapman and Hall/CRC, 2016.
Gee, J. C.; Haynor, D. R.; Martin, R. M. D.; Bajcsy, R. K. Finite element approach to warping of brain images. In: Proceedings of the SPIE 2167, Medical Imaging 1994: Image Processing, 327–337, 1994.
Kaufmann, P.; Wang, O.; Sorkine-Hornung, A.; Sorkine-Hornung, O.; Smolic, A.; Gross, M. Finite element image warping. Computer Graphics Forum Vol. 32, No. 2pt1, 31–39, 2013.
Wachspress, E. L. A Rational Finite Element Basis. Elsevier, 1975.
Meyer, M.; Barr, A.; Lee, H.; Desbrun, M. Generalized barycentric coordinates on irregular polygons. Journal of Graphics Tools Vol. 7, No. 1, 13–22, 2002.
Hormann, K.; Floater, M. S. Mean value coordinates for arbitrary planar polygons. ACM Transactions on Graphics Vol. 25, No. 4, 1424–1441, 2006.
Rand, A.; Gillette, A.; Bajaj, C. Quadratic serendipity finite elements on polygons using generalized barycentric coordinates. Mathematics of Computation Vol. 83, 2691–2716, 2014.
Sukumar, N. Quadratic maximum-entropy serendipity shape functions for arbitrary planar polygons. Computer Methods in Applied Mechanics and Engineering Vol. 263, 27–41, 2013.
Floater, M. S.; Lai, M. J. Polygonal spline spaces and the numerical solution of the Poisson equation. SIAM Journal on Numerical Analysis Vol. 54, No. 2, 797–824, 2016.
Cao, J.; Xiao, Y. Y.; Chen, Z. G.; Wang, W. P.; Bajaj, C. Functional data approximation on bounded domains using polygonal finite elements. Computer Aided Geometric Design Vol. 63, 149–163, 2018.
Engelke, U.; Wang, J. L.; Marendy, P. Perceptual relevance based image retargeting. IEEE Signal Processing Letters Vol. 22, No. 6, 705–708, 2015.
Wang, J.; Jiang, H.; Yuan, Z.; Cheng, M.-M.; Hu, X.; Zheng, N. Salient object detection: A discriminative regional feature integration approach. International Journal of Computer Vision Vol. 123, No. 2, 251–268, 2017.
Schneider, T. Theory and applications of bijective barycentric mappings. Ph.D. Thesis. Universita della Svizzera italiana, 2017.
Sieger, D.; Alliez, P.; Botsch, M. Optimizing Voronoi Diagrams for polygonal finite element computations. In: Proceedings of the 19th International Meshing Roundtable, 335–350, 2010.
Du, Q.; Faber, V.; Gunzburger, M. Centroidal voronoi tessellations: Applications and algorithms. SIAM Review Vol. 41, No. 4, 637–676, 1999.
Gupta, S.; Mazumdar, S. G. Sobel edge detection algorithm. International Journal of Computer Science and Management Research Vol. 2, No. 2, 1578–1583, 2013.
Laffont, P.-Y.; Jun, J. Y.; Wolf, C.; Tai, Y.-W.; Idrissi, K.; Drettakis, G.; Yoon, S.-e. Interactive content-aware zooming. In: Proceedings of the Graphics Interface, 79–87, 2010.
Zhang, G.-X.; Cheng, M.-M.; Hu, S.-M.; Martin, R. R. A shape-preserving approach to image resizing. Computer Graphics Forum Vol. 28, No. 7, 1897–1906, 2009.
Tang, F.; Dong, W.; Meng, Y.; Ma, C.; Wu, F.; Li, X.; Lee, T.-Y. Image retargetability. IEEE Transactions on Multimedia Vol. 22, No. 3, 641–654, 2020.
Rubinstein, M.; Gutierrez, D.; Sorkine, O.; Shamir, A. A comparative study of image retargeting. ACM Transactions on Graphics Vol. 29. No. 6, Article No. 160, 2010.
Liu, Y. J.; Han, Y. H.; Ye, Z. P.; Lai, Y. K. Ranking-preserving cross-source learning for image retargeting quality assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 42, No. 7, 1798–1805, 2020.
Li, X. Y.; Ju, T.; Hu, S. M. Cubic mean value coordinates. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 126, 2013.
Acknowledgements
We would like to thank the reviewers for their valuable comments. The research of Juan Cao was supported by the National Natural Science Foundation of China (Nos. 61872308, 61972327, and 62272402), and the Xiamen Youth Innovation Funds (No. 3502Z20206029). Yongjie Jessica Zhang was supported in part by NSF CMMI-1953323 and a Honda grant.
Author information
Authors and Affiliations
Corresponding author
Additional information
Declaration of competing interest
The authors have no competing interests to declare that are relevant to the content of this article.
Juan Cao is an associate professor in the School of Mathematical Sciences, Xiamen University, China. She received her Ph.D. degree in applied mathematics from Zhejiang University, China, in 2009. Her research interests include computer aided geometric design, and computer graphics.
Xiaoyi Zhang received her B.S. degree from Soochow University in 2018, and M.S. degree from Xiamen University in 2021. She is currently an engineer in Vivo. Her research interests include computer graphics and image processing.
Jiannan Huang received his B.S. degree from Ningxia University in 2015, and M.S. degree from Xiamen University in 2018. He is currently an engineer in Xiamen Road and Bridge Information Engineering Co. Ltd. His research interests include computer aided geometric design and computer graphics.
Yongjie Jessica Zhang is the George Tallman Ladd and Florence Barrett Ladd Professor of Mechanical Engineering at Carnegie Mellon University with a courtesy appointment in the Department of Biomedical Engineering. She received her B.Eng. degree in automotive engineering and M.Eng. degree in engineering mechanics from Tsinghua University, China, and M.Eng. degree in aerospace engineering and engineering mechanics and Ph.D. degree in computational engineering and sciences from the Oden Institute, the University of Texas at Austin. Her research interests include image processing, computational geometry, mesh generation, finite element methods, isogeometric analysis, data-driven simulation, and their applications in computational biomedicine, materials science, and engineering.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Cao, J., Zhang, X., Huang, J. et al. Polygonal finite element-based content-aware image warping. Comp. Visual Media 9, 367–383 (2023). https://doi.org/10.1007/s41095-022-0283-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-022-0283-7