Abstract
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location – the inpainting mask – is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in terms of a learned mask generation model. By emulating the complete inpainting pipeline with two networks for mask generation and neural surrogate inpainting, we obtain a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude. Our findings serve as a basis for making diffusion-based inpainting more attractive for applications such as image compression, where fast encoding is highly desirable.
This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 741215, ERC Advanced Grant INCOVID).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adam, R.D., Peter, P., Weickert, J.: Denoising by inpainting. In: Lauze, F., Dong, Y., Dahl, A.B. (eds.) Scale Space and Variational Methods in Computer Vision. LNCS, vol. 10302, pp. 121–132. Springer, Cham (2017)
Alt, T., Schrader, K., Augustin, M., Peter, P., Weickert, J.: Connections between numerical algorithms for PDEs and neural networks. arXiv:2107.14742v1 [math.NA], July 2021
Andris, S., Peter, P., Mohideen Kaja Mohideen, R., Weickert, J., Hoffmann, S.: Inpainting-based video compression in FullHD. In: Elmoataz, A., Fadili, J., Quéau, Y., Rabin, J., Simon, L. (eds.) SSVM 2021. LNCS, vol. 12679, pp. 425–436. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-75549-2_34
Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)
Belhachmi, Z., Bucur, D., Burgeth, B., Weickert, J.: How to choose interpolation data in images. SIAM J. Appl. Math. 70(1), 333–352 (2009)
Bonettini, S., Loris, I., Porta, F., Prato, M., Rebegoldi, S.: On the convergence of a linesearch based proximal-gradient method for nonconvex optimization. Inverse Probl. 33(5), 055005 (2017)
Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression. In: Kúkelová, Z., Heller, J. (eds.) Proceedings 19th Computer Vision Winter Workshop. Křtiny, Czech Republic, Feburary 2014
Chizhov, V., Weickert, J.: Efficient data optimisation for harmonic inpainting with finite elements. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds.) CAIP 2021. LNCS, vol. 13053, pp. 432–441. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89131-2_40
Dai, Q., Chopp, H., Pouyet, E., Cossairt, O., Walton, M., Katsaggelos, A.K.: Adaptive image sampling using deep learning and its application on X-ray fluorescence image reconstruction. IEEE Trans. Multimedia 22(10), 2564–2578 (2019)
Daropoulos, V., Augustin, M., Weickert, J.: Sparse inpainting with smoothed particle hydrodynamics. SIAM J. Appl. Math. 14(4), 1669–1704 (2021)
Demaret, L., Dyn, N., Iske, A.: Image compression by linear splines over adaptive triangulations. Sign. Process. 86(7), 1604–1616 (2006)
Floyd, R.W., Steinberg, L.: An adaptive algorithm for spatial grey scale. Proc. Soc. Inf. Disp. 17, 75–77 (1976)
Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imaging Vis. 31(2–3), 255–269 (2008)
Golts, A., Freedman, D., Elad, M.: Deep energy: task driven training of deep neural networks. IEEE J. Select. Top. Sign. Process. 15(2), 324–338 (2021)
Hoeltgen, L., et al.: Optimising spatial and tonal data for PDE-based inpainting. In: Bergounioux, M., Peyré, G., Schnörr, C., Caillau, J.P., Haberkorn, T. (eds.) Variational Methods in Imaging and Geometric Control, Radon Series on Computational and Applied Mathematics, vol. 18, pp. 35–83. De Gruyter, Berlin (2017)
Hoeltgen, L., Weickert, J.: Why does non-binary mask optimisation work for diffusion-based image compression? In: Tai, X.-C., Bae, E., Chan, T.F., Lysaker, M. (eds.) EMMCVPR 2015. LNCS, vol. 8932, pp. 85–98. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14612-6_7
Iijima, T.: Basic theory on normalization of pattern (in case of typical one-dimensional pattern). Bull. Electrotech. Lab. 26, 368–388 (1962). in Japanese
Isogawa, K., Ida, T., Shiodera, T., Takeguchi, T.: Deep shrinkage convolutional neural network for adaptive noise reduction. IEEE Sign. Process. Lett. 25(2), 224–228 (2017)
Karos, L., Bheed, P., Peter, P., Weickert, J.: Optimising data for exemplar-based inpainting. In: Blanc-Talon, J., Helbert, D., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2018. LNCS, vol. 11182, pp. 547–558. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01449-0_46
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of 3rd International Conference on Learning Representations. San Diego, CA, May 2015
Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: Proceedings 2019 IEEE/CVF International Conference on Computer Vision, pp. 4170–4179. Seoul, Korea, October 2017
Mainberger, M., et al.: Optimising Spatial and Tonal Data for Homogeneous Diffusion Inpainting. In: Bruckstein, A.M., ter Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 26–37. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24785-9_3
Marwood, D., Massimino, P., Covell, M., Baluja, S.: Representing images in 200 bytes: Compression via triangulation. In: Proceedings of 2018 IEEE International Conference on Image Processing, pp. 405–409. Athens, Greece, October 2018
Mohideen, R.M.K., Peter, P., Weickert, J.: A systematic evaluation of coding strategies for sparse binary images. Sign. Process. Image Commun. 99, 116424, November 2021
Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Proceedings 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544. Las Vegas, NV, June 2016
Peter, P., Hoffmann, S., Nedwed, F., Hoeltgen, L., Weickert, J.: Evaluating the true potential of diffusion-based inpainting in a compression context. Sign. Process. Image Commun. 46, 40–53 (2016)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Schmaltz, C., Peter, P., Mainberger, M., Ebel, F., Weickert, J., Bruhn, A.: Understanding, optimising, and extending data compression with anisotropic diffusion. Int. J. Comput. Vis. 108(3), 222–240 (2014)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446–9454. Salt Lake City, UT, June 2018
Vašata, D., Halama, T., Friedjungová, M.: Image inpainting using Wasserstein generative adversarial imputation network. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds.) ICANN 2021. LNCS, vol. 12892, pp. 575–586. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86340-1_46
Weickert, J., Welk, M.: Tensor field interpolation with PDEs. In: Weickert, J., Hagen, H. (eds.) Visualization and Processing of Tensor Fields, pp. 315–325. Springer, Berlin (2006)
Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Bartlett, P.L., Pereira, F.C.N., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Proceedings of 26th International Conference on Neural Information Processing Systems. Advances in Neural Information Processing Systems, vol. 25, pp. 350–358. Lake Tahoe, NV, December 2012
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 6721–6729. Honolulu, HI, July 2017
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514. Salt Lake City, UT, June 2018
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Alt, T., Peter, P., Weickert, J. (2022). Learning Sparse Masks for Diffusion-Based Image Inpainting. In: Pinho, A.J., Georgieva, P., Teixeira, L.F., Sánchez, J.A. (eds) Pattern Recognition and Image Analysis. IbPRIA 2022. Lecture Notes in Computer Science, vol 13256. Springer, Cham. https://doi.org/10.1007/978-3-031-04881-4_42
Download citation
DOI: https://doi.org/10.1007/978-3-031-04881-4_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04880-7
Online ISBN: 978-3-031-04881-4
eBook Packages: Computer ScienceComputer Science (R0)
