iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://unpaywall.org/10.1007/978-3-642-01082-8_2
Automatic Approximation of Expensive Functions with Active Learning | SpringerLink
Skip to main content
Springer Nature Link
Log in

Automatic Approximation of Expensive Functions with Active Learning

  • Chapter
Foundations of Computational, Intelligence Volume 1

Part of the book series: Studies in Computational Intelligence ((SCI,volume 201))

Abstract

The use of computer simulations has become a viable alternative for real-life controlled experiments. Due to the computational cost of these high fidelity simulations, surrogate models are often employed as an approximation for the original simulator. Because of their compact formulation and negligible evaluation time, global surrogate models are very useful tools for exploring the design space, optimization, visualization and sensitivity analysis. Additionally, multiple surrogate models can be chained together to model large scale systems where the direct use of the original expensive simulators would be too cumbersome. Many surrogate model types, such as neural networks, support vector machines and rational models have been proposed, and many more techniques have been developed to minimize the number of expensive simulations required to train a sufficiently accurate surrogate model. In this chapter, we present a fully automated and integrated global surrogate modeling methodology for regression modeling and active learning that readily enables the adoption of advanced global surrogate modeling methods by application scientists. The work brings together insights from distributed systems, artificial intelligence, and modeling & simulation, and has applications in a very wide range of fields. The merits of this approach are illustrated with several examples, and several surrogate model types.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Wang, G.G., Shan, S.: Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design 129(4), 370–380 (2007)

    Article  MathSciNet  Google Scholar 

  2. Gu, L.: A comparison of polynomial based regression models in vehicle safety analysis. In: Diaz, A. (ed.) 2001 ASME Design Automation Conference, ASME, Pittsburgh, PA (2001)

    Google Scholar 

  3. Giunta, A., McFarland, J., Swiler, L., Eldred, M.: The promise and peril of uncertainty quantification using response surface approximations. Structure & Infrastructure Engineering 2, 175–189 (2006)

    Google Scholar 

  4. Gorissen, D., Crombecq, K., Hendrickx, W., Dhaene, T.: Grid enabled metamodeling. In: Daydé, M., Palma, J.M.L.M., Coutinho, Á.L.G.A., Pacitti, E., Lopes, J.C. (eds.) VECPAR 2006. LNCS, vol. 4395. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  5. Hendrickx, W., Dhaene, T.: Sequential design and rational metamodelling. In: Kuhl, M., Steiger, N.M., Armstrong, F.B., Joines, J.A. (eds.) Proceedings of the 2005 Winter Simulation Conference, pp. 290–298 (2005)

    Google Scholar 

  6. Deschrijver, D., Dhaene, T., Zutter, D.D.: Robust parametric macromodeling using multivariate orthonormal vector fitting. IEEE Transactions on Microwave Theory and Techniques 56(7), 1661–1667 (2008)

    Article  Google Scholar 

  7. Simpson, T.W., Poplinski, J.D., Koch, P.N., Allen, J.K.: Metamodels for computer-based engineering design: Survey and recommendations. Eng. Comput (Lond.) 17(2), 129–150 (2001)

    MATH  Google Scholar 

  8. Balewski, L., Mrozowski, M.: Creating neural models using an adaptive algorithm for optimal size of neural network and training set. In: 15th International Conference on Microwaves, Radar and Wireless Communications, MIKON 2004, Conference proceedings of MIKON 2004, vol. 2, pp. 543–546 (2004)

    Google Scholar 

  9. Booker, A.J., Dennis, J.E., Frank, P.D., Serafini, D.B., Torczon, V., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogate. Structural and Multidisciplinary Optimization 17(1), 1–13 (1999)

    Google Scholar 

  10. Eldred, M.S., Dunlavy, D.M.: Formulations for surrogate-based optimization wiht data fit, multifidelity, and reduced-order models. In: 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Protsmouth, Virginia (2006)

    Google Scholar 

  11. Anguita, D., Ridella, S., Rivieccio, F., Zunino, R.: Automatic hyperparameter tuning for support vector machines. In: Dorronsoro, J.R. (ed.) ICANN 2002. LNCS, vol. 2415, pp. 1345–1350. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  12. Sanchez, E., Pintos, S., Queipo, N.: Toward an optimal ensemble of kernel-based approximations with engineering applications. In: Proceedings of the International Joint Conference on Neural Networks, 2006. IJCNN 2006, pp. 2152–2158 (2006)

    Google Scholar 

  13. Turner, C.J., Crawford, R.H., Campbell, M.I.: Multidimensional sequential sampling for nurbs-based metamodel development. Eng. with Comput. 23(3), 155–174 (2007)

    Article  MathSciNet  Google Scholar 

  14. Sugiyama, M., Ogawa, H.: Release from active learning/model selection dilemma: optimizing sample points and models at the same time. In: Neural Networks, 2002. IJCNN 2002. Proceedings of the 2002 International Joint Conference on Neural Networks, 2002, vol. 3, pp. 2917–2922 (2002)

    Google Scholar 

  15. Lin, Y.: An Efficient Robust Concept Exploration Method and Sequential Exploratory Experimental Design. PhD thesis, Georgia Institute of Technology (2004)

    Google Scholar 

  16. Busby, D., Farmer, C.L., Iske, A.: Hierarchical nonlinear approximation for experimental design and statistical data fitting. SIAM Journal on Scientific Computing 29(1), 49–69 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  17. Gramacy, R.B., Lee, H.K.H., Macready, W.G.: Parameter space exploration with gaussian process trees. In: ICML 2004: Proceedings of the twenty-first international conference on Machine learning, p. 45. ACM Press, New York (2004)

    Chapter  Google Scholar 

  18. Farhang-Mehr, A., Azarm, S.: Bayesian meta-modelling of engineering design simulations: a sequential approach with adaptation to irregularities in the response behaviour. International Journal for Numerical Methods in Engineering 62(15), 2104–2126 (2005)

    Article  MATH  Google Scholar 

  19. Devabhaktuni, V., Chattaraj, B., Yagoub, M., Zhang, Q.J.: Advanced microwave modeling framework exploiting automatic model generation, knowledge neural networks, and space mapping. IEEE Tran. on Microwave Theory and Techniques 51(7), 1822–1833 (2003)

    Article  Google Scholar 

  20. Ganser, M., Grossenbacher, K., Schutz, M., Willmes, L., Back, T.: Simulation meta-models in the early phases of the product development process. In: Proceedings of Efficient Methods for Robust Design and Optimization (EUROMECH 2007) (2007)

    Google Scholar 

  21. Clarke, S.M., Griebsch, J.H., Simpson, T.W.: Analysis of support vector regression for approximation of complex engineering analyses. In: Proceedings of the 29th Design Automation Conference (ASME Design Engineering Technical Conferences) (DAC/DETC 2003) (2003)

    Google Scholar 

  22. Gorissen, D., Crombecq, K., Couckuyt, I., Dhaene, T.: Automatic approximation of expensive functions with active learning. Technical Report TR-10-08, University of Antwerp, Middelheimlaan 1, 2020 Antwerp, Belgium (2008)

    Google Scholar 

  23. Goel, T., Haftka, R., Shyy, W.: Comparing error estimation measures for polynomial and kriging approximation of noise-free functions. Journal of Structural and Multidisciplinary Optimization (published online) (2008)

    Google Scholar 

  24. Ong, Y.S., Nair, P.B., Keane, A.J., Wong, K.W.: Surrogate-Assisted Evolutionary Optimization Frameworks for High-Fidelity Engineering Design Problems, Knowledge Incorporation in Evolutionary Computation. Studies in Fuzziness and Soft Computing Series, pp. 307–331. Springer, Heidelberg (2004)

    Google Scholar 

  25. Giunta, A., Eldred, M.: Implementation of a trust region model management strategy in the DAKOTA optimization toolkit. In: Proceedings of the 8th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA (2000)

    Google Scholar 

  26. Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. of Global Optimization 13(4), 455–492 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  27. Sasena, M.J., Papalambros, P.Y., Goovaerts, P.: Metamodeling sampling criteria in a global optimization framework. In: 8th AIAA/ USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA, AIAA Paper, 2000–4921 (2000)

    Google Scholar 

  28. Parmee, I., Abraham, J., Shackelford, M., Rana, O.F., Shaikhali, A.: Towards autonomous evolutionary design systems via grid-based technologies. In: Proceedings of ASCE Computing in Civil Engineering, Cancun, Mexico (2005)

    Google Scholar 

  29. Eldred, M., Outka, D., Fulcher, C., Bohnhoff, W.: Optimization of complex mechanics simulations with object-oriented software design. In: Proceedings of the 36th IAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, New Orleans, LA, pp. 2406–2415 (1995)

    Google Scholar 

  30. Zhang, Q.J., Gupta, K.C.: Neural Networks for RF and Microwave Design (Book + Neuromodeler Disk). Artech House, Inc., Norwood (2000)

    Google Scholar 

  31. Gano, S., Kim, H., Brown, D.: Comparison of three surrogate modeling techniques: Datascape, kriging, and second order regression. In: Proceedings of the 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA-2006-7048, Portsmouth, Virginia (2006)

    Google Scholar 

  32. Crombecq, K.: A gradient based approach to adaptive metamodeling. Technical report, University of Antwerp (2007)

    Google Scholar 

  33. De Geest, J., Dhaene, T., Faché, N., De Zutter, D.: Adaptive CAD-model building algorithm for general planar microwave structures. IEEE Transactions on Microwave Theory and Techniques 47(9), 1801–1809 (1999)

    Article  Google Scholar 

  34. Barton, R.R.: Design of experiments for fitting subsystem metamodels. In: WSC 1997: Proceedings of the 29th conference on Winter simulation, pp. 303–310. ACM Press, New York (1997)

    Chapter  Google Scholar 

  35. Forrester, A.I.J., Bressloff, N.W., Keane, A.J.: Optimization using surrogate models and partially converged computational fluid dynamics simulations. Proceedings of the Royal Society 462, 2177–2204 (2006)

    Article  MATH  Google Scholar 

  36. Yesilyurt, S., Ghaddar, C.K., Cruz, M.E., Patera, A.T.: Bayesian-validated surrogates for noisy computer simulations; application to random media. SIAM Journal on Scientific Computing 17(4), 973–992 (1996)

    Article  MathSciNet  Google Scholar 

  37. Chaveesuk, R., Smith, A.: Economic valuation of capital projects using neural network metamodels. The Engineering Economist 48(1), 1–30 (2003)

    Article  Google Scholar 

  38. Knight, D., Kohn, J., Rasheed, K., Weber, N., Kholodovych, V., Welsh, W., Smith, J.: Using surrogate modeling in the prediction of fibrinogen adsorption onto polymer surfaces. Journal of chemical information and computer science 55, 1088–1097 (2004)

    Google Scholar 

  39. Hidovic, D., Rowe, J.: Validating a model of colon colouration using an evolution strategy with adaptive approximations. In: Deb, K., et al. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 1005–1016. Springer, Heidelberg (2004)

    Google Scholar 

  40. Brown, M., Adams, S., Dunlavy, B., Gay, D., Swiler, D., Giunta, L., Hart, A., Watson, W., Eddy, J.P., Griffin, J., Hough, J., Kolda, P., Martinez-Canales, T., Eldred, M., Williams, P.: Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: Version 4.1 users manual. Technical Report SAND2006-6337, Sandia Labs (2007)

    Google Scholar 

  41. Kleijnen, J.P., Sanchez, S.M., Lucas, T.W., Cioppa, T.M.: State-of-the-art review: A user’s guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing 17(3), 263–289 (2005)

    Article  Google Scholar 

  42. Ding, M., Vemur, R.: An active learning scheme using support vector machines for analog circuit feasibility classification. In: 18th International Conference on VLSI Design, pp. 528–534 (2005)

    Google Scholar 

  43. Devabhaktuni, V.K., Zhang, Q.J.: Neural network training-driven adaptive sampling algorithm. In: Proceedings of 30th European Microwave Conference, Paris, France, vol. 3, pp. 222–225 (2000)

    Google Scholar 

  44. Robertazzi, T.G., Schwartz, S.C.: An accelerated sequential algorithm for producing d-optimal designs. SIAM Journal on scientific Computing 10, 341–358 (1989)

    Article  MathSciNet  Google Scholar 

  45. Keys, A.C., Rees, L.P.: A sequential-design metamodeling strategy for simulation optimization. Comput. Oper. Res. 31(11), 1911–1932 (2004)

    Article  MATH  Google Scholar 

  46. Sasena, M.: Flexibility and Efficiency Enhancements For Constrainted Global Design Optimization with Kriging Approximations. PhD thesis, University of Michigan (2002)

    Google Scholar 

  47. Jin, R., Chen, W., Sudjianto, A.: On sequential sampling for global metamodeling in engineering design, detc-dac34092. In: ASME Design Automation Conference, Montreal, Canada (2002)

    Google Scholar 

  48. Lehmensiek, R., Meyer, P., Muller, M.: Adaptive sampling applied to multivariate, multiple output rational interpolation models with applications to microwave circuits. International Journal of RF and microwave computer aided engineering 12(4), 332–340 (2002)

    Article  Google Scholar 

  49. Deschrijver, D., Dhaene, T.: Rational modeling of spectral data using orthonormal vector fitting. In: Proceedings of 9th IEEE Workshop on Signal Propagation on Interconnects, 2005, pp. 111–114 (2005)

    Google Scholar 

  50. Kingston, G., Maier, H., Lambert, M.: Calibration and validation of neural networks to ensure physically plausible hydrological modeling. Journal of Hydrology 314, 158–176 (2005)

    Article  Google Scholar 

  51. Srinivasulu, S., Jain, A.: A comparative analysis of training methods for artificial neural network rainfall-runoff models. Appl. Soft Comput. 6(3), 295–306 (2006)

    Article  Google Scholar 

  52. Wolpert, D.: The supervised learning no-free-lunch theorems. In: Proceedings of the 6th Online World Conference on Soft Computing in Industrial Applications (2001)

    Google Scholar 

  53. Triverio, P., Grivet-Talocia, S., Nakhla, M., Canavero, F.G., Achar, R.: Stability, causality, and passivity in electrical interconnect models. IEEE Transactions on Advanced Packaging 30(4), 795–808 (2007)

    Article  Google Scholar 

  54. Ihme, M., Marsden, A.L., Pitsch, H.: Generation of optimal artificial neural networks using a pattern search algorithm: Application to approximation of chemical systems. Neural Computation 20, 573–601 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  55. Lessmann, S., Stahlbock, R., Crone, S.: Genetic algorithms for support vector machine model selection. In: International Joint Conference on Neural Networks, 2006. IJCNN 2006, pp. 3063–3069 (2006)

    Google Scholar 

  56. JiGuan, G.L.: Modeling test responses by multivariable polynomials of higher degrees. SIAM Journal on Scientific Computing 28(3), 832–867 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  57. Ye, K., Li, W., Sudjianto, A.: Algorithmic construction of optimal symmetric latin hypercube designs. Journal of Statistical Planning and Inference 90, 145–159 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  58. Gorissen, D., Dhaene, T., Demeester, P., Broeckhove, J.: Grid enabled surrogate modeling. In: The Encyclopedia of Grid Computing Technologies and Applications (in press) (2008)

    Google Scholar 

  59. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation 6(2), 182–197 (2002)

    Article  Google Scholar 

  60. Vrugt, J.A., Gupta, H.V., Bouten, W., Sorooshian, S.: A shuffled complex evolution metropolis algorithm for optimization and uncertainty assessment of hydrologic model parameters. Water Resources Research 39(8), 1214–1233 (2003)

    Article  Google Scholar 

  61. Gorissen, D.: Heterogeneous evolution of surrogate models. Master’s thesis, Master of AI, Katholieke Universiteit Leuven, KUL (2007)

    Google Scholar 

  62. Gorissen, D., Tommasi, L.D., Croon, J., Dhaene, T.: Automatic model type selection with heterogeneous evolution: An application to rf circuit block modeling. In: Proceedings of the IEEE Congress on Evolutionary Computation, WCCI 2008, Hong Kong (2008)

    Google Scholar 

  63. Lim, D., Ong, Y.S., Jin, Y., Sendhoff, B.: A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. In: GECCO 2007: Proceedings of the 9th annual conference on Genetic and evolutionary computation, pp. 1288–1295. ACM, New York (2007)

    Chapter  Google Scholar 

  64. Goel, T., Haftka, R., Shyy, W., Queipo, N.: Ensemble of surrogates. Structural and Multidisciplinary Optimization 33, 199–216 (2007)

    Article  Google Scholar 

  65. Pozar, D.M.: Microwave Engineering, 2nd edn. John Wiley and Sons, Chichester (1998)

    Google Scholar 

  66. Foresee, F., Hagan, M.: Gauss-newton approximation to bayesian regularization. In: Proceedings of the 1997 International Joint Conference on Neural Networks, pp. 1930–1935 (1997)

    Google Scholar 

  67. Wang, Z.: Airfoil geometry design for minimum drag. Technical Report AAE 550, Purdue University (2005)

    Google Scholar 

  68. UIUC Airfoil Coordinates Database (2008), http://www.ae.uiuc.edu/m-selig/ads/coord_database.html

  69. Design and analysis of subsonic isolated airfoils (2008), http://web.mit.edu/drela/public/web/xfoil/

  70. Finkel, D.E., Kelley, C.T.: Additive scaling and the direct algorithm. J. of Global Optimization 36(4), 597–608 (2006)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INTEC-IBBT, Ghent University, Gaston Crommenlaan 8, 9050, Belgium

    Dirk Gorissen & Tom Dhaene

  2. University of Antwerp, Middelheimlaan 1, 2020, Antwerp, Belgium

    Karel Crombecq & Ivo Couckuyt

Authors
  1. Dirk Gorissen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karel Crombecq
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ivo Couckuyt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tom Dhaene
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. College of Business Administration, Quantitative and Information System Department, Kuwait University, P.O. Box 5486, 13055, Safat, Kuwait

    Aboul-Ella Hassanien

  2. Center of Excellence for Quantifiable, Quality of Service, Norwegian University of Science & Technology, O.S. Bragstads plass 2E, 7491, Trondheim, Norway

    Ajith Abraham

  3. Department of Computer and Telecommunications Engineering, University ofWestern Macedonia, Agios Dimitrios Park, 50 100, Kozani, Greece

    Athanasios V. Vasilakos

  4. Dept. Electrical and Computer Engineering, University of Alberta, T6J 2V4, Edmonton,Alberta, Canada

    Witold Pedrycz

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Gorissen, D., Crombecq, K., Couckuyt, I., Dhaene, T. (2009). Automatic Approximation of Expensive Functions with Active Learning. In: Hassanien, AE., Abraham, A., Vasilakos, A.V., Pedrycz, W. (eds) Foundations of Computational, Intelligence Volume 1. Studies in Computational Intelligence, vol 201. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01082-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-01082-8_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-01081-1

  • Online ISBN: 978-3-642-01082-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation