iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://doi.org/10.1007/978-3-030-78292-4_40
Better Model, Worse Predictions: The Dangers in Student Model Comparisons | SpringerLink
Skip to main content
Springer Nature Link
Log in

Better Model, Worse Predictions: The Dangers in Student Model Comparisons

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12748))

Included in the following conference series:

  • 3354 Accesses

Abstract

The additive factor model is a widely used tool for analyzing educational data, yet it is often used as an off-the-shelf solution without considering implementation details. A common practice is to compare multiple additive factor models, choose the one with the best predictive accuracy, and interpret the parameters of the model as evidence of student learning. In this work, we use simulated data to show that in certain situations, this approach can lead to misleading results. Specifically, we show how student skill distribution affects estimates of other model parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/LearnSphere/WorkflowComponents/tree/dev/AnalysisAfm.

  2. 2.

    https://github.com/LearnSphere/WorkflowComponents/tree/dev/AnalysisPyAfm.

References

  1. Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control 19(6), 716–723 (1974)

    Article  MathSciNet  Google Scholar 

  2. Aleven, V., Koedinger, K.R.: Knowledge component (kc) approaches to learner modeling. In: Sottilare, R.A., Graesser, A., Hu, X., Holden, H. (eds.) Design Recommendations for Intelligent Tutoring Systems, vol. 1, pp. 165–182. US Army Research Laboratory (2013)

    Google Scholar 

  3. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)

    Google Scholar 

  4. Čechák, J., Pelánek, R.: Item ordering biases in educational data. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019. LNCS (LNAI), vol. 11625, pp. 48–58. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23204-7_5

    Chapter  Google Scholar 

  5. Cen, H.: Generalized learning factors analysis: improving cognitive models with machine learning. Ph.D. thesis, Carnegie Mellon University (2009)

    Google Scholar 

  6. Cen, H., Koedinger, K.R., Junker, B.: Is over practice necessary?-Improving learning efficiency with the cognitive tutor through educational data mining. In: Proceedings of Artificial Intelligence in Education, pp. 511–518 (2007)

    Google Scholar 

  7. Durand, G., Goutte, C., Belacel, N., Bouslimani, Y., Leger, S.: Review, computation and application of the additive factor model (AFM). Technical report 23002483. National Research Council Canada (2017)

    Google Scholar 

  8. Effenberger, T., Pelánek, R., Čechák, J.: Exploration of the robustness and generalizability of the additive factors model. In: Proceedings of Learning Analytics & Knowledge, pp. 472–479 (2020)

    Google Scholar 

  9. Käser, T., Koedinger, K., Gross, M.: Different parameters-same prediction: an analysis of learning curves. In: Proceedings of Educational Data Mining (2014)

    Google Scholar 

  10. Koedinger, K.R., Baker, R.S., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J.: A data repository for the EDM community: the PSLC DataShop. Handbook Educ. Data Mining 43, 43–56 (2010)

    Google Scholar 

  11. Koedinger, K.R., McLaughlin, E.A.: Closing the loop with quantitative cognitive task analysis. In: Proceedings of Educational Data Mining (2016)

    Google Scholar 

  12. Koedinger, K.R., McLaughlin, E.A., Stamper, J.C.: Automated student model improvement. In: Proceedings of Educational Data Mining (2012)

    Google Scholar 

  13. Koedinger, K.R., Stamper, J.C., McLaughlin, E.A., Nixon, T.: Using data-driven discovery of better student models to improve student learning. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 421–430. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_43

    Chapter  Google Scholar 

  14. Liu, R., Koedinger, K.R.: Variations in learning rate: student classification based on systematic residual error patterns across practice opportunities. In: Proceedings of Educational Data Mining (2015)

    Google Scholar 

  15. Liu, R., Koedinger, K.R.: Going beyond better data prediction to create explanatory models of educational data. In: The Handbook of Learning Analytics, pp. 69–76 (2017)

    Google Scholar 

  16. Liu, R., Koedinger, K.R., McLaughlin, E.A.: Interpreting model discovery and testing generalization to a new dataset. In: Proceedings of Educational Data Mining, pp. 107–113 (2014)

    Google Scholar 

  17. Long, Y., Holstein, K., Aleven, V.: What exactly do students learn when they practice equation solving?: Refining knowledge components with the additive factors model. In: Proceedings of Learning Analytics and Knowledge, pp. 399–408. ACM (2018)

    Google Scholar 

  18. Murray, R.C., et al.: Revealing the learning in learning curves. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 473–482. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_48

    Chapter  Google Scholar 

  19. Nguyen, H., Wang, Y., Stamper, J., McLaren, B.M.: Using knowledge component modeling to increase domain understanding in a digital learning game. In: Proceedings of Educational Data Mining (2019)

    Google Scholar 

  20. Nixon, T., Fancsali, S., Ritter, S.: The complex dynamics of aggregate learning curves. In: Proceedings of Educational Data Mining, pp. 338–339 (2013)

    Google Scholar 

  21. Pelánek, R.: Metrics for evaluation of student models. J. Educ. Data Mining 7(2), 1–19 (2015)

    Google Scholar 

  22. Pelánek, R.: Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User Model. User-Adapted Interact. 27(3), 313–350 (2017). https://doi.org/10.1007/s11257-017-9193-2

    Article  Google Scholar 

  23. Pelánek, R.: The details matter: methodological nuances in the evaluation of student models. User Model. User-Adapted Interact. 27, 1–29 (2018). https://doi.org/10.1007/s11257-018-9204-y

    Article  Google Scholar 

  24. Rivers, K., Harpstead, E., Koedinger, K.: Learning curve analysis for programming: Which concepts do students struggle with? In: Proceedings of International Computing Education Research, pp. 143–151 (2016)

    Google Scholar 

  25. Schwarz, G.: Estimating the dimension of a model. Ann. Statist. 6(2), 461–464 (1978)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Masaryk University, Brno, Czech Republic

    Jaroslav Čechák & Radek Pelánek

Authors
  1. Jaroslav Čechák
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Radek Pelánek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jaroslav Čechák or Radek Pelánek .

Editor information

Editors and Affiliations

  1. Technion – Israel Institute of Technology, Haifa, Israel

    Ido Roll

  2. Arizona State University, Tempe, AZ, USA

    Danielle McNamara

  3. Utrecht University, Utrecht, The Netherlands

    Sergey Sosnovsky

  4. London Knowledge Lab, London, UK

    Rose Luckin

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Čechák, J., Pelánek, R. (2021). Better Model, Worse Predictions: The Dangers in Student Model Comparisons. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12748. Springer, Cham. https://doi.org/10.1007/978-3-030-78292-4_40

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78292-4_40

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78291-7

  • Online ISBN: 978-3-030-78292-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation