iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: https://unpaywall.org/10.1007/978-3-030-78086-9_33
Using a Neural Network to Detect Anomalies Given an N-gram Profile | SpringerLink
Skip to main content
Springer Nature Link
Log in

Using a Neural Network to Detect Anomalies Given an N-gram Profile

  • Conference paper
  • First Online:
Cyber Security Cryptography and Machine Learning (CSCML 2021)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 12716))

  • 1225 Accesses

Abstract

In order to detect unknown intrusions and runtime errors of computer programs, the cyber-security community has developed various detection techniques. Anomaly detection is an approach that is designed to profile the normal runtime behavior of computer programs in order to detect intrusions and errors as anomalous deviations from the observed normal. However, normal but unobserved behavior can trigger false positives. This limitation has significantly decreased the practical viability of anomaly detection techniques. Reported approaches to this limitation span a simple alert threshold definition to distribution models for approximating all normal behavior based on the limited observation. However, each assumption or approximation poses the potential for even greater false positive rates. This paper presents our study on how to explain the presence of anomalies using a neural network, particularly Long Short-Term Memory, independent of actual data distributions. We present and compare three anomaly detection models, and report on our experience running different types of attacks on an Apache Hypertext Transfer Protocol server. We performed a comparative study, focusing on each model’s ability to detect the onset of each attack while avoiding false positives resulting from unknown normal behavior. Our best-performing model detected the true onset of every attack with zero false positives.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The experiments were conducted with the permission of the system owner.

References

  1. Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity principles, implementations, and applications. ACM Trans. Inf. Syst. Secur. 13(1), 4 (2009)

    Article  Google Scholar 

  2. Anderson, D., Frivold, T., Valdes, A.: Next-generation intrusion detection expert system (NIDES): a summary. Technical Report. Menlo Park, CA: SRI-CSL-95-07. Computer Science Laboratory, SRI International Breiman, L. (1996). Bagging Predictors. Mach. Learn. 24, 123–140 (1995)

    Google Scholar 

  3. Anderson, D., Frivold, T., Valdes, A.: Next-generation Intrusion Detection Expert System (NIDES): A Summary. Technical Report. Menlo Park, CA: SRI-CSL-95-07. Computer Science Laboratory, SRI International (1995)

    Google Scholar 

  4. Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)

    MATH  Google Scholar 

  5. Carter, K.M., Streilein, W.W.: Probabilistic reasoning for streaming anomaly detection. In: Proceedings of Signal Processing Workshop on Statistical Signal and Array Processing, SSP 2012, pp. 377–380. IEEE(2012)

    Google Scholar 

  6. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 15:1–15:58 (2009)

    Google Scholar 

  7. Codenomicon. Heartbleed Bug (2014). http://heartbleed.com/. Accessed 25 Aug 2015

  8. de la Hoz, E., Ortiz, A., Ortega, J., de la Hoz, E.: Network anomaly classification by support vector classifiers ensemble and non-linear projection techniques. In: Pan, J.-S., Polycarpou, M.M., Woźniak, M., de Carvalho, A.C.P.L.F., Quintián, H., Corchado, E. (eds.) HAIS 2013. LNCS (LNAI), vol. 8073, pp. 103–111. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40846-5_11

    Chapter  Google Scholar 

  9. Denning, D.E.: An intrusion detection model. IEEE Trans. Softw. Eng. Vol SE-13. 2, 222–232 (1987)

    Google Scholar 

  10. Elgraini, M., Assem, N., Rachidi, T.: Host intrusion detection for long stealthy system call sequences. In: Proceedings of 2012 Colloquium in Information Science and Technology, CIST 2012, 22–24 October 2012, pp. 96–100. IEEE (2012)

    Google Scholar 

  11. Eskin, E.: Anomaly detection over noisy data using learned probability distributions. In: Proceedings of the International Conference on Machine Learning, pp. 255–262. Morgan Kaufmann (2000)

    Google Scholar 

  12. Eskin, E., Arnold, A., Prerau, M., Portnoy, L., Stolfo, S.: A geometric framework for unsupervised anomaly detection: detecting intrusions in unlabeled data. Adv. Inf. Secur. 6, 77–101 (2002)

    Article  Google Scholar 

  13. Ewell, B.: New Round of Email Worm, Here you have (2010). http://www.symantec.com/connect/blogs/new-round-email-worm-here-you-have. Accessed 25 Aug 2015

  14. Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of Symposium on Security and Privacy, SP 2003, 11–14 May 2003, Oakland, California, USA, pp. 62–75. IEEE (2003)

    Google Scholar 

  15. Forrest, S., Longstaff, T.A.: A sense of self for unix processes. In: Proceedings of Symposium on Security and Privacy, SP 1996, 6–8 May 1996, Oakland, California, USA, pp. 120–128. IEEE (1996)

    Google Scholar 

  16. Forrest, S., Hofmeyr, S., Somayaji, A., (2008). The evolution of system-call monitoring. In: Proceedings of Annual Computer Security Applications Conference, ACSAC 2008, 8–12 December, Anaheim, California, USA, pp. 418–430. IEEE, Washington DC (2008)

    Google Scholar 

  17. Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: Proceedings of the 14th USENIX Security Symposium, 9–13 August 2004, San Diego, California, USA, USENIX, pp. 103–118 (2004)

    Google Scholar 

  18. Gareth, J.: Majority Vote Classifiers: Theory and Applications (Ph.D. Thesis). Stanford University, May 1998

    Google Scholar 

  19. Heller, K., Svore, K., Keromytis, A., Stolfo, S.: One class support vector machines for detecting anomalous windows registry accesses. In: Proceedings of the Workshop on Data Mining for Computer Security in conjunction with the 3rd IEEE International Conference on Data Mining, DMSEC03, 19–22 November 2003, Melbourne, Florida, USA, pp. 2–9 (2003). (www.cs.fit.edu/ pkc/dmsec03)

  20. Henao, R.J., Espinosa, O.J.: Machine learning techniques applied to intruder detection in networks. Proceedings of the 47th International Carnahan Conference on Security Technology, ICCST 2013, 8–11 October 2013, Medelin, pp. 1–6. IEEE (2013)

    Google Scholar 

  21. Ho, T.: Multiple classifier combiation: lessons and next steps. In: Hybrid Methods in Pattern Recognition. World Scientific Press (2002)

    Google Scholar 

  22. Hu, W., Liao, Y., Vemuri, V.: Robust anomaly detection using support vector machines. In: Proceedings of the International Conference on Machine Learning, pp. 282–289 (2003)

    Google Scholar 

  23. Ilgun, K., Kemmerer, R.A., Porras, P.A.: State transition analysis: a rule-based intrusion detection approach. IEEE Trans. Softw. Eng. 21(3), 181–199 (1995)

    Article  Google Scholar 

  24. Kaspersky Lab. What You Should Know About the ’Here You Have’ Worm (2010). http://usa.kaspersky.com/resources/virus/what-you-should-know-about-here-you-have-worm. Accessed 25 Aug 2015

  25. Ko, C.: Logic induction of valid behavior specifications for intrusion detection. In: Proceedings of IEEE Symposium on Security and Privacy, Berkeley, CA, pp. 142–153. IEEE (2000)

    Google Scholar 

  26. Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of the 14th USENIX Security Symposium, 31 July - 5 Aug 2005, Baltimore, MD, USENIX, pp. 161–176 (2005)

    Google Scholar 

  27. Kuang, L., Zulkernine, M.: An anomaly intrusion detection method using the CSI-KNN algorithm. In: Proceedings of ACM Symposium on Applied Computing, SAC 2008, Fortaleza, Ceara, Brazil, pp. 921–926. ACM (2008)

    Google Scholar 

  28. Kumar, S., Spafford, E.H.: A software architecture to support misuse intrusion detection. In: Proceedings of the 18th National Information Security Conference, NISC 1995, 10–13 October 1995, Baltimore, Maryland, USA, NIST/NCSC, Gaithersburg, MD, pp. 194–204 (1995)

    Google Scholar 

  29. Lee, W., Stolfo, S.: Data mining approaches for intrusion detection. In: Proceedings of the 7th USENIX Security Symposium, 26–29 January 1998, San Antonio, Texas, USA.: USENIX, pp. 79–94 (1998)

    Google Scholar 

  30. Lee, W., Xiang, D.: Information-theoretic measures for anomaly detection. In: Proceedings of Symposium on Security and Privacy, SP 2001, 14–16 May 2001, Oakland, California, USA, pp. 130–143. IEEE (2001)

    Google Scholar 

  31. Liao, Y., Vemuri, V.: Use of K-nearest neighbor classifier for intrusion detection. Comput. Secur. 21(5), 439–448 (2002)

    Article  Google Scholar 

  32. Lunt, T., et al.: A Real-time Intrusion Detection Expert System (IDES). Technical Report. Menlo Park, CA: Computer Science Laboratory, SRI International (1992)

    Google Scholar 

  33. Michmerhuizen, D.: Here You Have Spam Teaches an Old Worm a New Trick (2010). https://barracudalabs.com/2010/09/here-you-have-spam-teaches-an-old-worm-a-new-trick/. Accessed 25 Aug 2015

  34. Roberts, S.: Control chart tests based on geometric moving averages. Technometrics 1(3), 239–250 (1959)

    Article  Google Scholar 

  35. Roesch, M.: Snort - Lightweight intrusion detection for networks. In: LISA 1999: 13th Systems Administration Conference, 7–12 November 1999, Seattle, Washington, USA: USENIX, pp. 229–238 (1999)

    Google Scholar 

  36. Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of Symposium on Security and Privacy, SP 2001, 14–16 May 2001, Oakland, California, USA, pp. 144–155. IEEE (2001)

    Google Scholar 

  37. Shevtsova, I.: Sharpening of the upper bound of the absolute constant in the Berry-Esseen inequality. Theor. Probab. Appl. 51(3), 549–553 (2007)

    Article  MathSciNet  Google Scholar 

  38. Tan, K., Maxion, R.: Why 6? defining the operational limits of stide, an anomaly-based intrusion detector. In: Proceedings of Symposium on Security and Privacy, SP 2002, 12–15 May 2002, Oakland, California, USA, pp. 188–201. IEEE (2002)

    Google Scholar 

  39. US-CERT. Malicious Email Campaign Circulating (2009). https://www.us-cert.gov/ncas/current-activity/2010/09/09/Malicious-Email-Campaign-Circulating. Accessed 25 August 2015

  40. US-CERT. Vulnerability Note VU720951 (2014). http://www.kb.cert.org/vuls/id/720951. Accessed 25 Aug 2015

  41. Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of Symposium on Security and Privacy, 14–16 May 2001, Oakland, California, pp. 156–168. IEEE (2001)

    Google Scholar 

  42. Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM conference on Computer and Communications Security, CCS 2002, 17–21 November 2002, Washington, DC, USA, pp. 255–264. ACM, New York, NY (2001)

    Google Scholar 

  43. Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: alternative data models. In: Proceedings of Symposium on Security and Privacy, SP 1999, May 9–12, 1999, Oakland, California, USA, pp. 133–145. IEEE (1999)

    Google Scholar 

  44. Webb, G. (2000). MultiBoosting: a technique for combining boosting and wagging. Mach. Learn. 40, 159–197 (1999)

    Google Scholar 

  45. Wheeler, D.A.: How to Prevent the next Heartbleed (2014). http://www.dwheeler.com/essays/heartbleed.html. Accessed 25 Aug 2015

  46. Xie, F., Xie., L.: Using information theory to measure call site information of system call in anomaly detection. In: Proceedings of the 15th IEEE International Conference on Communication Technology, ICCT 2013, 17 Nov-19 Nov 2013, Guilin, China, pp. 6–10. IEEE (2013)

    Google Scholar 

  47. Yolacan, E., Dy, J., Kaeli, D.: System call anomaly detection using multi-HMMs. In: Proceedings of the 8th IEEE International Conference on Software Security and Reliability-Companion, SERE-C 2014, San Francisco, California, USA, 2014, pp. 25–30. IEEE (2014)

    Google Scholar 

  48. Gers, F.A., Schmidhuber, J.: LSTM recurrent networks learn simple context free and context sensitive languages (PDF). IEEE Trans. Neural Netw. 12(6), 1333–1340. PMID 18249962 doi: 10.1109/72.963769 (2001)

    Google Scholar 

Download references

Acknowledgement

The authors acknowledge and appreciate Dmitry A Cousin, David Waltermire, and Lee Badger at the National Institute of Standards and Technology (NIST) for their review and comments on an early development version of this article. The authors also appreciate Dong H. Jeong regarding his help in the visualization of some of our experimental results.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Technology, University of the District of Columbia, Washington, DC, 20008, USA

    Byunggu Yu & Junwhan Kim

Authors
  1. Byunggu Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junwhan Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junwhan Kim .

Editor information

Editors and Affiliations

  1. Ben-Gurion University of the Negev, Be'er Sheva, Israel

    Shlomi Dolev

  2. Ben-Gurion University of the Negev, Be'er Sheva, Israel

    Oded Margalit

  3. Bar-Ilan University, Tel Aviv, Israel

    Benny Pinkas

  4. Augusta University, Augusta, GA, USA

    Alexander Schwarzmann

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yu, B., Kim, J. (2021). Using a Neural Network to Detect Anomalies Given an N-gram Profile. In: Dolev, S., Margalit, O., Pinkas, B., Schwarzmann, A. (eds) Cyber Security Cryptography and Machine Learning. CSCML 2021. Lecture Notes in Computer Science(), vol 12716. Springer, Cham. https://doi.org/10.1007/978-3-030-78086-9_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78086-9_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78085-2

  • Online ISBN: 978-3-030-78086-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation