An intrusion detection system (IDS) is a security measure that can help system administrators in enterprise environments detect attacks made against computer networks. In order to be a good enterprise security measure, the IDS solution should be effective when it comes to making system operators aware of on-going cyber-attacks. However, it is difficult and costly to evaluate the effectiveness of IDSs by experiments or observations. This paper describes the result of an alternative approach to studying this topic. The effectiveness of 24 different IDS solution scenarios pertaining to remote arbitrary code exploits is evaluated by 165 domain experts. The respondents’ answers were then combined according to Cooke’s classical method, in which respondents are weighted based on how well they perform on a set of test questions. Results show that the single most important factor is whether either a host-based IDS, or a network-based IDS is in place. Assuming that either one or the other is in place, the most important course of action is to tune the IDS to its environment. The results also show that an updated signature database influences the effectiveness of the IDS less than if the vulnerability that is being exploited is well-known and is possible to patch or not.

Abdolmohammadi, M. J., & Shanteau, J. (1992). Personal attributes of expertauditors. Organizational Behavior and Human Decision Processes, 53(2),158–172.

Alserhani, F., Akhlaq, M., Awan, I. U., Mellor, J., Cullen, A. J., & Mirchandani,P. (2009). Evaluating Intrusion Detection Systems in High Speed Networks.2009 Fifth International Conference on Information Assurance and Security,454–459. doi:10.1109/IAS.2009.276

Anderson, J. P. (1980). Computer security threat monitoring andsurveillance. Forth Washington: Technical report, James P. Anderson

Company, Fort Washington, Pennsylvania.

Ashfaq, A., Robert, M., Mumtaz, A., Ali, M., Sajjad, A., & Khayam, S. (2008). A comparative evaluation of anomaly detectors under portscan attacks. In Recent Advances in Intrusion Detection (pp. 351–371). Springer. Retrieved from http://www.springerlink.com/index/x8643207t2174l34.pdf

Ashton, A. H. (1985). Does consensus imply accuracy in accounting studies of decision making? The Accounting Review, 60(2), 173–185.

Axelsson, S. (2000a). Intrusion detection systems: A survey and taxonomy. Technical Report (Vol. 99, pp. 1–15). Göteborg, Sweden.

Axelsson, S. (2000b). The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security, 3(3), 186–205. doi:10.1145/357830.357849

Barry, B. I. A., & Chan, H. A. (2010). Intrusion detection systems. In P. Stavroulakis & M. Stamp (Eds.), Handbook of Information and Communication Security (Vol. 2001, pp. 193–205). Springer. doi:10.1016/S1361-3723(01)00614-5

Biermann, E. (2001). A comparison of Intrusion Detection systems. Computers & Security, 20(8), 676–683. doi:10.1016/S0167-4048(01)00806-9

Bolger, F., & Wright, G. (1994). Assessing the quality of expert judgment: Issues and analysis. Decision Support Systems, 11(1), 1–24. doi:10.1016/0167-9236(94)90061-2

Cavusgil, S. T., & Elvey-Kirk, L. A. (1998). Mail survey response behavior: A conceptualization of motivating factors and an empirical study. European Journal of Marketing, 32(11/12), 1165–1192. doi:10.1108/03090569810243776

Clemen, R. T., & Winkler, R. L. (1999). Combining probability distributions from experts in risk analysis. Risk Analysis, 19(187), 187–204.

Cooke, R. M. (1991). Experts in Uncertainty: Opinions and Subjective Probability in Science. New York, New York, USA: Open University Press.

Cooke, R. M. (2008). TU Delft expert judgment data base. Reliability Engineering & System Safety, 93(5), 657–674. doi:10.1016/j.ress.2007.03.005

Cooke, R. M., & Goossens, L. (2004). Expert judgement elicitation for risk assessments of critical infrastructures. Journal of Risk Research, 7(6), 643–656. From http://www.ingentaconnect.com/content/routledg/rjrr/2004/00000007/00000006/art00008

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. doi:10.1007/BF02310555

Cronbach, L. J., & Shavelson, R. J. (2004). My Current Thoughts on Coefficient Alpha and Successor Procedures. Educational and Psychological Measurement, 64(3), 391–418. doi:10.1177/0013164404266386

Denning, D. E. (1987). An Intrusion-Detection Model. IEEE Transactions on Software Engineering, SE-13(2), 222–232. doi:10.1109/TSE.1987.232894

Faysel & Haque. (2010). Towards Cyber Defense: Research in Intrusion Detection & Intrusion Prevention Systems. Journal of Computer Science, 10(7), 316–325.

Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984). Consensus methods: characteristics and guidelines for use. American Journal of Public Health, 74(9), 979–983. doi:10.2105/AJPH.74.9.979

Garciateodoro, P., Diazverdejo, J., Maciafernandez, G., & Vazquez, E. (2009). Anomaly-based network intrusion detection: Techniques, systems and challenges. Computers & Security, 28(1-2), 18–28. doi:10.1016/j.cose.2008.08.003

Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701.

Goodall, J. R., Lutters, W. G., & Komlodi, A. (2009). Developing expertise for network intrusion detection. Information Technology & People, 22(2),92–108. http://www.emeraldinsight.com/journals.htm?articleid=1793305&show=abstract

Holm, H. (2014). Signature Based Intrusion Detection for Zero-Day Attacks: (Not) A Closed Chapter? In 2014 47th Hawaii International Conference on System Sciences (pp. 4895–4904). Big Island, HI, United states: IEEE. doi:10.1109/HICSS.2014.600

Holm, H., Sommestad, T., Ekstedt, M., & Honeth, N. (2013). Indicators of expert judgement and their significance: an empirical investigation in the area of cyber security. Expert Systems, (Accepted), n/a–n/a. doi:10.1111/exsy.12039

Itoh, T., Takakura, H., Sawada, A., & Koyamada, K. (2006). Visualization of Network Intrusion Detection Data. IEEE Computer Graphics and Applications, 26(2), 40–47.

Julisch, K., & Dacier, M. (2002). Mining intrusion detection alarms for actionable knowledge. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 366–375). New York, New York, USA: ACM. doi:10.1145/775094.775101

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237–251. doi:10.1037/h0034747

Kanoun, W., Cuppens-Boulahia, N., Cuppens, F., Dubus, S., & Martin, A. (2009). Success Likelihood of Ongoing Attacks for Intrusion Detection and Response Systems. 2009 International Conference on Computational Science and Engineering, 83–91. doi:10.1109/CSE.2009.233

Krayer von Krauss, M. P., Casman, E. a, & Small, M. J. (2004). Elicitation of expert judgments of uncertainty in the risk assessment of herbicide-tolerant oilseed crops. Risk Analysis: An Official Publication of the Society for Risk Analysis, 24(6), 1515–27. doi:10.1111/j.0272-4332.2004.00546.x

Ktata, F. B., Kadhi, N. El, & Ghédira, K. (2009). Agent IDS based on Misuse Approach. Journal of Software, 4(6), 495–507. doi:10.4304/jsw.4.6.495-507

Lin, S. (2008). A study of expert overconfidence. Reliability Engineering & System Safety, 93(5), 711–721. doi:10.1016/j.ress.2007.03.014

McFadzean, E., Ezingeard, J.-N., & Birchall, D. (2011). Information Assurance and Corporate Strategy: A Delphi Study of Choices, Challenges, and Developments for the Future. Information Systems Management, 28(2), 102–129. doi:10.1080/10580530.2011.562127

McHugh, J. (2000). Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory. ACM Transactions on Information and System Security, 3(4), 262–294. doi:10.1145/382912.382923

Mell, P., Hu, V., Lippmann, R., Haines, J. W., & Zissman, M. (2003). An overview of issues in testing intrusion detection systems, (NIST IR 7007). Citeseer. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.8.5163

Mell, P., Scarfone, K., & Romanosky, S. (2007). A complete guide to the common vulnerability scoring system version 2.0. Published by FIRST-Forum of Incident Response and Security Teams. Retrieved Jan. 2014 from http://www.first.org/cvss/cvss-guide.pdf

Montgomery, D. C. (2008). Design and analysis of experiments. Hoboken, NJ: John Wiley & Sons Inc.

NIST Computer Security Resource Center (CSRC). (2011). National Vulnerability Database. Retrieved February 13, 2011, from www.nvd.nist.org

Salah, K., & Kahtani, a. (2009). Improving Snort performance under Linux. IET Communications, 3(12), 1883. doi:10.1049/iet-com.2009.0114

Scarfone, K., & Mell, P. (2007). Guide to intrusion detection and prevention systems. Nist Special Publications (Vol. 800). Gaithersburg, MD, USA.

Shaikh, S., Chivers, H., Nobles, P., Clark, J., & Chen, H. (2008). Characterising intrusion detection sensors. Network Security, 2008(9), 10–12. doi:10.1016/S1353-4858(08)70107-7

Shanteau, J., Weiss, D. J., Thomas, R. P., & Pounds, J. C. (2002). Performance-based assessment of expertise: How to decide if someone is an expert or not. European Journal of Operational Research,136(2), 253–263. doi:10.1016/S0377-2217(01)00113-8

Sommestad, T., & Hunstad, A. (2013). Intrusion detection and the role of the system administrator. Information Management & Computer Security, 21(1), 30 – 40. doi:10.1108/09685221311314400

Sumner, M. (2009). Information Security Threats: A Comparative Analysis of Impact, Probability, and Preparedness. Information Systems Management, 26(1), 2–12. doi:10.1080/10580530802384639

Thompson, R. S., Rantanen, E. M., & Yurcik, W. (2006). Network intrusion detection cognitive task analysis: Textual and visual tool usage and recommendations. In Human Factors and Ergonomics Society Annual Meeting Proceedings (Vol. 50, pp. 669–673). Human Factors and Ergonomics Society. www.ingentaconnect.com/content/hfes/hfproc/2006/00000050/00000005/art00011

Thompson, R. S., Rantanen, E. M., Yurcik, W., & Bailey, B. P. (2007). Command line or pretty lines?: comparing textual and visual interfaces for intrusion detection. In Proceedings of the SIGCHI conference on Human factors in computing systems (p. 1205). ACM. Retrieved from http://dl.acm.org/citation.cfm?id=1240807

Wang, K., Cretu, G., & Stolfo, S. (2006). Anomalous Payload-Based Worm Detection and Signature Generation. In Recent Advances in Intrusion Detection (pp. 227–246). Springer. From http://www.springerlink.com/index/75h308806288v3p1.pdf

Weiss, D. J. D. J., & Shanteau, J. (2003). Empirical Assessment of Expertise. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(1), 104–116. doi:10.1518/hfes.45.1.104.27233

Werlinger, R., Hawkey, K., Muldner, K., Jaferian, P., & Beznosov, K. (2008). The challenges of using an intrusion detection system: is it worth the effort? SOUPS ’08 Proceedings of the 4th Symposium on Usable Privacy and Security, (1), 107–118. From http://portal.acm.org/citation.cfm?id=1408679

Xenakis, C., Panos, C., & Stavrakakis, I. (2010). A comparative evaluation of intrusion detection architectures for mobile ad hoc networks. Computers & Security, 30(ii), 1–18. doi:10.1016/j.cose.2010.10.008

Young, G., & Pescatore, J. (2009). Magic quadrant for network intrusion prevention system appliances. Retrieved from http://www.adexsus.com/v2/pdf/Detectores de Intrusos/Gartner/CuadranteMagico.pdf

Zirkle, L. (2008). What is host-based intrusion detection? Intrusion Detection FAQ. From http://www.sans.org/security-resources/idfaq/host_based.php