To log or not to log using heuristics to identif

Authors: Jason King Jon Stallings Maria Riaz Laurie Williams

Publish Date: 2016/08/24

Volume: 22, Issue: 5, Pages: 2684-2717

PDF Link

Abstract

User activity logs should capture evidence to help answer who what when where why and how a security or privacy breach occurred However software engineers often implement logging mechanisms that inadequately record mandatory log events MLEs user activities that must be logged to enable forensicsWe conducted a controlled experiment with 103 computer science students enrolled in a graduatelevel software security course All subjects were first asked to identify MLEs described in a set of requirements statements during the preperiod task In the postperiod task subjects were randomly assigned statements from one type of software artifact traditional requirements usecasebased requirements or user manual one readability score simple or complex and one method standards resource or heuristicsdriven We evaluated subject performance using three metrics statement classification correctness values from 0 to 1 MLE identification correctness values from 0 to 1 and response time seconds We test the effect of the three factors on the three metrics using generalized linear modelsClassification correctness for statements that did not contain MLEs increased 031 from pre to postperiod task MLE identification correctness was inconsistent across treatment groups For simple user manual statements MLE identification correctness decreased 017 and 012 for the standards and heuristicsdriven methods respectively For simple traditional requirements statements MLE identification correctness increased 016 and 017 for the standards and heuristicsdriven methods respectively Average response time decreased 417 s from the pre to postperiod taskWe expected the performance of subjects using the heuristicsdriven method to improve from pre to posttask and to consistently demonstrate higher MLE identification correctness than the standardsdriven and resourcedriven methods across domains and readability levels However neither method consistently helped subjects more correctly identify MLEs at a statistically significant level Our results indicate additional training and enforcement may be necessary to ensure subjects understand and consistently apply the assigned methods for identifying MLEsThis work is funded by the United States National Security Agency NSA Science of Security Lablet Any opinions expressed in this report are those of the authors and do not necessarily reflect the views of the NSA We thank the Realsearch research group for providing helpful feedback on this work This study was approved by the North Carolina State University Institutional Review Board 5354

Keywords: