際際滷

際際滷Share a Scribd company logo

Improved Security Detection & Response via Optimized Alert Output: A Usability Study

Russ McRee
Russ McRee

Dissertation Defense: Improved Security Detection & Response via Optimized Alert Output: A Usability Study

1 of 15
Download to read offline
Improved Security Detection & Response via Optimized Alert Output: A Usability Study CapitolTechnology University Dissertation Defense by G. Russell McRee Dissertation Chair: Ian McAndrew PhD FRAeS Dissertation Committee: Dr. Atta-Ur-Rahman (Examiner), Allen H. Exner (Ex Officio) 17 AUG 2021
Statement of the Problem Organizations risk data breach, loss of valuable human resources, reputation, and revenue due to excessive security alert volume and a lack of fidelity in security event data These organizations face a large burden due to alert overload, where 99% of security professionals surveyed acknowledge that high volumes of security alerts are problematic
Rationale for the Study This study addresses challenges inherent in data overload and complexity, using security data analytics derived from machine learning (ML) and data science models that produce alert output for analysts Security analysts benefit in two ways: Efficiency of results derived at scale via ML models Benefit of quality alert results derived from the same models.
Literature Overview Security data visualization can be used to address related human cognitive limitations (Rajivan, 2011) Giacobe (2013) discussed the effectiveness of visual analytics and data fusion techniques on situation awareness in cyber-security, and focused on visual analytics, data fusion, and cybersecurity Giacobe found that participants using the visual analytics (VA) interface performed better than those on the text-oriented interface, where the visual analytic interface yielded a performance that was quicker and more accurate that the text interface. Giacobe conducted an experiment and survey separately This study merged quasi-experiment in survey
Research Methodology/Design Quantitative, quasi-experimental, explanatory study TechnologyAcceptance Model (TAM) Methodology utilized to statistically measure security analysts acceptance of two security alert output types: visual alert output (VAO) & text alert output (TAO) A qualitative methodology & design was not considered as the business problem is one of data.The studys data-driven findings can contribute to data-informed business decisions.
Data Analysis DV: level of acceptance of the security alert output and is based on the four individual TAM components: PU, PEU, AU, and IU Within-subjects IV: Scenario (3x), all participants subject to all scenarios Between-subjects IV: Maximum Visual Two levels: a preference forVAO in all three scenarios, and a preference forTAO in at least one of the scenarios Mixed ANOVA to test level of acceptance of alert outputs as influenced by the within- subjects variable Scenario and the between-subjects variable Maximum Visual Mann-Whitney U test performed to compare level of acceptance of alert outputs of the two levels of MaximumVisual Friedman test performed to compare level of acceptance across the three scenarios
Findings (non-parametric) Significant difference (U = 863.5, p = 0.023) in level of acceptance of alert output between respondents who selected visual output across all scenarios (n = 59) compared to the respondents who provided mixed responses (n = 22). No significant difference between scenarios (^2 (2)=5.496, < .064). Scenario mean ranks did not differ significantly from scenario to scenario when not also factoring for responses based on output preference (MaximumVisual).
Findings Mixed ANOVA AllTAM measures (留 = .05): a significant main effect of MaximumVisual scores (F(1, 79) = 4.111, p = .046, 侶p2 = .049) on the level of acceptance of alert output as indicated by sum of participants' scores for allTAM components (PU, PEU, AU, and IU) between-subjects
Perceived Usability (留 = .0125): a significant main effect of MaximumVisual scores (F(1, 79) = 7.643, p = .007, 侶p2 = .088) on the level of acceptance of alert output as indicated by sum of participants' scores for Perceived Usability (PU) between-subjects Perceived Ease of Use (留 = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = .842, p = .362, 侶p2 = .011) on the level of acceptance of alert output as indicated by sum of participants' scores for Perceived Ease of Use (PEU) between-subjects Findings: Mixed ANOVA
Findings: Mixed ANOVA AttitudeToward Using (留 = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = 4.566, p = .036, 侶p2 = .055) on the level of acceptance of alert output as indicated by sum of participants' scores for Attitude Toward Using (AU) between-subjects Intention To Use (留 = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = 4.378, p = .040, 侶p2 = .053) on the level of acceptance of alert output as indicated by sum of participants' scores for Intention to Use (IU) between-subjects
Findings RQ1 RQ1: Is there a difference in the level of acceptance of security alert output between those with a preference for visual alert outputs (VAO) and those with a preference for text alert outputs (TAO), withVAO andTAO generated via data science/machine learning methods, as predicted by the Technology Acceptance Model (TAM)? Yes. Non-parametric (between-subjects): U = 863.5, p = 0.023 Parametric: Within-subjects: (F (1.455, 114.915) = 5.634, p = 0.010, 侶p2 = .067) Between-subjects: (F (1, 79) = 4.111, p = .046, 侶p2 = .049)
Findings SQ1 SQ1: Does the adoption ofVAO have a significant impact on the four individualTAM components, perceived usefulness (PU), perceived ease of use (PEU), attitude toward using (AU), and intention to use (IU)? In part. TheTAM components perceived usability (PU) and perceived ease of use (PEU) are not significantly influenced by the adoption ofVAO within-subjects while attitude toward using (AU), and intention to use (IU) are significantly influenced by the adoption ofVAO within-subjects. TheTAM component perceived usability (PU) is significantly influenced by the adoption ofVAO between-subjects.
Findings SQ2 SQ2: Does the adoption ofTAO have a significant impact on the four individualTAM components, perceived usefulness (PU), perceived ease of use (PEU), attitude toward using (AU), and intention to use (IU)? No. No individualTAM component is significantly influenced byTAO adoption, andTAO adoption trailedVAO in near totality.
Recommendations for Research Security analysts likely seek an initial visual alert inclusive of the options to dive deeper into the raw data. A future study could expose the degree to which analysts seek multifaceted options A future study could further explore the perceptions of, and interactions with, dynamic visualizations versus static visualizations Further explore, even under online survey constraints, a framework that more robustly assesses user experience Opportunity exists to develop more nuanced data where information specific to participant gender, location, age group, company or organization size, and business sector could lead to improved insights
Thank you Questions? Once in a while, you get shown the light In the strangest of places if you look at it right ~Garcia/Hunter

More Related Content

Improved Security Detection & Response via Optimized Alert Output: A Usability Study

  • 1. Improved Security Detection & Response via Optimized Alert Output: A Usability Study CapitolTechnology University Dissertation Defense by G. Russell McRee Dissertation Chair: Ian McAndrew PhD FRAeS Dissertation Committee: Dr. Atta-Ur-Rahman (Examiner), Allen H. Exner (Ex Officio) 17 AUG 2021
  • 2. Statement of the Problem Organizations risk data breach, loss of valuable human resources, reputation, and revenue due to excessive security alert volume and a lack of fidelity in security event data These organizations face a large burden due to alert overload, where 99% of security professionals surveyed acknowledge that high volumes of security alerts are problematic
  • 3. Rationale for the Study This study addresses challenges inherent in data overload and complexity, using security data analytics derived from machine learning (ML) and data science models that produce alert output for analysts Security analysts benefit in two ways: Efficiency of results derived at scale via ML models Benefit of quality alert results derived from the same models.
  • 4. Literature Overview Security data visualization can be used to address related human cognitive limitations (Rajivan, 2011) Giacobe (2013) discussed the effectiveness of visual analytics and data fusion techniques on situation awareness in cyber-security, and focused on visual analytics, data fusion, and cybersecurity Giacobe found that participants using the visual analytics (VA) interface performed better than those on the text-oriented interface, where the visual analytic interface yielded a performance that was quicker and more accurate that the text interface. Giacobe conducted an experiment and survey separately This study merged quasi-experiment in survey
  • 5. Research Methodology/Design Quantitative, quasi-experimental, explanatory study TechnologyAcceptance Model (TAM) Methodology utilized to statistically measure security analysts acceptance of two security alert output types: visual alert output (VAO) & text alert output (TAO) A qualitative methodology & design was not considered as the business problem is one of data.The studys data-driven findings can contribute to data-informed business decisions.
  • 6. Data Analysis DV: level of acceptance of the security alert output and is based on the four individual TAM components: PU, PEU, AU, and IU Within-subjects IV: Scenario (3x), all participants subject to all scenarios Between-subjects IV: Maximum Visual Two levels: a preference forVAO in all three scenarios, and a preference forTAO in at least one of the scenarios Mixed ANOVA to test level of acceptance of alert outputs as influenced by the within- subjects variable Scenario and the between-subjects variable Maximum Visual Mann-Whitney U test performed to compare level of acceptance of alert outputs of the two levels of MaximumVisual Friedman test performed to compare level of acceptance across the three scenarios
  • 7. Findings (non-parametric) Significant difference (U = 863.5, p = 0.023) in level of acceptance of alert output between respondents who selected visual output across all scenarios (n = 59) compared to the respondents who provided mixed responses (n = 22). No significant difference between scenarios (^2 (2)=5.496, < .064). Scenario mean ranks did not differ significantly from scenario to scenario when not also factoring for responses based on output preference (MaximumVisual).
  • 8. Findings Mixed ANOVA AllTAM measures (留 = .05): a significant main effect of MaximumVisual scores (F(1, 79) = 4.111, p = .046, 侶p2 = .049) on the level of acceptance of alert output as indicated by sum of participants' scores for allTAM components (PU, PEU, AU, and IU) between-subjects
  • 9. Perceived Usability (留 = .0125): a significant main effect of MaximumVisual scores (F(1, 79) = 7.643, p = .007, 侶p2 = .088) on the level of acceptance of alert output as indicated by sum of participants' scores for Perceived Usability (PU) between-subjects Perceived Ease of Use (留 = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = .842, p = .362, 侶p2 = .011) on the level of acceptance of alert output as indicated by sum of participants' scores for Perceived Ease of Use (PEU) between-subjects Findings: Mixed ANOVA
  • 10. Findings: Mixed ANOVA AttitudeToward Using (留 = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = 4.566, p = .036, 侶p2 = .055) on the level of acceptance of alert output as indicated by sum of participants' scores for Attitude Toward Using (AU) between-subjects Intention To Use (留 = .0125): an insignificant main effect of MaximumVisual scores (F(1, 79) = 4.378, p = .040, 侶p2 = .053) on the level of acceptance of alert output as indicated by sum of participants' scores for Intention to Use (IU) between-subjects
  • 11. Findings RQ1 RQ1: Is there a difference in the level of acceptance of security alert output between those with a preference for visual alert outputs (VAO) and those with a preference for text alert outputs (TAO), withVAO andTAO generated via data science/machine learning methods, as predicted by the Technology Acceptance Model (TAM)? Yes. Non-parametric (between-subjects): U = 863.5, p = 0.023 Parametric: Within-subjects: (F (1.455, 114.915) = 5.634, p = 0.010, 侶p2 = .067) Between-subjects: (F (1, 79) = 4.111, p = .046, 侶p2 = .049)
  • 12. Findings SQ1 SQ1: Does the adoption ofVAO have a significant impact on the four individualTAM components, perceived usefulness (PU), perceived ease of use (PEU), attitude toward using (AU), and intention to use (IU)? In part. TheTAM components perceived usability (PU) and perceived ease of use (PEU) are not significantly influenced by the adoption ofVAO within-subjects while attitude toward using (AU), and intention to use (IU) are significantly influenced by the adoption ofVAO within-subjects. TheTAM component perceived usability (PU) is significantly influenced by the adoption ofVAO between-subjects.
  • 13. Findings SQ2 SQ2: Does the adoption ofTAO have a significant impact on the four individualTAM components, perceived usefulness (PU), perceived ease of use (PEU), attitude toward using (AU), and intention to use (IU)? No. No individualTAM component is significantly influenced byTAO adoption, andTAO adoption trailedVAO in near totality.
  • 14. Recommendations for Research Security analysts likely seek an initial visual alert inclusive of the options to dive deeper into the raw data. A future study could expose the degree to which analysts seek multifaceted options A future study could further explore the perceptions of, and interactions with, dynamic visualizations versus static visualizations Further explore, even under online survey constraints, a framework that more robustly assesses user experience Opportunity exists to develop more nuanced data where information specific to participant gender, location, age group, company or organization size, and business sector could lead to improved insights
  • 15. Thank you Questions? Once in a while, you get shown the light In the strangest of places if you look at it right ~Garcia/Hunter