狠狠撸

狠狠撸Share a Scribd company logo

Big Data Log File Use Cases And User Interface Design

?Download as DOCX, PDF?
?
Rajasekaran kandhasamy
Rajasekaran kandhasamy

Explained how big data analytics can apply for log files and what kind of user interface is appropriate for each case.

1 of 6
Download to read offline
Big data Analytics On Log Files And UI Design Page 1 BigData Analytics On Log File And UI Design Introduction:...............................................................................................................................1 Specific Focus:.............................................................................................................................1 Generic Focus:............................................................................................................................1 UI Design:...................................................................................................................................2 Introduction: In belowsectionbrieflyexplainedhow bigdataanalyticscan applyfor log files and what kind of user interface is appropriate for each case. Specific Focus: 1. Identify security vulnerabilities a. Anomaly Detection: Use k-means clustering algorithm to detect anomalies, outliers, exceptions, malwares and so forth. b. Identify DoSattacks(simulatemoreusers than websitecan handle): Normally result in an abnormal number of requests (hits) in a short period of time and create firewall rules to block a specific IP Address to avoid DoS. 2. Host/Date based event statistics a. The process for returning all the events for a particular date. b. All eventsinthe collectionforaparticularhost ona particulardate.Thiskind of analysis may be useful for investigating suspicious behavior by a specific user. 3. Predictive analysis a. Categorizing the errors, events and co relating the errors would provide predictive analysis on system down. 4. Automated route cause analysis a. E.g.: Whythe memoryutilizationishighatspecifictime?Correlate system log with web logwithspecifictime andprovide the memoryutilizationishighbecause of highvolume of user logged in as RCA. Generic Focus: Web Logs: 5. Visitors activity statistics: a. Monthly, Weekly, Daily, Hourly visitors, b. Monthly, Weekly, Daily, Hourly Hits, c. Monthly, Weekly, Daily, Hourly Bandwidth, d. Monthly, Weekly, Daily, Hourly Visit Duration e. Monthly, Weekly, Daily, Hourly Page Views per Visitor f. Monthly, Weekly, Daily, Hourly Pages by View Time g. How long they staying in.
Big data Analytics On Log Files And UI Design Page 2 6. Visitors system statistics a. Operating Systems used, b. Browsers used, 7. Geo location statistics a. N/W traffic by county/countries b. Visitors c. Hits d. Bandwidth e. Time served 8. Error statistics SystemLogs (iostat,vmstat,netstat): 9. Predict future resource needs based on long-term predictive reports a. These reports generate a long-term trend line for performance. The data from these reportsisoften used with a linear regression to predict when additional RAMmemory or CPU power is required for the server. 10. Capacity planning predictions a. Use graph algorithmstopredicttoevaluate hardware resourcesrequiredforthe specific environment. UI Design: 1. Monthlycalendarheatmap, ? Bandwidth,Hits,Visitors countspermonthincalendarview. 2. Sparkline charts orderbymost loggedinforthat week.
Big data Analytics On Log Files And UI Design Page 3 ? Col 1 : User id ? Col 2 : No.Of loginsforthatweek ? Col 3 : Monthlyloginsforthe usergroup by 4 weeks ? Col 4 : No.Of page viewsforthatweek ? Col 5 : Monthly page view forthe user groupby 4 weeks ? Col 6 : No.Of errorcodesreturnedforthat week ? Col 7 : Monthlyerror codesview forthe usergroup by4 weeks 3. Dailyactivities table andchart ? Drilldownlinksforeachhits,page views,visitorscolumns. ? Drilldownisuserbasedforthatday.
Big data Analytics On Log Files And UI Design Page 4 4. Popularpagesintable andchart format ? Drilldownlinksforeachhits,visitorscolumns. 5. No of hitsdistributedbymodule ? Claims,Provider,Member,Operationsandsoon
Big data Analytics On Log Files And UI Design Page 5 6. Infographictype display
Big data Analytics On Log Files And UI Design Page 6

More Related Content

Big Data Log File Use Cases And User Interface Design

  • 1. Big data Analytics On Log Files And UI Design Page 1 BigData Analytics On Log File And UI Design Introduction:...............................................................................................................................1 Specific Focus:.............................................................................................................................1 Generic Focus:............................................................................................................................1 UI Design:...................................................................................................................................2 Introduction: In belowsectionbrieflyexplainedhow bigdataanalyticscan applyfor log files and what kind of user interface is appropriate for each case. Specific Focus: 1. Identify security vulnerabilities a. Anomaly Detection: Use k-means clustering algorithm to detect anomalies, outliers, exceptions, malwares and so forth. b. Identify DoSattacks(simulatemoreusers than websitecan handle): Normally result in an abnormal number of requests (hits) in a short period of time and create firewall rules to block a specific IP Address to avoid DoS. 2. Host/Date based event statistics a. The process for returning all the events for a particular date. b. All eventsinthe collectionforaparticularhost ona particulardate.Thiskind of analysis may be useful for investigating suspicious behavior by a specific user. 3. Predictive analysis a. Categorizing the errors, events and co relating the errors would provide predictive analysis on system down. 4. Automated route cause analysis a. E.g.: Whythe memoryutilizationishighatspecifictime?Correlate system log with web logwithspecifictime andprovide the memoryutilizationishighbecause of highvolume of user logged in as RCA. Generic Focus: Web Logs: 5. Visitors activity statistics: a. Monthly, Weekly, Daily, Hourly visitors, b. Monthly, Weekly, Daily, Hourly Hits, c. Monthly, Weekly, Daily, Hourly Bandwidth, d. Monthly, Weekly, Daily, Hourly Visit Duration e. Monthly, Weekly, Daily, Hourly Page Views per Visitor f. Monthly, Weekly, Daily, Hourly Pages by View Time g. How long they staying in.
  • 2. Big data Analytics On Log Files And UI Design Page 2 6. Visitors system statistics a. Operating Systems used, b. Browsers used, 7. Geo location statistics a. N/W traffic by county/countries b. Visitors c. Hits d. Bandwidth e. Time served 8. Error statistics SystemLogs (iostat,vmstat,netstat): 9. Predict future resource needs based on long-term predictive reports a. These reports generate a long-term trend line for performance. The data from these reportsisoften used with a linear regression to predict when additional RAMmemory or CPU power is required for the server. 10. Capacity planning predictions a. Use graph algorithmstopredicttoevaluate hardware resourcesrequiredforthe specific environment. UI Design: 1. Monthlycalendarheatmap, ? Bandwidth,Hits,Visitors countspermonthincalendarview. 2. Sparkline charts orderbymost loggedinforthat week.
  • 3. Big data Analytics On Log Files And UI Design Page 3 ? Col 1 : User id ? Col 2 : No.Of loginsforthatweek ? Col 3 : Monthlyloginsforthe usergroup by 4 weeks ? Col 4 : No.Of page viewsforthatweek ? Col 5 : Monthly page view forthe user groupby 4 weeks ? Col 6 : No.Of errorcodesreturnedforthat week ? Col 7 : Monthlyerror codesview forthe usergroup by4 weeks 3. Dailyactivities table andchart ? Drilldownlinksforeachhits,page views,visitorscolumns. ? Drilldownisuserbasedforthatday.
  • 4. Big data Analytics On Log Files And UI Design Page 4 4. Popularpagesintable andchart format ? Drilldownlinksforeachhits,visitorscolumns. 5. No of hitsdistributedbymodule ? Claims,Provider,Member,Operationsandsoon
  • 5. Big data Analytics On Log Files And UI Design Page 5 6. Infographictype display
  • 6. Big data Analytics On Log Files And UI Design Page 6