Dtrace Overview
6 likes925 views
The document discusses the DTrace framework, which is a tool for dynamic instrumentation and analysis of live systems. It allows inspection of all parts of a system without slowing it down. The DTrace framework includes probes throughout the operating system and user programs, a D programming language, and a toolkit of scripts for common tasks like analyzing CPU, memory, network and disk usage. A demo shows example scripts for tasks like monitoring system calls and web server performance.
![SYSTEM INSPECTION AND ANALYSIS
D LANGUAGE : AGGREGATIONS
@name[keys] = aggfunction(args)
Used to aggregate data and look for trends
Simple to generate report about :
Total system calls by a process or an application
Total number of read or write by process
Aggregating functions: count() min()
sum() max()
avg() quantize()
20
COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.](https://image.slidesharecdn.com/dtraceoverview-120703153701-phpapp01/85/Dtrace-Overview-20-320.jpg)
Dtrace Overview
- 1. DTRACE Overview J辿r担me Gauthier May 2011 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 2. AGENDA 1. Context 2. DTrace Framework 3. System inspection and analysis 4. DTrace Toolkit 5. Demo 2 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 3. Context DTrace Framework System inspection and analysis DTrace Toolkit Demo 3 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 4. CONTEXT We need tools to debug and observe different situations Understand the Analyze the system system utilization or performance saturation Debug an Debug a application system/kernel crash 4 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 5. CONTEXT in several areas pkill pargs pflags truss mdb pgrep pstop Process Process Process control statistics debugging System Kernel Kernel statistics statistics debugging vmstat lockstat iostat cpustat kstat kmdb Examples of some Solaris commands 5 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 6. CONTEXT Some existing utilities are process based Some are only inspecting certain parts of the system Disks Virtual memory Network Kernel Need a tool to simplify the analysis of all parts of a system This tool must not make the system any slower 6 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 7. Context DTrace Framework System inspection and analysis DTrace Toolkit Demo 7 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 8. DTRACE FRAMEWORK DTrace : A facility for dynamic instrumentation of production systems Another way to debug and observe the entire system and understand the big picture Open source (CDDL = Common Development and Distribution License) Supported platforms : Unix-like systems : 8 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 9. DTRACE FRAMEWORK User level Operating System DTRACE Programs Kernel level Examine how the system works Track down performance problems across many layers of software Locate the cause of aberrant behavior Does not replace or retire other system utilities 9 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 10. DTRACE FRAMEWORK A framework for real-time analysis and observation Designed for live production systems A totally safe way to inspect live data on production systems No need to stop or restart applications Live measurement and interpretation Over 30 000 data monitoring points spread in all system Zero effect when disabled, safe and no system slowdown when enabled 10 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 11. DTRACE FRAMEWORK Everything about DTrace is virtualized per consumer There is no limit on the number of concurrent DTrace consumers Possibility to create custom programs to dynamically instrument the system Example of problems that can be detected and analyzed: High CPU utilization Acute memory pressure Abnormal I/O activity Excessive network traffic 11 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 12. Context DTrace Framework System inspection and analysis DTrace Toolkit Demo 12 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 13. SYSTEM INSPECTION AND ANALYSIS PROBES Probes are programmable sensors placed all over the system A probe fires when the event happens Then, certain actions are executed only if the predicate expression is true General form of a probe clause : probe description /predicate/ { actions } 13 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 14. probe description SYSTEM INSPECTION AND ANALYSIS /predicate/ PROBES { actions } A predicate allow action to only be taken when certain conditions are met Example : Look only for a process which has the pid=1203 Match a process which has the name firefox-bin An action can be record a stack trace, a timestamp, or the argument to a function When some probes fire, DTrace gathers the data and reports it back to you If no action specified, DTrace will just take note of each time the probe fires 14 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 15. SYSTEM INSPECTION AND ANALYSIS PROBES Actions are used to record data to a DTrace buffer Different types of actions: Type of actions Example Explanation trace() Records the result of trace to the buffer Data recording printf() Traces a D expression stack() Records a kernel stack trace stop() Stops the process which has executed the probe Destructive raise() Signal a process at a precise point during execution panic() Force a crash dump exit() Stop tracing and exits Special copyin() Creates a buffer and returns its address strlen() Returns the length of a string in bytes 15 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 16. probe description SYSTEM INSPECTION AND ANALYSIS /predicate/ PROBES { actions } Each probe is uniquely identified by a 4-tuple : < provider, module, function, name > Probe description The name of the module in which the probe is located Module Either the name of a kernel module or the name of a user library If this probe correspond to a specific program location, its the name Function of the program function in which the probe is located. The final component of the tuple is the string name of the probe that Name give you some idea of the probes semantic meaning. Some probes does not have a module and function (e.g. : BEGIN, END) We can identify a probe only by its name 16 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 17. SYSTEM INSPECTION AND ANALYSIS PROVIDERS < provider, module, function, name > DTrace probes come from a set of kernel modules called providers Which correspond to a particular kind of instrumentation We can list the probes that a provider can transmit to the DTrace Framework A provider pass the control to DTrace when you decide to enable one of its probes 17 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 18. SYSTEM INSPECTION AND ANALYSIS PROVIDERS Example of some providers : SYSCALL PROC SCHED Holds the entire Handle process creation CPU scheduling: sleeping, communication from and termination running threads userland to kernel space Sending and handling CPU time: which threads Every system call on the signals are run by which CPU and system for how long IO MIB PROFILE I/O system Counters for Time based probing at Disk input and output management information specific interval of times requests bases To sample anything in I/O by device, process, IP, IPv6, ICMP, IPSec the system every unit time size, filename 18 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 19. SYSTEM INSPECTION AND ANALYSIS D LANGUAGE /* D Program */ A D program consist of one or more probe clauses probe1 description /predicate/ { D Language is like C language with some actions constructs similar with awk : } Support ANSI C operators probe2 description /predicate/ Support Strings { actions D expressions are based on built-in variables : } pid, execname, timestamps, curthread probe3 description /predicate/ No control-flow constructs : loops, if statements { actions Floating-point arithmetic is not permitted (only } Integers) 19 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 20. SYSTEM INSPECTION AND ANALYSIS D LANGUAGE : AGGREGATIONS @name[keys] = aggfunction(args) Used to aggregate data and look for trends Simple to generate report about : Total system calls by a process or an application Total number of read or write by process Aggregating functions: count() min() sum() max() avg() quantize() 20 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 21. SYSTEM INSPECTION AND ANALYSIS D LANGUAGE : SECURITY D program source files userland DTrace consumer: dtrace() D programs are compiled into a safe intermediate form that is used for DTrace safe execution execution when a probe fires environment kernel DTrace providers Programming mistakes : DTrace will report errors and disable instrumentation It is impossible to construct a bad script that would cause damage to the system DTrace handle run-time errors and report them (dividing by zero, dereferencing invalid memory, ) 21 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 22. Context DTrace Framework System inspection and analysis DTrace Toolkit Demo 22 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 23. DTRACE TOOLKIT Collection of over 100 useful documented scripts built on top of DTrace Framework Developed by the OpenSolaris DTrace community The toolkit contains : the scripts the man pages the example documentation the notes files the tutorials The script are sorted by categories 23 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 24. DTRACE TOOLKIT CATEGORIES DTrace Toolkit Extra, User, System Applications Processes Network Memory System Kernel Disk CPU DTrace Framework 24 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 25. Context DTrace Framework System inspection and analysis DTrace Toolkit Demo 25 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 26. DEMO Command line scripts Chime scripts tcpstat system calls System calls counter by process, TCP bytes received and send function, module, httpdstat php calltime Measure PHP elapsed times for Real-time Apache Web Server stats functions 26 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 27. Questions? jerome.gauthier@alcatel-lucent.com 27 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 28. REFERENCES & ADDITIONAL INFORMATION PowerPoint presentations Joyent Performance Visualization Brendan Gregg DTrace & DTrace Toolkit Stefan Parvu Links DTrace community site: http://hub.opensolaris.org/bin/view/Community+Group+dtrace/WebHome Brendan Gregg site: http://www.brendangregg.com/dtrace.html DTrace Blogs: http://dtrace.org/blogs/ DTrace online guide: http://wikis.sun.com/display/DTrace/Documentation DTrace wiki: http://www.solarisinternals.com/wiki/index.php/DTraceToolkit DTrace training: http://dtracehol.com 28 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
- 29. 29 COPYRIGHT 息 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.