### Local Build Installation Source: https://github.com/git/systemtap/blob/master/AGENTS.md Configure the build with a local prefix for convenient local installation, then build and install the package. ```bash configure --prefix=`pwd`/INST # Convenient for local build make all install # Installs to ./INST/ ``` -------------------------------- ### Build and Install SystemTap Source: https://github.com/git/systemtap/blob/master/interactive-notebook/README.md Standard commands to build and install SystemTap from source. ```bash make make install ``` -------------------------------- ### Run Python Example with SystemTap Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.html Executes a Python example script using SystemTap. Ensure the PYTHONPATH is set correctly. ```bash PYTHONPATH=$PY3EXECDIR stap -g --suppress-time-limits -c \ '/usr/bin/python3 -m HelperSDT pyexample.py 35' pyexample.stp python3 ``` -------------------------------- ### Systemtap Examples Cell Source: https://github.com/git/systemtap/blob/master/interactive-notebook/ISystemtap.ipynb Demonstrates the '%%examples' cell for searching, previewing, and running Systemtap example scripts. It shows how to specify script names and arguments. ```Systemtap %%examples ``` -------------------------------- ### Start SystemTap Container Source: https://github.com/git/systemtap/blob/master/interactive-notebook/README.md A quick command to start the SystemTap Jupyter container, assuming it's already pulled or built. ```bash stap-jupyter-container --run ``` -------------------------------- ### Install vim-plug for Vim Source: https://github.com/git/systemtap/blob/master/language-server/README.md Installs the Minimalist Vim plugin manager. This is a prerequisite for installing other Vim plugins. ```bash # For unix installation $ curl -fLo ~/.vim/autoload/plug.vim --create-dirs \ https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim ``` -------------------------------- ### Install ISystemtap Locally Source: https://github.com/git/systemtap/blob/master/interactive-notebook/README.md Installs the ISystemtap kernel, syntax highlighting, and language server dependencies in the user's SystemTap Jupyter directory. ```bash stap-jupyter-install ``` -------------------------------- ### SystemTap Example Search Widget Source: https://github.com/git/systemtap/blob/master/interactive-notebook/tests/stap_test_widgets.ipynb This Python code defines a VBox widget for searching SystemTap examples. It includes options to search by example name, keywords, and a text field for title content. The widget provides buttons for searching and clearing the search criteria. ```python %%examples Result: VBox(children=(VBox(children=(Label(value='Systemtap Example Search', style=LabelStyle(font_size='1.5em', font_weight='bold')), HBox(children=(Label(value='Name ', style=LabelStyle(font_weight='bold')), Combobox(value='', ensure_option=True, options=( 'perf.stp', 'syscallerrorsbypid.stp', 'functioncallcount.stp', 'fntimes.stp', 'fileline-profile.stp', 'pf3.stp', 'errno.stp', 'latencytap.stp', 'timeout.stp', 'ioctl_handler.stp', 'container_check.stp', 'syscallsbypid.stp', 'topsys.stp', 'syscalllatency.stp', 'ucalls.stp', 'pf2.stp', 'sched_switch.stp', 'periodic.stp', 'syscallsrw.stp', 'pf4.stp', 'linetimes.stp', 'thread-times.stp', 'execsnoop-nd.stp', 'fslatency-nd.stp', 'accept2close-nd.stp', 'killsnoop-nd.stp', 'biolatency-nd.stp', 'syscallbypid-nd.stp', 'fsslower-nd.stp', 'bitesize-nd.stp', 'opensnoop-nd.stp', 'rwtime-nd.stp', 'mmwriteback.stp', 'vm.tracepoints.stp', 'hw_watch_addr.stp', 'mmreclaim.stp', 'numa_faults.stp', 'hugepage_split.stp', 'cachestat.stp', 'overcommit.stp', 'last_100_frees.stp', 'hugepage_clear_delays.stp', 'hugepage_collapse.stp', 'hw_watch_sym.stp', 'glibc-malloc.stp', 'mmfilepage.stp', 'hugepage_cow_delays.stp', 'pfaults.stp', 'kmalloc-top', 'mmanonpage.stp', 'interrupts-by-dev.stp', 'scf.stp', 'qemu_count.stp', 'qemu_io.stp', 'kvm_service_time.stp', 'callgraph.stp', 'graphs.stp', 'badname.stp', 'sizeof.stp', 'whythefail.stp', 'keyhack.stp', 'pyexample.stp', 'regex.stp', 'func_time_stats.stp', 'sizeof_interactive.stp', 'also_ran.stp', 'stopwatches.stp', 'socket-events.stp', 'eventcount.stp', 'tcl-funtop.stp', 'watchdog.stp', 'measureinterval.stp', 'helloworld.stp', 'cpu_throttle.stp', 'para-callgraph.stp', 'key.stp', 'ansi_colors2.stp', 'ansi_colors.stp', 'alias_suffixes.stp', 'para-callgraph-verbose.stp', 'tcl-trace.stp', 'floatingpoint.stp', 'varwatch.stp', 'gmalloc_watch.stp', 'php-trace.stp', 'stap_time.stp', 'libguestfs_log.stp', 'mbrwatch.stp', 'traceaio.stp', 'eatmydata.stp', 'ioblktime.stp', 'traceio2.stp', 'inodewatch2.stp', 'enospc.stp', 'iodevstats.stp', 'iotop.stp', 'capture_ssl_master_secrets.stp', 'slowvfs.stp', 'iostats.stp', 'nfs_func_users.stp', 'traceio.stp', 'deviceseeks.stp', 'iotime.stp', 'inodewatch.stp', 'io_submit.stp', 'switchfile.stp', 'disktop.stp', 'iostat-scsi.stp', 'ttyspy.stp', '2048.stp', 'lifegame.stp', 'pingpong.stp', 'block.stp', 'eater.stp', 'bkl.stp', 'bkl_stats.stp', 'cve-2012-0056.stp', 'cve-2013-2094.stp', 'cve-2016-5195.stp', 'cve-2015-0235.stp', 'cve-2011-4127.stp', 'cve-2015-3456.stp', 'cve-2016-0728.stp', 'cve-2017-6074.stp', 'cve-2008-0600.stp', 'cve-2015-7547.stp', 'cve-2021-4155.stp', 'cve-2021-4034.stp', 'cve-2014-7169.stp', 'cve-2018-1000001.stp', 'cve-2018-14634.stp', 'cve-2018-6485.stp', 'nfsdtop.stp', 'socktop', 'tcpdumplike.stp', 'netfilter_summary_json.stp', 'stp_dump.stp', 'tcp_trace.stp', 'nfsd-trace.stp', 'tcp_retransmission.stp', 'netdev.stp', 'dropwatch.stp', 'tcp_connections.stp', 'netfilter_summary.stp', 'tcp_init_cwnd.stp', 'tcpipstat.stp', 'autofs4.stp', 'sk_stream_wait_memory.stp', 'who_sent_it.stp', 'nfsd-recent.stp', 'nettop.stp', 'net_xmit_json.stp', 'netfilter_drop.stp', 'nfsd_unlink.stp', 'packet_contents.stp', 'connect_stat.stp', 'socket-trace.stp', 'syscalls_by_pid.stp', 'sleeptime.stp', 'rlimit_nofile.stp', 'futexes2.stp', 'threadstacks.stp', 'migrate.stp', 'procmod_watcher.stp', 'pfiles.stp', 'ltrace.stp', 'errsnoop.stp', 'forktracker.stp', 'pstrace_exec.stp', 'semop-watch.stp', 'sleepingBeauties.stp', 'schedtimes.stp', 'psig.stp', 'thread-business.stp', 'mutex-contention.stp', 'noptrace.stp', 'wait4time.stp', 'strace.stp', 'plimit.stp', 'sigmon.stp', 'pstree.stp', 'spawn_seeker.stp', 'syscalltimes', 'sig_by_proc.stp', 'futexes.stp', 'sig_by_pid.stp', 'proctop.stp', 'auditbt.stp', 'syscalls_by_proc.stp', 'sched-latency.stp', 'chng_cpu.stp', 'cycle_thief.stp', 'sigkill.stp'), placeholder='ExampleScript.stp')), Label(value='Title Contains', style=LabelStyle(font_weight='bold')), Text(value=''))), HBox(children=(Label(value='Keywords', style=LabelStyle(font_weight='bold')), TagsInput(value=[], allow_duplicates=False, allowed_tags=['thread', 'tty', 'diagram', 'process', 'regex', 'watchpoint', 'interrupt', 'nanosleep', 'guru', 'simple', 'socket', '_best', 'trace', 'function', 'limits', 'futex', 'retransmission', 'profiling', 'tcp', 'tracepoint', 'qemu', 'nfs', 'statistics', 'kvm', 'filesystem', 'monitoring', 'backtrace', 'watchdog', 'floatingpoint', 'application', 'file', 'security', 'numa', 'stapgames', 'container', 'io', 'json', 'signals', 'syscall', 'memory', 'virtualization', 'scsi', 'locking', 'packets', 'scheduler', 'traffic', 'callgraph', 'dyninst', 'interactive', 'speculation', 'format', 'bpf', 'time', 'prometheus', 'disk', 'network']))), HBox(children=(Button(button_style='info', icon='search', style=ButtonStyle(), tooltip='Search'), Button(button_style='info', icon='refresh', style=ButtonStyle(), tooltip='Clear'))))), HBox(children=(VBox())))) ``` -------------------------------- ### Measure Interval in SystemTap Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/keyword-index.txt This example demonstrates how to measure intervals using SystemTap. It aggregates intervals into a single entry if no sixth argument is provided. ```bash # stap measureinterval.stp "gettimeofday_us()" "tid()" @sum vm.pagefault \ vm.pagefault.return "execname()" -c 'sleep 3' ``` -------------------------------- ### Running fslatency-nd.stp with Interval and Count Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/lwtools/fslatency-nd_example.txt This example shows how to run the fslatency-nd.stp script with optional interval and count arguments to control the reporting frequency. Use this to monitor file system latency over specific time periods. ```Shell fslatency-nd.ntp takes an optional interval and count as arguments. Eg, to show latency every second: ``` -------------------------------- ### proctop.stp Sorted by State Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/process/proctop.txt This example demonstrates how to configure proctop.stp to sort processes by their state, showing a short list of processes that have stopped while the script was running. It also limits the output to 5 processes and a 5-second runtime. ```shell stap proctop.stp -T 12 -G show=5 -G sort=state -G sortby=decrease ``` -------------------------------- ### Measure read/write syscall latency for 'bash' processes Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/lwtools/rwtime-nd_example.txt This example shows how to use rwtime-nd.stp to measure read() and write() syscall latency for processes named 'bash'. The output highlights significantly higher latencies compared to the 'node' example, with a detailed histogram for read latencies. ```bash # ./rwtime-nd.stp bash Tracing read/write syscalls for processes named "bash"... Hit Ctrl-C to end. ^C syscall read latency (ns): value |-------------------------------------------------- count 256 | 0 512 | 0 1024 |@ 1 2048 |@ 1 4096 | 0 8192 | 0 16384 | 0 32768 |@ 1 65536 |@@@@@@ 6 131072 |@@@@@@@@@ 11 262144 | 0 524288 | 0 ~ 2097152 | 0 4194304 | 0 8388608 |@ 1 16777216 | 0 33554432 |@ 1 67108864 |@@@ 3 134217728 |@@@@@@@@@ 9 268435456 |@@@ 3 536870912 |@@ 2 1073741824 |@@ 2 2147483648 | 0 4294967296 |@ 1 8589934592 | 0 17179869184 | 0 syscall write latency (ns): value |-------------------------------------------------- count 512 | 0 1024 | 0 2048 |@@@@@@@@@@@@@@@@@@@@@ 21 4096 |@@@@@@@@@@@@@@@@@@@@@@ 22 8192 |@@@@@@@@@@@@@@@@@@ 18 16384 | 0 32768 | 0 ``` -------------------------------- ### Vim Configuration for Systemtap Language Server Source: https://github.com/git/systemtap/blob/master/language-server/README.md Configures Vim to use the Systemtap language server via the vim-lsp plugin. Includes setup for completion and server registration. ```vim " The Vim Language-Server plugins call plug#begin() Plug 'prabirshrestha/vim-lsp' Plug 'prabirshrestha/asyncomplete.vim' Plug 'prabirshrestha/asyncomplete-lsp.vim' call plug#end() " Optional logs " let g:lsp_log_verbose = 1 " let g:lsp_log_file = expand('LOG_FILE_NAME.log') " Tab Completion " Tab and Shift-Tab back and forth through completion options " Tab Ctrl-y to accept a completion inoremap pumvisible() ? "\" : "\" inoremap pumvisible() ? "\" : "\" inoremap pumvisible() ? "\\" : "\" if (executable('stap')) autocmd User lsp_setup call lsp#register_server({ \ 'name': 'systemtap-language-server', \ 'cmd': {server_info->['stap', '--language-server']}, \ 'whitelist': ['stp'] \ }) endif ``` -------------------------------- ### Example Output of varwatch.stp Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/general/varwatch.txt This output displays the traced events from the varwatch.stp script. It shows calls to various 'sys_*' functions within 'fs/open.c', including their line numbers and captured parameters. The output is generated by running the 'ls' command. ```text sh[32715] kernel.function("SyS_access@fs/open.c:395") $$parms: filename=0x7f340e5cb750 mode=0x4 sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340e5ca048 flags=0x80000 mode=0x1 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340e7c22cb flags=0x80000 mode=0x7f340e7d0168 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340e7c9e89 flags=0x80000 mode=0x7f340e7d0168 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340e7caf39 flags=0x80000 mode=0x7f340e7d0168 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x4a8ddf flags=0x802 mode=0x6eb0c8 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340df403d0 flags=0x80000 mode=0x7f340e17a768 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340df3d2ea flags=0x80000 mode=0x1b6 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? sh[32715] kernel.function("SyS_open@fs/open.c:992") $$parms: filename=0x7f340df3ea80 flags=0x0 mode=0x0 sh[32715] kernel.function("SyS_close@fs/open.c:1053") $$parms: fd=0x3 sh[32715] kernel.function("SyS_close@fs/open.c:1058") $$parms: fd=? [...] ``` -------------------------------- ### SystemTap Floating Point Program Example Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/keyword-index.txt A basic SystemTap script that implements a 'Floating Point' program. It extracts floating point data and performs various operations. ```SystemTap # stap floatingpoint.stp -c "stap --benchmark-sdt" ``` -------------------------------- ### Run SystemTap Container Source: https://github.com/git/systemtap/blob/master/interactive-notebook/README.md Manages the SystemTap Jupyter container. Use --run to start, --pull to fetch, --build to create locally, --publish to upload, and --remove to delete. ```bash stap-jupyter-container [--repo REPOSITORY] [--image IMAGE] [--tag TAG] [--keyname KEYNAME] --{run, pull, build, publish, remove} ``` -------------------------------- ### KVM Exit Reason Statistics Output Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/virtualization/kvm_service_time.txt Example output from the kvm_service_time.stp script, showing counts and time statistics for various KVM exit reasons. The output format includes reason code, count, total time, minimum time, average time, and maximum time in microseconds. ```text # Intel exit reasons are EXIT_REASON_* in linux/arch/x86/include/asm/vmx.h # reason: count sum(us) min(us) avg(us) max(us) 0: 55 123 0 2 12 1: 16668 23165 0 1 77 7: 65 129 0 1 21 12: 182 29145513 58 160140 500054 28: 1 2 2 2 2 30: 1561 5126 0 3 43 32: 10728 7748 0 0 32 40: 4966918 1273170 0 0 47 49: 105 201 0 1 22 ``` -------------------------------- ### SystemTap Thread Syscall Report (Idle System) Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/process/thread-business.txt Example output of the 'thread-business.stp' script when the system is idle. It shows the 'execname', 'tid', 'count' of syscalls, and a list of 'recent syscalls' for various threads. ```text execname tid count recent syscalls stapio 20196 50 ppoll read ppoll read ppoll read ppoll read ppoll sleep 20194 30 nanosleep close mmap2 fstat open brk brk brk munma systemd-udevd 491 23 epoll_wait munmap close read read mmap2 fstat open irqbalance 625 15 nanosleep munmap close read read mmap2 fstat open systemd-udevd 20193 14 exit_group close close munmap munmap munmap munmap stapio 20192 8 pselect6 fcntl read fcntl kill fcntl read fcntl stapio 20194 6 execve execve execve rt_sigprocmask rt_sigaction r ``` -------------------------------- ### SystemTap Thread Syscall Report (System Under Load) Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/process/thread-business.txt Example output of the 'thread-business.stp' script during a period of high system activity, such as during a build process. It highlights the increased syscall activity and variety across different processes. ```text $ stap thread-business.stp -T 5 execname tid count recent syscalls sh 20967 13559 exit_group rt_sigprocmask rt_sigprocmask read lsee sh 20826 5088 exit_group rt_sigprocmask rt_sigprocmask read lsee git-diff-files 21404 2967 exit_group write fstat mmap2 fstat lstat lstat lst git-diff-files 21348 2967 exit_group write fstat mmap2 fstat lstat lstat lst sshd 9634 2394 select clock_gettime write clock_gettime rt_sigpro make 21312 1110 exit_group munmap close write chdir stat rt_sigpro make 21368 1110 exit_group munmap close write chdir stat rt_sigpro make 21411 1011 exit_group munmap close write chdir rt_sigprocmask make 21410 983 exit_group munmap close write chdir rt_sigprocmask sh 21370 978 exit_group read rt_sigprocmask rt_sigprocmask read sh 21314 978 exit_group read rt_sigprocmask rt_sigprocmask read git-describe 21344 759 exit_group write fstat close getdents getdents ope git-describe 21400 759 exit_group write fstat close getdents getdents ope sh 20699 539 exit_group rt_sigprocmask rt_sigprocmask rt_sigact find 20825 461 exit_group close munmap close write close fchdir c sh 21499 403 exit_group rt_sigreturn wait4 rt_sigprocmask close find 20965 381 exit_group close munmap close write close fchdir c install 21505 359 exit_group close close close lseek lsetxattr close sh 21515 340 exit_group rt_sigreturn wait4 rt_sigprocmask close make 21367 334 exit_group munmap close write chdir rt_sigprocmask execname tid count recent syscalls ``` -------------------------------- ### Measure read/write syscall latency for 'node' processes Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/lwtools/rwtime-nd_example.txt This example demonstrates how to use rwtime-nd.stp to measure the latency of read() and write() syscalls for processes named 'node'. The output shows a histogram of latency values in nanoseconds. ```bash # ./rwtime-nd.stp node Tracing read/write syscalls for processes named "node"... Hit Ctrl-C to end. ^C syscall read latency (ns): value |-------------------------------------------------- count 512 | 0 1024 | 0 2048 | 2 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 278 8192 | 1 16384 | 5 32768 | 0 65536 | 0 syscall write latency (ns): value |-------------------------------------------------- count 4096 | 0 8192 | 0 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 135 32768 |@@ 8 65536 | 0 131072 | 0 ``` -------------------------------- ### Interpreting Bimodal Block I/O Latency Distribution Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/lwtools/biolatency-nd_example.txt This example demonstrates a more complex block I/O latency distribution, showing a bimodal pattern. The output helps identify potential on-disk cache hits and high-latency events likely caused by queueing. ```bash # ./biolatency-nd.stp Tracing block I/O... Hit Ctrl-C to end. ^C bio latency (ns): value |-------------------------------------------------- count 32768 | 0 65536 | 0 131072 | 10 262144 |@@@@@@@@@ 450 524288 |@@@ 159 1048576 | 26 2097152 | 35 4194304 | 18 8388608 |@@ 112 16777216 |@@@@ 195 33554432 |@@@@@@@@@ 452 67108864 |@@@@@@@@@@@@@@@@@@@@@@ 1065 134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2398 268435456 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2008 536870912 | 0 1073741824 | 0 ``` -------------------------------- ### Trace libguestfs Startup Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.txt Prints a log of libguestfs startup steps, showing time elapsed between events and event names. Execute with a timeout of 1 second. ```bash # stap libguestfs_log.stp -T 1 ``` -------------------------------- ### Profile Kernel and User Functions Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/keyword-index.html This script sets up time-based sampling to profile kernel and user functions. Every five seconds, it prints a sorted list of the top twenty kernel and/or user functions with samples. Use options like --ldd, --all-modules, or -d MODULE to add more symbolic information. ```SystemTap stap pf3.stp ``` -------------------------------- ### Build and Publish SystemTap Container Image Source: https://github.com/git/systemtap/blob/master/interactive-notebook/README.md Builds the SystemTap container image locally and then publishes it to quay.io. Requires quay.io login credentials. ```bash stap-jupyter-container --build && stap-jupyter-container --publish ``` -------------------------------- ### SystemTap Hello World Program Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.html A basic SystemTap script that prints a 'hello world' message and exits. Suitable for beginners. ```SystemTap # stap helloworld.stp ``` -------------------------------- ### Running strace.stp Script Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/process/strace.txt This command demonstrates how to run the strace.stp script to trace system calls for a specific command. The output shows the system calls made by the 'echo hello world' command. ```bash $ stap strace.stp -w -c "echo hello world" hello world Fri Feb 13 15:37:16 2015.057695 rt_sigsuspend() = -514 Fri Feb 13 15:37:16 2015.057749 rt_sigreturn() = -4 (EINTR) Fri Feb 13 15:37:16 2015.057756 rt_sigaction(SIGUSR1, {SIG_DFL}, 0x0, 8) = 0 Fri Feb 13 15:37:16 2015.057772 rt_sigprocmask(SIG_SETMASK, [EMPTY], 0x0, 8) = 0 Fri Feb 13 15:37:16 2015.057794 execve("/home/ajakop/work/codebase/install/bin/echo" "hello" "world") = -2 (ENOENT) Fri Feb 13 15:37:16 2015.057813 execve("/usr/local/bin/echo" "hello" "world") = -2 (ENOENT) Fri Feb 13 15:37:16 2015.057821 execve("/usr/bin/echo" "hello" "world") = 0 Fri Feb 13 15:37:16 2015.058032 brk(0x0) = 17408000 Fri Feb 13 15:37:16 2015.058048 mmap2(0x0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbbb2cbc000 Fri Feb 13 15:37:16 2015.058060 access("/etc/ld.so.preload", R_OK) = -2 (ENOENT) Fri Feb 13 15:37:16 2015.058089 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 Fri Feb 13 15:37:16 2015.058099 fstat(3, 0x7fff6992ac80) = 0 Fri Feb 13 15:37:16 2015.058105 mmap2(0x0, 91161, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbbb2ca5000 Fri Feb 13 15:37:16 2015.058110 close(3) = 0 Fri Feb 13 15:37:16 2015.058122 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 Fri Feb 13 15:37:16 2015.058131 read(3, 0x7fff6992ae20, 832) = 832 Fri Feb 13 15:37:16 2015.058136 fstat(3, 0x7fff6992acd0) = 0 Fri Feb 13 15:37:16 2015.058141 mmap2(0x0, 3920480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fbbb26e0000 Fri Feb 13 15:37:16 2015.058147 mprotect(0x7fbbb2894000, 2093056, PROT_NONE) = 0 Fri Feb 13 15:37:16 2015.058155 mmap2(0x7fbbb2a93000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 1781760) = 0x7fbbb2a93000 Fri Feb 13 15:37:16 2015.058166 mmap2(0x7fbbb2a99000, 16992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fbbb2a99000 Fri Feb 13 15:37:16 2015.058176 close(3) = 0 Fri Feb 13 15:37:16 2015.058189 mmap2(0x0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbbb2ca4000 Fri Feb 13 15:37:16 2015.058195 mmap2(0x0, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbbb2ca2000 Fri Feb 13 15:37:16 2015.058202 arch_prctl(4098, 0x7fbbb2ca2740) = 0 Fri Feb 13 15:37:16 2015.058262 mprotect(0x7fbbb2a93000, 16384, PROT_READ) = 0 Fri Feb 13 15:37:16 2015.058273 mprotect(0x606000, 4096, PROT_READ) = 0 Fri Feb 13 15:37:16 2015.058279 mprotect(0x7fbbb2cbd000, 4096, PROT_READ) = 0 Fri Feb 13 15:37:16 2015.058286 munmap(0x7fbbb2ca5000, 91161) = 0 Fri Feb 13 15:37:16 2015.058350 brk(0x0) = 17408000 Fri Feb 13 15:37:16 2015.058354 brk(0x10bb000) = 17543168 Fri Feb 13 15:37:16 2015.058357 brk(0x0) = 17543168 Fri Feb 13 15:37:16 2015.058364 open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 Fri Feb 13 15:37:16 2015.058372 fstat(3, 0x7fbbb2a98b60) = 0 Fri Feb 13 15:37:16 2015.058377 mmap2(0x0, 106066864, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbbac1b8000 Fri Feb 13 15:37:16 2015.058384 close(3) = 0 Fri Feb 13 15:37:16 2015.058416 fstat(1, 0x7fff6992b3c0) = 0 Fri Feb 13 15:37:16 2015.058421 mmap2(0x0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbbb2cbb000 Fri Feb 13 15:37:16 2015.058430 write(1, "hello world\n", 12) = 12 Fri Feb 13 15:37:16 2015.058451 close(1) = 0 Fri Feb 13 15:37:16 2015.058454 munmap(0x7fbbb2cbb000, 4096) = 0 Fri Feb 13 15:37:16 2015.058461 close(2) = 0 Fri Feb 13 15:37:16 2015.058466 exit_group(0) = ``` -------------------------------- ### Install Jupyter LSP Extension Source: https://github.com/git/systemtap/blob/master/language-server/README.md Installs the JupyterLab LSP extension using pip. This extension enables language server support within Jupyter environments. ```bash $ pip install jupyterlab-lsp ``` -------------------------------- ### Uninstall ISystemtap Locally Source: https://github.com/git/systemtap/blob/master/interactive-notebook/README.md Removes the locally installed ISystemtap components. ```bash stap-jupyter-install --remove ``` -------------------------------- ### Hello World Script Source: https://github.com/git/systemtap/blob/master/interactive-notebook/ISystemtap.ipynb A basic 'Hello World' script to demonstrate the '%%edit' and '%%run' cells in ISystemtap. It prints a greeting and the current namespace. ```Systemtap %%edit helloworld probe oneshot{ printf("Hello ISystemtap\nI am in the namespace %s\n", module_name()) } ``` ```Systemtap %%run helloworld ``` -------------------------------- ### Running ttyspy Script Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/io/ttyspy.txt Execute the ttyspy script using the stap command. This will start monitoring terminal activity. ```bash stap -g ttyspy.txt ``` -------------------------------- ### Run SystemTap Script Source: https://github.com/git/systemtap/blob/master/interactive-notebook/RedHatBlog.ipynb Executes a previously defined SystemTap script named 'helloworld'. ```SystemTap %%run helloworld ``` -------------------------------- ### Generate Process Diagram in DOT Form Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.txt Generates a process diagram in DOT form, useful for visualizing processes started during operations like 'make'. ```bash # stap pstree.stp -T 1 ``` -------------------------------- ### Basic Controls Namespace Source: https://github.com/git/systemtap/blob/master/interactive-notebook/ISystemtap.ipynb Introduces basic controls for managing Systemtap scripts, including starting, pausing, and resetting probes and globals within a namespace. ```Systemtap %%edit controls global t = 0 probe timer.s(1) { t++ printf("I have counted up to %d\n", t) } probe end { printf("I have completed\n") } ``` ```Systemtap %%run controls ``` -------------------------------- ### Run bitesize-nd.stp and view I/O size distribution Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/lwtools/bitesize-nd_example.txt Execute the bitesize-nd.stp script to trace block I/O and observe the resulting size distribution histograms for various processes. Press Ctrl-C to stop tracing. ```bash # ./bitesize-nd.stp Tracing block I/O... Hit Ctrl-C to end. ^C I/O size (bytes): process name: flush-20:1 value |-------------------------------------------------- count 1024 | 0 2048 | 0 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 33 8192 |@@@@@@@@@@@@@@@@ 16 16384 |@@@@@@@@@ 9 32768 |@@@@ 4 65536 |@@@ 3 131072 | 0 262144 | 0 process name: kjournald value |-------------------------------------------------- count 1024 | 0 2048 | 0 4096 |@ 4 8192 | 2 16384 | 2 32768 |@ 7 65536 |@ 5 131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 182 262144 | 0 524288 | 0 process name: randread.pl value |-------------------------------------------------- count 1024 | 0 2048 | 0 4096 | 4 8192 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1859 16384 | 0 32768 | 5 65536 | 18 131072 | 0 262144 | 0 process name: tar value |-------------------------------------------------- count 1024 | 0 2048 | 0 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1645 8192 |@@@@@@@@@@@@@@@@@@@@@@@@@ 836 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1394 32768 |@@@@@@@@@@@@@@@@ 532 65536 |@@@@@ 180 131072 |@@@@@@ 219 262144 | 0 524288 | 0 ``` -------------------------------- ### Track Start and Stop of Processes Due to Network Buffer Space Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/keyword-index.txt Prints a timestamp, executable, and other details when processes are affected by network buffer space limitations. ```SystemTap # stap sk_stream_wait_memory.stp ``` -------------------------------- ### Profile Kernel/User Functions Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.html Sets up time-based sampling to print a sorted list of the top twenty kernel and/or user functions with samples every five seconds. Use options like --ldd, --all-modules, -d MODULE, or -d /PATH/TO/EXEC to add more symbolic information. ```SystemTap profiling/pf3.stp ``` -------------------------------- ### Tally Executables Run on System Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.txt Tallies each time an executable is started or a shared library is loaded. Makes information available via procfs in Prometheus readable format. ```bash # stap also_ran.stp -T 1 ``` -------------------------------- ### Profile Kernel and User Backtraces Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/keyword-index.html This script sets up time-based sampling to profile kernel and user stack backtraces on a per-CPU basis. Every five seconds, it prints a sorted list of the top twenty kernel and/or user stack backtraces. Use options like --ldd, --all-modules, or -d MODULE to add more symbolic information. ```SystemTap stap pf4.stp ``` -------------------------------- ### Identify Senders to a Specific Destination Port Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/network/who_sent_it.txt Use this script to find which process is sending packets to a particular destination port. The example filters for DNS traffic (port 53). ```bash # stap who_sent_it.stp -G the_dport=53 -c "ping -c 1 sourceware.org" ``` -------------------------------- ### Execute Systemtap Script with Arguments Source: https://github.com/git/systemtap/blob/master/interactive-notebook/ISystemtap.ipynb This snippet demonstrates how to run a Systemtap script using the `%%script` cell, passing arguments and Systemtap options. It includes probes for the `begin`, `timer`, and `process` points, along with a `syscall` probe to count system call usage. ```python %%script my_own_script -v -g --suppress-time-limits --suppress-handler-errors --args(10) /* * Write a script which prints "Hello World" at the start * Then after 10 seconds, prints a histogram for the top 4 most used systemcalls */ # TODO: Define a global called word1 global word1 = "Hello" # ^ global word2 = "World" probe begin { # TODO: Complete the second argument to word2 printf("%s %s\n", word1, wo) # ^ } global runtime = 1 # TODO: Complete the probe point to timer.s(1) probe timer { # ^ runtime++ printf("%d seconds left\n", $1 - runtime) if(runtime >= $1){ exit() } } global syscall_counts # TODO: Complete the process path to "/bin/python" probe process("/bin/python").syscall{ # ^ if(execname() == "jupyter-lab"){ # TODO: Complete the context variable to $syscall syscall_counts[syscall_name($sys)] <<< 1 # ^ } } probe end { # Iterate over the processeses foreach (syscall in syscall_counts- limit 4) { # TODO: Complete the macro to @hist_linear print(@(syscall_counts[syscall], 1, 1, 1)) # ^ } } ``` -------------------------------- ### Watch Kernel Symbol Accesses Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/index.txt This script watches accesses to the starting address of a kernel symbol and prints a traceback for each access. It requires root privileges to access hardware breakpoints. ```SystemTap # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -T 5 ``` -------------------------------- ### Initialize SystemTap Transport Source: https://github.com/git/systemtap/blob/master/runtime/transport/transport.txt Initializes the SystemTap transport layer by registering procfs entries and starting a work queue for exit operations. This function is part of the module initialization process. ```c int _stp_transport_init(void) { /* allocates buffers, creates /proc/systemtap/stuff */ if (_stp_register_procfs() < 0) return -1; /* starts up the work queue */ schedule_delayed_work(&stp_exit, STP_WORK_TIMER); /* always succeeds */ return 0; } ``` -------------------------------- ### Run pf4.stp for Profiling Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/profiling/pf4.txt Execute the pf4.stp script to profile kernel and user backtraces. Use options like --all-modules and --ldd to include additional symbolic information. The -T option sets the sampling interval. ```bash $ stap pf4.stp -T 5 --all-modules --ldd ``` -------------------------------- ### SystemTap Per-Pass Verbosity Source: https://github.com/git/systemtap/blob/master/AGENTS.md Use per-pass verbosity flags like --vp 02 to get compiler debug messages without polluting stdout, which is useful for test suites. ```bash stap --vp 02 script.stp ``` -------------------------------- ### Python Path with PYTHONPATH Environment Variable Source: https://github.com/git/systemtap/blob/master/python/HelperSDT/README.txt Demonstrates how setting the PYTHONPATH environment variable adds the developer build installation directory to the Python path. This change is effective immediately for the current session. ```python sys.path = [ '/root/rhel7-2.ppc64', '/usr/local/lib/python2.7/site-packages', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages', ] USER_BASE: '/root/.local' (doesn't exist) USER_SITE: '/root/.local/lib/python2.7/site-packages' (doesn't exist) ENABLE_USER_SITE: True ``` -------------------------------- ### Profile Kernel/User Backtraces with Sampling Source: https://github.com/git/systemtap/blob/master/testsuite/systemtap.examples/keyword-index.html Sets up time-based sampling to print the top twenty kernel and/or user stack backtraces every five seconds on a per-cpu basis. Supports adding symbolic info with options like --ldd or -d. ```SystemTap # stap pf4.stp --ldd ```