{"id":4244,"date":"2022-06-14T15:01:55","date_gmt":"2022-06-14T15:01:55","guid":{"rendered":"https:\/\/nenadnoveljic.com\/blog\/?p=4244"},"modified":"2022-06-14T15:03:55","modified_gmt":"2022-06-14T15:03:55","slug":"killsnoop-limitation","status":"publish","type":"post","link":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/","title":{"rendered":"killsnoop Limitation"},"content":{"rendered":"<h1>killsnoop<\/h1>\n<p>The bcc script <a href=\"https:\/\/github.com\/brendangregg\/perf-tools\/blob\/master\/killsnoop\">killsnoop<\/a> traces signals.<\/p>\n<p>A short demo:<\/p>\n<pre><code>\/usr\/share\/bcc\/tools\/killsnoop -s SIGUSR2<\/code><\/pre>\n<pre><code>kill -SIGUSR2 1219543<\/code><\/pre>\n<pre><code>TIME      PID    COMM             SIG  TPID   RESULT\n14:51:23  601489 bash             12   1219543 0<\/code><\/pre>\n<p>The current implementation of killsnoop is based on tracing the <a href=\"https:\/\/man7.org\/linux\/man-pages\/man2\/kill.2.html\">kill()<\/a> system call. kill() sends a signal to a process group or process. killsnoop prints the caller&#8217;s pid and comm, and also extracts the receiver information from the arguments.<\/p>\n<h1>tgkill()<\/h1>\n<p>However, in Linux, there&#8217;s another system call that can send signals: <a href=\"https:\/\/linux.die.net\/man\/2\/tgkill\">tgkill()<\/a>. The system call tgkill() sends a signal to a specific thread. Since killsnoop doesn&#8217;t trace tgkill(), it won&#8217;t show the signals sent by it.<\/p>\n<p>Unlike kill(), tgkill() doesn&#8217;t have a libc wrapper. There isn&#8217;t a shell command that can send signals to a specific thread, either.<\/p>\n<p>The following C program makes a system call to tgkill():<\/p>\n<pre><code>#include <unistd.h>\n#include <signal.h>\n#include <stdlib.h>\n#include <sys syscall.h=\"\">\n\nint main( int argc, char *argv[] ){\n    long ret;\n    int tgid, tid;\n\n    tgid = atoi(argv[1]);\n    tid = atoi(argv[2]);\n\n    ret = syscall(SYS_tgkill, tgid, tid, SIGUSR2);\n    return ret ;\n}<\/sys><\/stdlib.h><\/signal.h><\/unistd.h><\/code><\/pre>\n<p>The signal sent with tgkill() is captured by strace, but doesn&#8217;t appear in the killsnoop output:<\/p>\n<pre><code>my_tgkill 1219543 1219543<\/code><\/pre>\n<pre><code>strace -e trace=none -e signal=SIGUSR2 -p 1219543\n--- SIGUSR2 {si_signo=SIGUSR2, si_code=SI_TKILL, si_pid=244299, si_uid=1000} ---<\/code><\/pre>\n<p>I became aware of the killsnoop limitation <a href=\"https:\/\/nenadnoveljic.com\/blog\/perilous-new-undocumented-tracing-feature-in-oracle-21c\/\">while troubleshooting an Oracle issue where heavy signalling was taking place<\/a>. The Oracle background processes were issuing tgkill() system calls for sending signals that weren&#8217;t captured by killsnoop.<\/p>\n<p>The bpftrace script below traces signals that were sent with tgkill():<\/p>\n<pre><code>sudo bpftrace -e 'BEGIN\n{\n  printf (\"%-16s \", \"TIME\");\n  printf (\"%-16.16s %-6s %-8s %-10s %-12s %4s\\n\", \"COMM\", \"PID\", \"TGID\", \"TPID\", \"SIGNAL\", \"RETURN\");\n}\n\ntracepoint:syscalls:sys_enter_tgkill \n{\n  @args_tgid[tid] = args-&gt;tgid;\n  @args_pid[tid] = args-&gt;pid;\n  @args_sig[tid] = args-&gt;sig;\n}\n\ntracepoint:syscalls:sys_exit_tgkill \n\/ @args_tgid[tid] \/\n{\n  time(\"%D:%M:%S \");\n  printf(\"%-16.16s %-6d %-8d %-10d %-12d %-4d\\n\", comm, pid, @args_tgid[tid], @args_pid[tid], @args_sig[tid], args-&gt;ret);\n  delete(@args_tgid[tid]);\n  delete(@args_pid[tid]);\n  delete(@args_sig[tid]);\n}'<\/code><\/pre>\n<pre><code>TIME             COMM             PID    TGID     TPID       SIGNAL       RETURN\n06\/14\/22:58:42 my_tgkill        304892 1219543  1219543    12           0<\/code><\/pre>\n<p>You can run it in parallel with killsnoop to get the full picture.<\/p>\n<h1>Summary<\/h1>\n<p>The current implementation of the killsnoop bcc script is based on tracing the kill() system call. Consequently, it misses the signals sent by the system call tgkill(). Oracle background processes call tgkill() for sending signals. You can run the bpftrace script provided above concurrently with killsnoop to capture all signals.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The current implementation of the eBPF bcc tool killsnoop doesn&#8217;t capture signals sent by the tgkill() system call. <a href=\"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/\" class=\"more-link\">Continue Reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[53],"tags":[],"class_list":["post-4244","post","type-post","status-publish","format-standard","hentry","category-ebpf"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>killsnoop Limitation - All-round Database Topics<\/title>\n<meta name=\"description\" content=\"The current implementation of the eBPF bcc tool killsnoop doesn&#039;t capture signals sent by the tgkill() system call.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"killsnoop Limitation - All-round Database Topics\" \/>\n<meta property=\"og:description\" content=\"The current implementation of the eBPF bcc tool killsnoop doesn&#039;t capture signals sent by the tgkill() system call.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/\" \/>\n<meta property=\"og:site_name\" content=\"All-round Database Topics\" \/>\n<meta property=\"article:published_time\" content=\"2022-06-14T15:01:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-06-14T15:03:55+00:00\" \/>\n<meta name=\"author\" content=\"Nenad Noveljic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@NenadNoveljic\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nenad Noveljic\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/\"},\"author\":{\"name\":\"Nenad Noveljic\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"headline\":\"killsnoop Limitation\",\"datePublished\":\"2022-06-14T15:01:55+00:00\",\"dateModified\":\"2022-06-14T15:03:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/\"},\"wordCount\":254,\"commentCount\":0,\"articleSection\":[\"eBPF\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/\",\"name\":\"killsnoop Limitation - All-round Database Topics\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\"},\"datePublished\":\"2022-06-14T15:01:55+00:00\",\"dateModified\":\"2022-06-14T15:03:55+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"description\":\"The current implementation of the eBPF bcc tool killsnoop doesn't capture signals sent by the tgkill() system call.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/killsnoop-limitation\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"killsnoop Limitation\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\",\"name\":\"All-round Database Topics\",\"description\":\"Nenad Noveljic\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\",\"name\":\"Nenad Noveljic\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"caption\":\"Nenad Noveljic\"},\"sameAs\":[\"nenad-noveljic-9b746a6\",\"https:\\\/\\\/x.com\\\/NenadNoveljic\"],\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/author\\\/nenad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"killsnoop Limitation - All-round Database Topics","description":"The current implementation of the eBPF bcc tool killsnoop doesn't capture signals sent by the tgkill() system call.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/","og_locale":"en_US","og_type":"article","og_title":"killsnoop Limitation - All-round Database Topics","og_description":"The current implementation of the eBPF bcc tool killsnoop doesn't capture signals sent by the tgkill() system call.","og_url":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/","og_site_name":"All-round Database Topics","article_published_time":"2022-06-14T15:01:55+00:00","article_modified_time":"2022-06-14T15:03:55+00:00","author":"Nenad Noveljic","twitter_card":"summary_large_image","twitter_creator":"@NenadNoveljic","twitter_misc":{"Written by":"Nenad Noveljic","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/#article","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/"},"author":{"name":"Nenad Noveljic","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"headline":"killsnoop Limitation","datePublished":"2022-06-14T15:01:55+00:00","dateModified":"2022-06-14T15:03:55+00:00","mainEntityOfPage":{"@id":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/"},"wordCount":254,"commentCount":0,"articleSection":["eBPF"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/","url":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/","name":"killsnoop Limitation - All-round Database Topics","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#website"},"datePublished":"2022-06-14T15:01:55+00:00","dateModified":"2022-06-14T15:03:55+00:00","author":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"description":"The current implementation of the eBPF bcc tool killsnoop doesn't capture signals sent by the tgkill() system call.","breadcrumb":{"@id":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/nenadnoveljic.com\/blog\/killsnoop-limitation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nenadnoveljic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"killsnoop Limitation"}]},{"@type":"WebSite","@id":"https:\/\/nenadnoveljic.com\/blog\/#website","url":"https:\/\/nenadnoveljic.com\/blog\/","name":"All-round Database Topics","description":"Nenad Noveljic","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nenadnoveljic.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa","name":"Nenad Noveljic","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","caption":"Nenad Noveljic"},"sameAs":["nenad-noveljic-9b746a6","https:\/\/x.com\/NenadNoveljic"],"url":"https:\/\/nenadnoveljic.com\/blog\/author\/nenad\/"}]}},"_links":{"self":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/4244","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/comments?post=4244"}],"version-history":[{"count":2,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/4244\/revisions"}],"predecessor-version":[{"id":4248,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/4244\/revisions\/4248"}],"wp:attachment":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/media?parent=4244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/categories?post=4244"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/tags?post=4244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}