{"id":1526,"date":"2017-05-14T23:12:52","date_gmt":"2017-05-14T23:12:52","guid":{"rendered":"http:\/\/nenadnoveljic.com\/blog\/?p=1526"},"modified":"2017-05-14T23:20:14","modified_gmt":"2017-05-14T23:20:14","slug":"dissecting-emulex-hba-driver","status":"publish","type":"post","link":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/","title":{"rendered":"Dissecting Emulex HBA Driver"},"content":{"rendered":"
\n

Some time ago I observed high IO latency on an Oracle Solaris x86 server which hosts Oracle databases. I used DTrace to narrow down the layer on the IO stack which had been causing the problem. It turned out to be the Emulex HBA driver.<\/span><\/p>\n

First, I analyzed the the driver’s source code of illumos<\/a> to get an idea about the workflow which is used to process fiber channel packets. Then, using the acquired knowledge I developed a DTrace script to measure the time the fiber channel packages spent in each part of the driver’s code.\u00a0<\/span><\/span><\/p>\n

This blog post is an index page to the previously published articles.<\/span><\/p>\n

Part 1 – Measuring Latency<\/a>
\n<\/span><\/p>\n

The blog post describes the methodology to measure the time between the generic fiber channel driver makes an IO request and consumes the packet. The time elapsed includes the SAN response time and the processing time in the HBA driver.\u00a0<\/span><\/span><\/p>\n

Part 2 – Packet Flow<\/a>
\n<\/span><\/p>\n

In the second blog post I drilled down and identified three different stages of processing a packet within the HBA driver: I\/O start, I\/O interrupt and I\/O done.<\/span><\/p>\n

Part 3 – Measuring Overhead<\/a>
\n<\/span><\/p>\n

In the third blog post I provided the DTrace script for measuring the time spent in each of the processing stages. Since, in general, the packet gets consumed by a different kernel thread than the one that submitted the I\/O request, I’ve used the memory address of the fiber channel packet for identifying the packet after the interrupt thread puts it into the completion queue.<\/span><\/p>\n

Part 4 – Comparing Latency on Emulex and QLogic HBA Driver<\/a>
\n<\/span><\/p>\n

Finally, I shared my observations on the cause of the latency in the Emulex driver. The problem is, that there is a delay in consuming the packet in the completion thread. I also compared the performance of the Emulex and QLogic drivers.<\/span><\/p>\n

Related reading<\/span><\/strong><\/p>\n

If you’re a Solaris x86 ZFS user and you’re bothered by IO outliers you might be interested in the following blog posts, where I investigated the impact of ZFS ARC maintenance on the performance:<\/span><\/p>\n<\/div>\n

ZFS ARC Resizing (user_reserve_hint_pct)<\/a><\/div>\n
Solaris 11.3 Hangs Because of Kernel Object Manager<\/a><\/div>\n
<\/div>\n
<\/div>\n","protected":false},"excerpt":{"rendered":"

Analyzing Emulex HBA driver performance with DTrace Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[15,14],"tags":[],"class_list":["post-1526","post","type-post","status-publish","format-standard","hentry","category-emulex","category-solaris"],"yoast_head":"\nDissecting Emulex HBA Driver - All-round Database Topics<\/title>\n<meta name=\"description\" content=\"Analyzing Emulex HBA driver performance with DTrace\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Dissecting Emulex HBA Driver - All-round Database Topics\" \/>\n<meta property=\"og:description\" content=\"Analyzing Emulex HBA driver performance with DTrace\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/\" \/>\n<meta property=\"og:site_name\" content=\"All-round Database Topics\" \/>\n<meta property=\"article:published_time\" content=\"2017-05-14T23:12:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-05-14T23:20:14+00:00\" \/>\n<meta name=\"author\" content=\"Nenad Noveljic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@NenadNoveljic\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nenad Noveljic\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/\"},\"author\":{\"name\":\"Nenad Noveljic\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"headline\":\"Dissecting Emulex HBA Driver\",\"datePublished\":\"2017-05-14T23:12:52+00:00\",\"dateModified\":\"2017-05-14T23:20:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/\"},\"wordCount\":368,\"commentCount\":0,\"articleSection\":[\"Emulex\",\"Solaris\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/\",\"name\":\"Dissecting Emulex HBA Driver - All-round Database Topics\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\"},\"datePublished\":\"2017-05-14T23:12:52+00:00\",\"dateModified\":\"2017-05-14T23:20:14+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"description\":\"Analyzing Emulex HBA driver performance with DTrace\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/dissecting-emulex-hba-driver\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Dissecting Emulex HBA Driver\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\",\"name\":\"All-round Database Topics\",\"description\":\"Nenad Noveljic\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\",\"name\":\"Nenad Noveljic\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"caption\":\"Nenad Noveljic\"},\"sameAs\":[\"nenad-noveljic-9b746a6\",\"https:\\\/\\\/x.com\\\/NenadNoveljic\"],\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/author\\\/nenad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Dissecting Emulex HBA Driver - All-round Database Topics","description":"Analyzing Emulex HBA driver performance with DTrace","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/","og_locale":"en_US","og_type":"article","og_title":"Dissecting Emulex HBA Driver - All-round Database Topics","og_description":"Analyzing Emulex HBA driver performance with DTrace","og_url":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/","og_site_name":"All-round Database Topics","article_published_time":"2017-05-14T23:12:52+00:00","article_modified_time":"2017-05-14T23:20:14+00:00","author":"Nenad Noveljic","twitter_card":"summary_large_image","twitter_creator":"@NenadNoveljic","twitter_misc":{"Written by":"Nenad Noveljic","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/#article","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/"},"author":{"name":"Nenad Noveljic","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"headline":"Dissecting Emulex HBA Driver","datePublished":"2017-05-14T23:12:52+00:00","dateModified":"2017-05-14T23:20:14+00:00","mainEntityOfPage":{"@id":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/"},"wordCount":368,"commentCount":0,"articleSection":["Emulex","Solaris"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/","url":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/","name":"Dissecting Emulex HBA Driver - All-round Database Topics","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#website"},"datePublished":"2017-05-14T23:12:52+00:00","dateModified":"2017-05-14T23:20:14+00:00","author":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"description":"Analyzing Emulex HBA driver performance with DTrace","breadcrumb":{"@id":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/nenadnoveljic.com\/blog\/dissecting-emulex-hba-driver\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nenadnoveljic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Dissecting Emulex HBA Driver"}]},{"@type":"WebSite","@id":"https:\/\/nenadnoveljic.com\/blog\/#website","url":"https:\/\/nenadnoveljic.com\/blog\/","name":"All-round Database Topics","description":"Nenad Noveljic","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nenadnoveljic.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa","name":"Nenad Noveljic","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","caption":"Nenad Noveljic"},"sameAs":["nenad-noveljic-9b746a6","https:\/\/x.com\/NenadNoveljic"],"url":"https:\/\/nenadnoveljic.com\/blog\/author\/nenad\/"}]}},"_links":{"self":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/1526","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/comments?post=1526"}],"version-history":[{"count":1,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/1526\/revisions"}],"predecessor-version":[{"id":1538,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/1526\/revisions\/1538"}],"wp:attachment":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/media?parent=1526"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/categories?post=1526"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/tags?post=1526"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}