{"id":716,"date":"2016-04-17T16:44:29","date_gmt":"2016-04-17T16:44:29","guid":{"rendered":"http:\/\/nenadnoveljic.com\/blog\/?p=716"},"modified":"2018-05-17T16:15:06","modified_gmt":"2018-05-17T16:15:06","slug":"arc-resizing-user_reserve_hint_pct","status":"publish","type":"post","link":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/","title":{"rendered":"ZFS ARC Resizing (user_reserve_hint_pct)"},"content":{"rendered":"<h1>Introduction<\/h1>\n<p>While analyzing an IO latency issue I&#8217;ve noticed that\u00a0a completion thread of the QLogic\u00a0HBA \u00a0driver occasionally spends a lot of time in the\u00a0<a href=\"http:\/\/docs.oracle.com\/cd\/E19253-01\/816-5180\/6mbbf0206\/index.html\" target=\"_blank\">cv_wait<\/a> kernel function. <em>cv_wait\u00a0<\/em>waits for the signal coming from the I\/O interrupt handler and atomically allocates the mutex. I traced a single execution of <em>cv_wait<\/em> to narrow down the reason for the elapsed time:<\/p>\n<pre><code>\r\n#!\/usr\/sbin\/dtrace\r\n# pragma D option flowindent\r\n\r\nfbt::cv_wait:entry\r\n\/ arg0 &amp;&amp; arg1 &amp;&amp; \\\r\n\u00a0 caller &gt;= (uintptr_t)&amp;qlc`qlc_completion_thread &amp;&amp; \\\r\n\u00a0 caller &lt; (uintptr_t)&amp;qlc`qlc_completion_thread+sizeof(qlc`qlc_completion_thread)\r\n\/\r\n{\r\nself-&gt;key = arg0 ;\r\nprintf(\"%s %s %d \\n\",probemod,probefunc,timestamp);\r\n}\r\n\r\nfbt:::entry,\r\nfbt:::return\r\n\/ self-&gt;key \/\r\n{\r\nprintf(\"%s %s %d \\n\",probemod,probefunc,timestamp);\r\n}\r\n\r\nfbt::cv_wait:return\r\n\/ self-&gt;key \/\r\n{\r\nexit(0);\r\n}<\/code><\/pre>\n<p>Here I used\u00a0<a href=\"https:\/\/www.joyent.com\/blog\/dtrace-caller-builtin\" target=\"_blank\">Max Bruning&#8217;s cunning trick<\/a> for tracing a function when called from a specific caller. In this case it is\u00a0<em>qlc_completion_thread<\/em>.<\/p>\n<p>Below is the relevent excerpt from the output:<\/p>\n<pre><code>37 -&gt; resume\u00a0\u00a0\u00a0unix resume 27471744230937588\r\n37\u00a0\u00a0\u00a0-&gt; do_splx\u00a0\u00a0\u00a0unix do_splx 27471744238529161<\/code><\/pre>\n<p>In the recorded case, 8 ms were spent in the kernel function <em>resume<\/em> until <em>do_splx<\/em> was called. <em>resume <\/em>is the kernel function which\u00a0handles\u00a0the switching\u00a0of the thread on the\u00a0processor. I have\u00a0captured\u00a0<em>resume <\/em>executions taking up to\u00a0200 ms which is abnormal. <em>do_splxs<\/em>, the next function call, gives us a hint about the possible cause. <em>do_splxs<\/em> is namely\u00a0the function used to process CPU cross-calls. The cross-calls are performed to clean up processor&#8217;s MMU when user processes issue a virtual address unmap operation.<\/p>\n<p>For further reading, I recommend the <a href=\"http:\/\/www.oracle.com\/technetwork\/server-storage\/solaris\/solaris-internals-137242.html\" target=\"_blank\">Solaris Internals<\/a> book by Richard Dougall and Jim Mauro as it is the best\u00a0reference on the\u00a0Solaris kernel.<\/p>\n<h1>Measuring Cross Calls<\/h1>\n<p>As for the next step, I examined the cross call activity on the server:<\/p>\n<pre><code>#!\/usr\/sbin\/dtrace\r\n\r\nsysinfo:::xcalls\r\n{\r\n@[stack(),execname] = count() ;\r\n}\r\n\r\nprofile:::tick-1s\r\n{\r\nprintf(\"\\n\\n%Y\\n\",walltimestamp);\r\nprinta(\"%k %s %@d\\n\",@);\r\ntrunc(@);\r\n}<\/code><\/pre>\n<p>The following stack trace generated\u00a0608&#8217;388 cross calls in just one second! Besides that, it can be seen that the stack has been executed by the <em>kmem_task<\/em> process.<\/p>\n<pre><code>unix`xc_call+0x46\r\nunix`hat_tlb_inval+0x498\r\nunix`x86pte_inval+0xad\r\nunix`hat_pte_unmap+0xec\r\nunix`hat_unload_callback+0x181\r\nunix`hat_unload+0x41\r\nunix`segkmem_free_pages_internal+0x8d\r\nunix`segkmem_free_vn+0x41\r\nunix`<span style=\"color: #ff0000;\">segkmem_zio_free<\/span>+0x2e\r\ngenunix`vmem_xfree+0x10a\r\ngenunix`vmem_free+0x27\r\ngenunix`kmem_slab_destroy+0x87\r\ngenunix`kmem_slab_free+0x2c7\r\ngenunix`kmem_magazine_destroy+0xdf\r\ngenunix`kmem_depot_ws_reap+0x77\r\ngenunix`kmem_cache_reap+0x76\r\ngenunix`kmem_do_processing+0x28\r\ngenunix`kmem_async+0x159\r\ngenunix`kmem_sysdc_pinit+0x9f\r\nunix`thread_start+0x8\r\n<span style=\"color: #ff0000;\">kmem_task 608388<\/span><\/code><\/pre>\n<p>The total CPU time of <em>kmem_task <\/em>was also being very high and was permanently inreasing:<\/p>\n<pre><code>ps -ef | grep kmem_task\r\n...\u00a0 Jul 02 ?\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">18554:51<\/span> kmem_task<\/code><\/pre>\n<h1>ZFS ARC<\/h1>\n<p>By looking at the stack, it can be concluded that the cross-calls\u00a0have been\u00a0caused by memory deallocations. Furthermore, there is a zfs function on the stack (<em>segkmem_zio_free<\/em>). Therefore, the main suspect is the ZFS ARC cache. The ARC resizing operations can be tracked with the <a href=\"https:\/\/github.com\/richardelling\/dtrace\/blob\/master\/arcreap.d\" target=\"_blank\">arcreap.d<\/a> script. The execution of <em>arc_adjust<\/em> indeed correlates with the execution of the observed stack by <em>kmem_task<\/em>\u00a0which causes cross-call storms. Lastly, I observed the size of ARC which was changing over time and was revolving around 140GB:<\/p>\n<pre><code>kstat -n arcstats | grep c\r\nmodule: zfs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instance: 0\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">141817191504<\/span>\r\nc_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 548525592576\r\n\r\nkstat -n arcstats | grep c\r\nmodule: zfs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instance: 0\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">141495408624<\/span>\r\nc_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 548525592576\r\nkstat -n arcstats | grep c\r\nmodule: zfs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instance: 0\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">142111343152<\/span>\r\nc_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 548525592576\r\nkstat -n arcstats | grep c\r\nmodule: zfs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instance: 0\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">142900093648<\/span>\r\nc_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 548525592576\r\nkstat -n arcstats | grep c\r\nmodule: zfs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instance: 0\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">142900038656<\/span>\r\nc_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 548525592576\r\nkstat -n arcstats | grep c\r\nmodule: zfs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instance: 0\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">142890142016\r\n<\/span>c_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 548525592576<\/code><\/pre>\n<p>As c_max equals the total physical memory on the system, it can be concluded that zfs_arc_max is not set. Inspite of that, the Kernel struggled to keep the memory at a certain size which caused numerous ARC shrinks, which, in turn, were the reason for the fatal cross-call storms.<\/p>\n<h1>user_reserve_hint_pct<\/h1>\n<p>Everything above was measured on Solaris 11.2 and 11.3. I crosschecked it with Solaris 11.1 and observerd completetely different behaviour.<\/p>\n<ul>\n<li>No cross calls.<\/li>\n<li><em>kmem_task<\/em> consumed a fraction of the CPU time:<\/li>\n<\/ul>\n<pre><code>ps -ef | grep kmem_task\r\n...\u00a0 Oct 04 ?\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">222:09 kmem_task<\/span><\/code><\/pre>\n<ul>\n<li>memory consumption constant = max_size<\/li>\n<\/ul>\n<pre><code>kstat -n arcstats | grep c\r\nname:\u00a0\u00a0 arcstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 class:\u00a0\u00a0\u00a0 misc\r\nc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">256000000000\r\n<\/span>c_max\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <span style=\"color: #ff0000;\">256000000000<\/span><\/code><\/pre>\n<ul>\n<li>max_size &lt; total memory<\/li>\n<\/ul>\n<p>Then I compared the relevant Kernel parameters and found out the difference:<\/p>\n<p>Solaris 11.1:<\/p>\n<pre><code>zfs_arc_max=0x3B9ACA0000<\/code><\/pre>\n<p>Solaris 11.2,11.3:<\/p>\n<pre><code>user_reserve_hint_pct=60<\/code><\/pre>\n<p>Instead of setting the hard max limit in Solaris 11.2 and 11.3, the new parameter <em>user_reserve_hint_pct<\/em> was set. According to the\u00a0Oracle Documentation\u00a0the new parameter was introduced for flexibility with the intention to replace <em>zfs_arc_max<\/em>. However, as it is not a hard limit it introduced the overhead of continous resizing which causes cross-call storms.<\/p>\n<p>As a solution I removed <em>user_reserve_hint_pct <\/em>and <em>set zfs_arc_max<\/em> also on Solaris\u00a011.3 which\u00a0completely resolved the problem with I\/O outliers.<\/p>\n<h1>Conclusion<\/h1>\n<p>I observed increased I\/O latencies on the HBA device driver. The analysis showed that the most of the time was spent on\u00a0switching the completion thread on the CPU. The reason were the cross call storms which, in turn, were caused by constant shrinking of ZFS ARC cache. The side-effect was caused by the kernel parameter <em>user_reserve_hint_pct.<\/em> As a workaround, I unset the parameter and falled back to\u00a0<em>zfs_arc_max.<\/em><\/p>\n<h1>References<\/h1>\n<ul>\n<li>Richard Dougall and Jim Mauro: <a href=\"http:\/\/www.oracle.com\/technetwork\/server-storage\/solaris\/solaris-internals-137242.html\" target=\"_blank\">Solaris Internals<\/a><\/li>\n<li>Brendan Gregg and Jim Mauro:<a href=\"http:\/\/www.dtracebook.com\/index.php\/Main_Page\" target=\"_blank\"> DTrace Dynamic Tracing in Oracle Solaris<\/a><\/li>\n<li>Oracle Corporation: <a href=\"https:\/\/docs.oracle.com\/cd\/E53394_01\/html\/E54818\/index.html\" target=\"_blank\">Oracle\u00ae Solaris 11.3 Tunable Parameters Reference Manual<\/a><\/li>\n<li>Max Bruning: <a href=\"https:\/\/www.joyent.com\/blog\/dtrace-caller-builtin\" target=\"_blank\">Bruning Questions: DTrace Caller Builtin<\/a><\/li>\n<\/ul>\n<h1>Updates<\/h1>\n<h2>1. December 2016<\/h2>\n<p>This problem is related to the problem described in my other blog post <a href=\"http:\/\/nenadnoveljic.com\/blog\/solaris-11-3-hang-kernel-object-manager\/\" target=\"_blank\">Solaris 11.3 Hangs Because of Kernel Object Manager<\/a> . In the meantime, Oracle released the patches which help mitigate the problem. However, the issue has not been completely resolved yet. Recently, Oracle\u00a0published the following note\u00a0<a href=\"https:\/\/support.oracle.com\/epmos\/faces\/DocumentDisplay?id=2205638.1\" target=\"_blank\">Solaris 11.3: kmem cache reaping of buffers with large number of entries can lead to long delays which could cause Cluster node eviction (Doc ID 2205638.1)<\/a> .<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>New kernel parameter user_reserve_hint_pct for limiting the size of ZFS ARC cache causes frequent shrinks which, in turn, gives rise to CPU cross-call storms.  <a href=\"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/\" class=\"more-link\">Continue Reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[24,19,14],"tags":[],"class_list":["post-716","post","type-post","status-publish","format-standard","hentry","category-dtrace","category-kmem_task","category-solaris"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>ZFS ARC Resizing (user_reserve_hint_pct) - All-round Database Topics<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ZFS ARC Resizing (user_reserve_hint_pct) - All-round Database Topics\" \/>\n<meta property=\"og:description\" content=\"New kernel parameter user_reserve_hint_pct for limiting the size of ZFS ARC cache causes frequent shrinks which, in turn, gives rise to CPU cross-call storms. Continue Reading &rarr;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/\" \/>\n<meta property=\"og:site_name\" content=\"All-round Database Topics\" \/>\n<meta property=\"article:published_time\" content=\"2016-04-17T16:44:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-05-17T16:15:06+00:00\" \/>\n<meta name=\"author\" content=\"Nenad Noveljic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@NenadNoveljic\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nenad Noveljic\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/\"},\"author\":{\"name\":\"Nenad Noveljic\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"headline\":\"ZFS ARC Resizing (user_reserve_hint_pct)\",\"datePublished\":\"2016-04-17T16:44:29+00:00\",\"dateModified\":\"2018-05-17T16:15:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/\"},\"wordCount\":752,\"commentCount\":2,\"articleSection\":[\"DTrace\",\"kmem_task\",\"Solaris\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/\",\"name\":\"ZFS ARC Resizing (user_reserve_hint_pct) - All-round Database Topics\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\"},\"datePublished\":\"2016-04-17T16:44:29+00:00\",\"dateModified\":\"2018-05-17T16:15:06+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/arc-resizing-user_reserve_hint_pct\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ZFS ARC Resizing (user_reserve_hint_pct)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\",\"name\":\"All-round Database Topics\",\"description\":\"Nenad Noveljic\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\",\"name\":\"Nenad Noveljic\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"caption\":\"Nenad Noveljic\"},\"sameAs\":[\"nenad-noveljic-9b746a6\",\"https:\\\/\\\/x.com\\\/NenadNoveljic\"],\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/author\\\/nenad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"ZFS ARC Resizing (user_reserve_hint_pct) - All-round Database Topics","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/","og_locale":"en_US","og_type":"article","og_title":"ZFS ARC Resizing (user_reserve_hint_pct) - All-round Database Topics","og_description":"New kernel parameter user_reserve_hint_pct for limiting the size of ZFS ARC cache causes frequent shrinks which, in turn, gives rise to CPU cross-call storms. Continue Reading &rarr;","og_url":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/","og_site_name":"All-round Database Topics","article_published_time":"2016-04-17T16:44:29+00:00","article_modified_time":"2018-05-17T16:15:06+00:00","author":"Nenad Noveljic","twitter_card":"summary_large_image","twitter_creator":"@NenadNoveljic","twitter_misc":{"Written by":"Nenad Noveljic","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/#article","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/"},"author":{"name":"Nenad Noveljic","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"headline":"ZFS ARC Resizing (user_reserve_hint_pct)","datePublished":"2016-04-17T16:44:29+00:00","dateModified":"2018-05-17T16:15:06+00:00","mainEntityOfPage":{"@id":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/"},"wordCount":752,"commentCount":2,"articleSection":["DTrace","kmem_task","Solaris"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/","url":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/","name":"ZFS ARC Resizing (user_reserve_hint_pct) - All-round Database Topics","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#website"},"datePublished":"2016-04-17T16:44:29+00:00","dateModified":"2018-05-17T16:15:06+00:00","author":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"breadcrumb":{"@id":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/nenadnoveljic.com\/blog\/arc-resizing-user_reserve_hint_pct\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nenadnoveljic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"ZFS ARC Resizing (user_reserve_hint_pct)"}]},{"@type":"WebSite","@id":"https:\/\/nenadnoveljic.com\/blog\/#website","url":"https:\/\/nenadnoveljic.com\/blog\/","name":"All-round Database Topics","description":"Nenad Noveljic","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nenadnoveljic.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa","name":"Nenad Noveljic","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","caption":"Nenad Noveljic"},"sameAs":["nenad-noveljic-9b746a6","https:\/\/x.com\/NenadNoveljic"],"url":"https:\/\/nenadnoveljic.com\/blog\/author\/nenad\/"}]}},"_links":{"self":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/716","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/comments?post=716"}],"version-history":[{"count":2,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/716\/revisions"}],"predecessor-version":[{"id":1202,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/716\/revisions\/1202"}],"wp:attachment":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/media?parent=716"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/categories?post=716"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/tags?post=716"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}