{"id":506,"date":"2016-03-03T22:43:00","date_gmt":"2016-03-03T22:43:00","guid":{"rendered":"http:\/\/nenadnoveljic.com\/blog\/?p=506"},"modified":"2021-11-02T15:53:47","modified_gmt":"2021-11-02T15:53:47","slug":"simultaneous-patching-not-possible-datapatch","status":"publish","type":"post","link":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/","title":{"rendered":"Simultaneous patching not possible with datapatch"},"content":{"rendered":"<h4>Unfortunately&#8230;<\/h4>\n<p>Since the datapatch utility has been introduced it is not longer possible to patch more databases at the same time. Consequently, if you try to patch two databases simultaneously one of them will fail.<\/p>\n<h4>The process<\/h4>\n<p>The (oversimplified) flow chart below depicts the process:<\/p>\n<p><a href=\"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif\" rel=\"attachment wp-att-507\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-507\" src=\"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif\" alt=\"flow\" width=\"369\" height=\"287\"><\/a><\/p>\n<p>The key point for explaining the root cause is that at some point the <em>opatch lsinventory<\/em>&nbsp;is&nbsp;invoked to produce the xml output file which, in turn, gets loaded into the database.<\/p>\n<h4>What&#8217;s the problem?<\/h4>\n<p>The xml file produced by opatch in the <em>$ORACLE_HOME\/QOPatch<\/em> directory file is awkwardly named <em>xml_file.xml<\/em>. Since the file name doesn&#8217;t contain the database name, it will get overriden by&nbsp;a concurrent&nbsp;process running against&nbsp;another database. As a consequence, the error &#8220;ORA-31011: XML parsing failed&#8221; will be thrown.<\/p>\n<h4>Remediation<\/h4>\n<p>Oracle has recognized and&nbsp;addressed this problem by releasing the patches for the following bugs:<\/p>\n<ul>\n<li>19215058: HPI_12102_RAC:ORA-12012 FOUND IN DB ALERT LOG WHEN RESTART ALL NODES&#8217; CRS STACK<\/li>\n<li>20244108: QOPIPREP.BAT MODIFIES XML INVENTORY WHILE READING IT<\/li>\n<\/ul>\n<p>Notes: The both patches update the same script &#8211; $ORACLE_HOME\/QOPatch\/qopiprep.bat. The patch for the bug 20244108 superseeds the patch for the bug 19215058.<\/p>\n<p>After applying the patch the file xml file contains the database name &#8211; <em>xml_file_${ORACLE_SID}.xml<\/em> &#8211; which resolves the&nbsp;problem.<\/p>\n<h4>Any more problems?<\/h4>\n<p>Sadly, the problem is not solved completely, because if the datapatch is invoked more than once in the same second it will invoke opatch which, in turn, will try to create the log file with the timestamp in its name. One of the opatch calls will fail with the exit code 73, because it will not be able to create the logfile which is created by the first process. Beware, the failed opatch will cause datapatch to fail silently, because the erroneous opatch execution&nbsp;is not being&nbsp;caught by datapatch:<\/p>\n<pre><code>Connecting to database...OK\nBootstrapping registry and package to current versions...done\nDetermining current state...\n<\/code><\/pre>\n<p>The opatch error was not obvious, I found&nbsp;it in the log files.<\/p>\n<p>However, the problem can be demonstrated by two concurrent opatch lsinventory calls. The one of them will fail with the following error message:<\/p>\n<pre><code>Inventory load failed... OPatch cannot load inventory for the given Oracle Home.\nLsInventorySession failed: LsInventory cannot create the log directory\n\/u00\/oracle\/orabase\/product\/12.1.0.2.3_a\/cfgtoollogs\/opatch\/lsinv\/lsinventory2016-03-03_11-58-00AM.txt\nOPatch failed with error code 73\n<\/code><\/pre>\n<p>The solution&nbsp;is actually&nbsp;very simple &#8211; it is enough to embedd the process ID in the opatch lsinvetory log file name, so that the names of the log files would be unique even for the processes started in the same second.<\/p>\n<h4>I wish&#8230;<\/h4>\n<p>It would be nice to fix the concurrency problem with opatch and then include the both patches in the future PSUs, as this would reduce the total patching downtime in highly consolidated environments.<\/p>\n<h4>Workaround<\/h4>\n<p>Until&nbsp;this problem gets fixed I&#8217;m planning to implement the following workaround in my patching robot: serialize the calls to datapatch, delay them for a second and hope I won&#8217;t hit any other concurrency issues with opatch. Though, if I do, I&#8217;ll update this blog post with new information.<\/p>\n<p><em>Update 08.03.2016: <\/em>Unfortuntately the described workaround didn&#8217;t work out. The <em>datapatch apply<\/em> invoked the <em>opatch<\/em> utility 4 times leaving 4 log files with the same disturbing pattern:<\/p>\n<pre><code>Mar&nbsp; 9 08:28 lsinventory2016-03-09_08-28-43AM.txt\nMar&nbsp; 9 08:28 lsinventory2016-03-09_08-28-46AM.txt\nMar&nbsp; 9 08:28 lsinventory2016-03-09_08-28-49AM.txt\nMar&nbsp; 9 08:28 lsinventory2016-03-09_08-28-51AM.txt\n<\/code><\/pre>\n<p>Four is the hard coded parallel degree in sqlpatch.pm:<\/p>\n<pre><code>my $catcon_num_processes = 4;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # number of processes to use\n<\/code><\/pre>\n<p><em>Update 19.03:<\/em> Oracle Support suggested the workaround that I successfully tested: <em>datapatch -apply &lt;patch_id&gt; -force -bundle_series DBBP <\/em>. opatch lsinventory is not&nbsp;executed when&nbsp;datapatch is invoked with the force option.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Unfortunately&#8230; Since the datapatch utility has been introduced it is not longer possible to patch more databases at the same time. Consequently, if you try to patch two databases simultaneously one of them will fail. The process The (oversimplified) flow&hellip; <a href=\"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/\" class=\"more-link\">Continue Reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[48,5],"tags":[],"class_list":["post-506","post","type-post","status-publish","format-standard","hentry","category-datapatch","category-oracle"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Simultaneous patching not possible with datapatch - All-round Database Topics<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Simultaneous patching not possible with datapatch - All-round Database Topics\" \/>\n<meta property=\"og:description\" content=\"Unfortunately&#8230; Since the datapatch utility has been introduced it is not longer possible to patch more databases at the same time. Consequently, if you try to patch two databases simultaneously one of them will fail. The process The (oversimplified) flow&hellip; Continue Reading &rarr;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/\" \/>\n<meta property=\"og:site_name\" content=\"All-round Database Topics\" \/>\n<meta property=\"article:published_time\" content=\"2016-03-03T22:43:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-11-02T15:53:47+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif\" \/>\n<meta name=\"author\" content=\"Nenad Noveljic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@NenadNoveljic\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nenad Noveljic\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/\"},\"author\":{\"name\":\"Nenad Noveljic\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"headline\":\"Simultaneous patching not possible with datapatch\",\"datePublished\":\"2016-03-03T22:43:00+00:00\",\"dateModified\":\"2021-11-02T15:53:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/\"},\"wordCount\":568,\"commentCount\":2,\"image\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/nenadnoveljic.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/03\\\/flow.gif\",\"articleSection\":[\"datapatch\",\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/\",\"name\":\"Simultaneous patching not possible with datapatch - All-round Database Topics\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/nenadnoveljic.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/03\\\/flow.gif\",\"datePublished\":\"2016-03-03T22:43:00+00:00\",\"dateModified\":\"2021-11-02T15:53:47+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#primaryimage\",\"url\":\"http:\\\/\\\/nenadnoveljic.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/03\\\/flow.gif\",\"contentUrl\":\"http:\\\/\\\/nenadnoveljic.com\\\/blog\\\/wp-content\\\/uploads\\\/2016\\\/03\\\/flow.gif\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/simultaneous-patching-not-possible-datapatch\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Simultaneous patching not possible with datapatch\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\",\"name\":\"All-round Database Topics\",\"description\":\"Nenad Noveljic\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\",\"name\":\"Nenad Noveljic\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"caption\":\"Nenad Noveljic\"},\"sameAs\":[\"nenad-noveljic-9b746a6\",\"https:\\\/\\\/x.com\\\/NenadNoveljic\"],\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/author\\\/nenad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Simultaneous patching not possible with datapatch - All-round Database Topics","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/","og_locale":"en_US","og_type":"article","og_title":"Simultaneous patching not possible with datapatch - All-round Database Topics","og_description":"Unfortunately&#8230; Since the datapatch utility has been introduced it is not longer possible to patch more databases at the same time. Consequently, if you try to patch two databases simultaneously one of them will fail. The process The (oversimplified) flow&hellip; Continue Reading &rarr;","og_url":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/","og_site_name":"All-round Database Topics","article_published_time":"2016-03-03T22:43:00+00:00","article_modified_time":"2021-11-02T15:53:47+00:00","og_image":[{"url":"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif","type":"","width":"","height":""}],"author":"Nenad Noveljic","twitter_card":"summary_large_image","twitter_creator":"@NenadNoveljic","twitter_misc":{"Written by":"Nenad Noveljic","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#article","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/"},"author":{"name":"Nenad Noveljic","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"headline":"Simultaneous patching not possible with datapatch","datePublished":"2016-03-03T22:43:00+00:00","dateModified":"2021-11-02T15:53:47+00:00","mainEntityOfPage":{"@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/"},"wordCount":568,"commentCount":2,"image":{"@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#primaryimage"},"thumbnailUrl":"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif","articleSection":["datapatch","Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/","url":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/","name":"Simultaneous patching not possible with datapatch - All-round Database Topics","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#primaryimage"},"image":{"@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#primaryimage"},"thumbnailUrl":"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif","datePublished":"2016-03-03T22:43:00+00:00","dateModified":"2021-11-02T15:53:47+00:00","author":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"breadcrumb":{"@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#primaryimage","url":"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif","contentUrl":"http:\/\/nenadnoveljic.com\/blog\/wp-content\/uploads\/2016\/03\/flow.gif"},{"@type":"BreadcrumbList","@id":"https:\/\/nenadnoveljic.com\/blog\/simultaneous-patching-not-possible-datapatch\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nenadnoveljic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Simultaneous patching not possible with datapatch"}]},{"@type":"WebSite","@id":"https:\/\/nenadnoveljic.com\/blog\/#website","url":"https:\/\/nenadnoveljic.com\/blog\/","name":"All-round Database Topics","description":"Nenad Noveljic","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nenadnoveljic.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa","name":"Nenad Noveljic","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","caption":"Nenad Noveljic"},"sameAs":["nenad-noveljic-9b746a6","https:\/\/x.com\/NenadNoveljic"],"url":"https:\/\/nenadnoveljic.com\/blog\/author\/nenad\/"}]}},"_links":{"self":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/506","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/comments?post=506"}],"version-history":[{"count":1,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/506\/revisions"}],"predecessor-version":[{"id":4045,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/506\/revisions\/4045"}],"wp:attachment":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/media?parent=506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/categories?post=506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/tags?post=506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}