Simultaneous patching not possible with datapatch

Unfortunately…

Since the datapatch utility has been introduced it is not longer possible to patch more databases at the same time. Consequently, if you try to patch two databases simultaneously one of them will fail.

The process

The (oversimplified) flow chart below depicts the process:

The key point for explaining the root cause is that at some point the opatch lsinventory is invoked to produce the xml output file which, in turn, gets loaded into the database.

What’s the problem?

The xml file produced by opatch in the $ORACLE_HOME/QOPatch directory file is awkwardly named xml_file.xml. Since the file name doesn’t contain the database name, it will get overriden by a concurrent process running against another database. As a consequence, the error “ORA-31011: XML parsing failed” will be thrown.

Remediation

Oracle has recognized and addressed this problem by releasing the patches for the following bugs:

• 19215058: HPI_12102_RAC:ORA-12012 FOUND IN DB ALERT LOG WHEN RESTART ALL NODES’ CRS STACK
• 20244108: QOPIPREP.BAT MODIFIES XML INVENTORY WHILE READING IT

Notes: The both patches update the same script – $ORACLE_HOME/QOPatch/qopiprep.bat. The patch for the bug 20244108 superseeds the patch for the bug 19215058.

After applying the patch the file xml file contains the database name – xml_file_${ORACLE_SID}.xml – which resolves the problem.

Any more problems?

Sadly, the problem is not solved completely, because if the datapatch is invoked more than once in the same second it will invoke opatch which, in turn, will try to create the log file with the timestamp in its name. One of the opatch calls will fail with the exit code 73, because it will not be able to create the logfile which is created by the first process. Beware, the failed opatch will cause datapatch to fail silently, because the erroneous opatch execution is not being caught by datapatch:

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...

The opatch error was not obvious, I found it in the log files.

However, the problem can be demonstrated by two concurrent opatch lsinventory calls. The one of them will fail with the following error message:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
LsInventorySession failed: LsInventory cannot create the log directory
/u00/oracle/orabase/product/12.1.0.2.3_a/cfgtoollogs/opatch/lsinv/lsinventory2016-03-03_11-58-00AM.txt
OPatch failed with error code 73

The solution is actually very simple – it is enough to embedd the process ID in the opatch lsinvetory log file name, so that the names of the log files would be unique even for the processes started in the same second.

I wish…

It would be nice to fix the concurrency problem with opatch and then include the both patches in the future PSUs, as this would reduce the total patching downtime in highly consolidated environments.

Workaround

Until this problem gets fixed I’m planning to implement the following workaround in my patching robot: serialize the calls to datapatch, delay them for a second and hope I won’t hit any other concurrency issues with opatch. Though, if I do, I’ll update this blog post with new information.

Update 08.03.2016: Unfortuntately the described workaround didn’t work out. The datapatch apply invoked the opatch utility 4 times leaving 4 log files with the same disturbing pattern:

Mar  9 08:28 lsinventory2016-03-09_08-28-43AM.txt
Mar  9 08:28 lsinventory2016-03-09_08-28-46AM.txt
Mar  9 08:28 lsinventory2016-03-09_08-28-49AM.txt
Mar  9 08:28 lsinventory2016-03-09_08-28-51AM.txt

Four is the hard coded parallel degree in sqlpatch.pm:

my $catcon_num_processes = 4;             # number of processes to use

Update 19.03: Oracle Support suggested the workaround that I successfully tested: datapatch -apply <patch_id> -force -bundle_series DBBP . opatch lsinventory is not executed when datapatch is invoked with the force option.