Oracle database JVM imposes some undocumented hard-coded limits leading to java.lang.OutOfMemoryError<\/i>. In this blog post, I’m going to provide general guidelines on how to analyze such exceptions with OS tools when a higher-level information is missing. I used DTrace, but you can easily obtain the same information with, for example, gdb or any tool for tracing the application function calls.<\/p>\n
One way to provoke java.lang.OutOfMemoryError<\/i> is by allocating a large array (let me know if you have other test cases, please). But let’s first try to reproduce it within a conventional, i.e., non-database JVM.<\/p>\nConventional JVM<\/h1>\n
The following class allocates an array of integers, whose size is specified as the command line argument:<\/p>\n
public class Demo {\n public static void defineIntArray( int p_size) throws Exception {\n int[] v = new int[p_size];\n }\n public static void main( String args[] ) throws Exception {\n System.out.println(args[0]);\n defineIntArray(Integer.parseInt(args[0]));\n }\n}<\/code><\/pre>\nThe execution will break for 500 million elements, because the default maximum Java heap size is simply too low:<\/p>\n
java Demo 500000000 \n500000000\nException in thread \"main\" java.lang.OutOfMemoryError: Java heap space\n at Demo.defineIntArray(Demo.java:3)\n at Demo.main(Demo.java:7)<\/code><\/pre>\nGiven that each integer takes four bytes, we can easily estimate the total required memory. After specifying a sufficiently large maximum Java heap size, the program indeed completes without errors:<\/p>\n
java -Xmx2g Demo 500000000 \n500000000<\/code><\/pre>\nThis experiment confirmed that in a conventional JVM java.lang.OutOfMemoryError<\/i> really means what it says, that is, not enough memory for completing the operation. Consequently, the problem can be remedied by simply assigning more memory to JVM.<\/p>\nDatabase JVM<\/h1>\n
Now, let’s create the same Java code in an Oracle database (19c, in this case):<\/p>\n
create or replace and compile java source named \"Demo\" as\npublic class Demo {\t\n public static void defineIntArray( int p_size) throws Exception {\n int[] v = new int[p_size];\n }\n};\n\/\n\nCREATE OR REPLACE procedure p_define_int_array (p_size number) \n as LANGUAGE JAVA NAME 'Demo.defineIntArray(int)' ;\n\/<\/code><\/pre>\nThe maximum array size – irrespective of the available memory – is 134217723 (0x\u202d7FFFFFB\u202c). Adding just one additional element will cause OutOfMemoryError<\/i>:<\/p>\nexec p_define_int_array(134217724);\nBEGIN p_define_int_array(134217724); END;\n\n*\nERROR at line 1:\nORA-29532: Java call terminated by uncaught Java exception:\njava.lang.OutOfMemoryError\nORA-06512: at \"SYS.P_DEFINE_INT_ARRAY\", line 1\nORA-06512: at line 1<\/code><\/pre>\nUnlike in a conventional JVM, this problem can’t be solved by adding more memory. For example, PGA limit wasn’t set at all, and there was plenty of memory on both server and in the java pool. Also, the absence of common memory shortage errors (ORA-04030, ORA-04031 and ORA-04036) suggests that the lack of memory isn’t a problem. In other words, this limit is hard-coded and, therefore, cannot be avoided by resizing the memory parameters. That’s a serious disadvantage. On top of that, the error message doesn’t specify which (undocumented) limit was reached. The only clue is the bare error message java.lang.OutOfMemoryError<\/i>. But, luckily, that error message is also written into the server process trace file. Why is that beneficial? Because we can capture the call stack which writes the error message, and after doing so, get an idea which Oracle C functions participate in the error processing:<\/p>\ndtrace -p 12473 -x ustackframes=100 -s call_stack_trace.d<\/a> '\"java.lang.OutOfMemoryError\"'\n\nException in thread \"Root Thread\" java.lang.OutOfMemoryError\n\nlibc.so.1`__write+0xa\na.out`sdbgrfuwf_write_file+0x3d\na.out`sdbgrfwf_write_file+0x3c\na.out`dbgtfdFileWrite+0x20d\na.out`dbgtfdFileAccessCbk+0x19f\na.out`dbgtfPutStr+0x2d7\na.out`dbktPri+0x95\na.out`ksdwrf+0x1bc\na.out`ksdout+0xa9\na.out`jox_out_to_trc+0xce\na.out`jox_out+0xa5e\na.out`ioc_ewrite+0x25\na.out`iocbf_ewrite+0x1c\na.out`ioc_do_call+0x13a\na.out`joet_switched_env_callback+0x276\na.out`ioct_ewrite+0x2a\na.out`jonfos_write_bytes+0x29a\na.out`jonfos8_write_bytes__cst__+0x2e2\na.out`jonfos8_write_bytes+0x6b\n0x7fffbc7540df\n0x7fffbc753f80\n0x7fffbc753cf8\n0x7fffbc70d2ec\n0x7fffbc7132ea\n0x7fffbc7128c4\n0x7fffbc58a3d5\n0x7fffbc3dced7<\/span>\na.out`sjoninvk_jit+0x411\na.out`joevm_invoke_jit_0ret+0x18a\na.out`joevm_invokenonvirtual_setup_and_go+0x577\na.out`joevm_invokenonvirtual+0x2ec\na.out`joe_run_vm+0x2948\na.out`joe_run+0x81e\na.out`joez_jit_trampoline+0x1e1b\na.out`joe_jit_void_trampoline+0x17\na.out`sjoe_jit_void_callback+0x1a7\n0x7fffbc71f50c<\/span>\na.out`sjoninvk_jit+0x411\na.out`joevm_invoke_jit_0ret+0x18a\na.out`joevm_invokevirtual_setup_and_go+0x600\na.out`joevm_invokevirtualq_first+0xa5\na.out`joevm_invokevirtual+0x827\na.out`joe_run_vm+0x4d04\na.out`joe_run+0x81e\na.out`joet_aux_thread_main+0x1254\na.out`seoa_note_stack_outside+0x1d\na.out`joet_thread_main+0x43\na.out`sjontlo_initialize+0x94\na.out`joe_enter_vm+0x7ef\na.out`ioei_call_java+0x38ca\na.out`ioesub_CALL_JAVA+0x2e5\na.out`seoa_note_stack_outside+0x1d\na.out`ioe_call_java+0x15d\na.out`jox_invoke_java_+0x3e2\na.out`kkxmjexe+0x63d\na.out`kgmexcb+0x3a\na.out`kkxmswu+0x47\na.out`kgmexwi+0x56c\na.out`kgmexec+0x54f\na.out`pefjavacal+0x5dd\na.out`pefcal+0x37d\na.out`pevm_FCAL+0xe4\na.out`pfrinstr_FCAL+0xb9\noracle`pfrrun_no_tool+0x3c\noracle`pfrrun+0x53a\noracle`plsql_run+0x2c0\n...<\/code><\/pre>\nJust-in-time compiler<\/h1>\n
As you can see, the stack above also contains some private memory addresses. These memory locations contain precompiled java code stored in \/tmp<\/i> and mapped by the process [1]<\/a>. This code was produced by the just-in-time (JIT) compiler, which is the default way to execute Java code in the database [2]<\/a>:<\/p>\nshow parameter jit\n\nNAME TYPE VALUE\n------------------------------------ ----------- ------------------------------\njava_jit_enabled boolean TRUE<\/code><\/pre>\nThe precompiled code performs better, but on the downside, it’s extremely difficult to analyze the problem without having function names on the stack.<\/p>\n
Interpreted Java<\/h1>\n
Therefore, I temporarily switched off the JIT compiler, reproduced the error once again, and, finally, captured the stack. The following stack with interpreted Java code is far more descriptive:<\/p>\n
alter session set java_jit_enabled=false ;\n\ndtrace -p 12473 -x ustackframes=100 -s call_stack_trace.d<\/a> '\"java.lang.OutOfMemoryError\"'\n\nException in thread \"Root Thread\" java.lang.OutOfMemoryError\n\nlibc.so.1`__write+0xa\na.out`sdbgrfuwf_write_file+0x3d\na.out`sdbgrfwf_write_file+0x3c\na.out`dbgtfdFileWrite+0x20d\na.out`dbgtfdFileAccessCbk+0x19f\na.out`dbgtfPutStr+0x2d7\na.out`dbktPri+0x95\na.out`ksdwrf+0x1bc\na.out`ksdout+0xa9\na.out`jox_out_to_trc+0xce\na.out`jox_out+0xa5e\na.out`ioc_ewrite+0x25\na.out`iocbf_ewrite+0x1c\na.out`ioc_do_call+0x13a\na.out`joe<\/span>t_switched_env_callback+0x276\na.out`ioct_ewrite+0x2a\na.out`jonfos_write_bytes+0x29a\na.out`jonfos9_write_bytes+0x21\na.out`joe<\/span>vm_invoke_sysnative+0x20f\na.out`joe<\/span>vm_invokenonvirtual_setup_and_go+0x577\na.out`joe<\/span>vm_invokenonvirtual+0x2ec\na.out`joe<\/span>_run_vm+0x2948\na.out`joe<\/span>_run+0x81e\na.out`joe<\/span>t_aux_thread_main+0x1254\na.out`seoa_note_stack_outside+0x1d\na.out`joe<\/span>t_thread_main+0x43\na.out`sjontlo_initialize+0x94\na.out`joe<\/span>_enter_vm+0x7ef\na.out`ioei_call_java+0x38ca\na.out`ioesub_CALL_JAVA+0x2e5\na.out`seoa_note_stack_outside+0x1d\na.out`ioe_call_java+0x15d\na.out`jox_invoke_java_+0x3e2\na.out`kkxmjexe+0x63d\na.out`kgmexcb+0x3a\na.out`kkxmswu+0x47\na.out`kgmexwi+0x56c\na.out`kgmexec+0x54f\na.out`pefjavacal+0x5dd\na.out`pefcal+0x37d\na.out`pevm_FCAL+0xe4\na.out`pfrinstr_FCAL+0xb9\noracle`pfrrun_no_tool+0x3c\noracle`pfrrun+0x53a\noracle`plsql_run+0x2c0\n...<\/code><\/pre>\nIt’s worth noting, that many functions begin with joe<\/span> (which possibly stands for [J]ava [O]racle runtime [e]nvironment or something similar). Therefore, tracing all joe*<\/span> functions during the good and bad execution should provide further clues:<\/p>\n#!\/usr\/sbin\/dtrace -s\n\n#pragma D option flowindent\n#pragma D option dynvarsize=8m\n\npid$target:oracle:joe*<\/span>:entry{}\npid$target:oracle:joe*<\/span>:return{}<\/code><\/pre>\nGood execution:<\/p>\n
-> joe_invoke\n -> joe_run\n -> joe_run_vm\n -> joevm_invokerecursive\n -> joevm_first_time_setup\n <- joevm_first_time_setup\n -> joevm_invoke_interp_1loc\n <- joevm_invoke_interp_1loc\n <- joevm_invokerecursive\n -> joe_make_primitive_array\n -> joet_switched_env_callback\n <- joet_switched_env_callback\n -> joet_switched_env_callback\n <- joet_switched_env_callback\n -> joet_switched_env_callback\n <- joet_switched_env_callback\n<- joe_make_primitive_array\n-> joe_return<\/code><\/pre>\nBad execution:<\/p>\n
-> joe_invoke\n -> joe_run\n -> joe_run_vm \n -> joevm_invokerecursive\n -> joevm_first_time_setup\n <- joevm_first_time_setup\n -> joevm_invoke_interp_1loc\n <- joevm_invoke_interp_1loc\n <- joevm_invokerecursive\n -> joe_make_primitive_array\n -> joe_really_blow_it_internal<\/span>\n -> joe_well_known\n <- joe_well_known\n -> joe_blow_status_internal\n -> joet_exf_stack_top\n <- joet_exf_stack_top\n <- joe_blow_status_internal<\/code><\/pre>\nThe key difference between two executions is the call to joe_really_blow_it_internal<\/span>, which sounds like a bad comedy name, but is actually the Oracle C function for processing Java unhandled exceptions.<\/p>\nException handling<\/h1>\n
Now, we can capture the full stack in joe_really_blow_it_internal<\/span> execution:<\/p>\ndtrace -n 'pid$target:oracle:joe_really_blow_it_internal<\/span>:entry { ustack() } -p 12473\n\noracle`joe_really_blow_it_internal<\/span>\noracle`joe_make_primitive_array<\/span>+0x204\noracle`joe_run_vm+0x397d\noracle`joe_run+0x81e\noracle`joe_invoke+0x7e3\noracle`joet_aux_thread_main+0x17b9\na.out`seoa_note_stack_outside+0x1d\noracle`joet_thread_main+0x43\noracle`sjontlo_initialize+0x94\noracle`joe_enter_vm+0x7ef\noracle`ioei_call_java+0x38ca\na.out`ioesub_CALL_JAVA+0x2e5\na.out`seoa_note_stack_outside+0x1d\na.out`ioe_call_java+0x15d\na.out`jox_invoke_java_+0x3e2\na.out`kkxmjexe+0x63d\na.out`kgmexcb+0x3a\na.out`kkxmswu+0x47\na.out`kgmexwi+0x56c\na.out`kgmexec+0x54f<\/code><\/pre>\nIn this case, we can see that the exception was raised by joe_make_primitive_array<\/span>, which we can map to the array definition in our Java code.<\/p>\nPrimitive arrays<\/h1>\n
From here on, the analysis is case-specific. In our example, we need to focus on joe_make_primitive_array<\/span> and try to identify the application Java code and conditions that lead to the error.<\/p>\nThe array size is passed by the rdx CPU register (that is, the 3rd argument) and, thus, can be traced with the following DTrace script:<\/p>\n
#pragma D option quiet\npid$target:oracle:joe_make_primitive_array<\/span>:entry {\n printf(\"%s array size: %d\\n\", probefunc, arg2);\n}<\/code><\/pre>\ndtrace -s make_primitive_array.d -p 12473\njoe_make_primitive_array<\/span> array size: 134217724<\/code><\/pre>\nBy trying out to allocate the arrays of different sizes we can deduce the limit. Then, we can record the allocation sizes with the script above to identify the particular call that caused java.lang.OutOfMemoryError<\/i>.<\/p>\nGeneral guidelines<\/h1>\n
Finally, we can derive the general process for analyzing opaque JVM exceptions in the database:
\n1. switch off JIT compiler to obtain descriptive call stack
\n2. collect the stack for joe_really_blow_it_internal<\/i>
\n3. analyse the joe_really_blow_it_internal<\/i> caller<\/p>\nSummary<\/h1>\n
Unlike in conventional JVMs, java.lang.OutOfMemoryError<\/i>, surprisingly, doesn’t mean a memory shortage in the database JVM. The error message rather indicates that some hard-coded limitation is exceeded. Unfortunately, there aren’t any further clues about what limit exactly was reached, which makes the analysis difficult. A possibile way to go about it is to capture the call stack when the exception is thrown. The caller has to be analyzed further until we can map it to the application Java code.<\/p>\nUpdate December 6, 2019 – data types<\/h1>\n
The maximum number of elements in array depends on the data type. However, what’s always the same is the maximum total size in bytes: 536870895. In other words, the larger the data type, the less elements the array can store. By the way, this limit is the same accross different releases (tested with 12.1, 18.5 and 19.5).<\/p>\n
While that’s usually sufficient, it might be the problem when handling large BLOBs (in fact, that was exactly the use case which prompted this research). A 3rd party application uses database Java to spool a BLOB in the file, and, before doing so, allocates the buffer of the size of the BLOB:<\/p>\n
byte[] buffer = new byte[bufferSize];<\/code><\/pre>\nTherefore, OutOfMemoryError<\/i> will be thrown if the BLOB is larger than the hard-coded limit.<\/p>\nReferences<\/h1>\n
[1]<\/a> Tanel Poder, What the heck are the \/dev\/shm\/JOXSHM_EXT_x files on Linux?<\/i> May 9, 2014
\n[2]<\/a> Oracle Corp., Database Reference<\/i> February, 2019<\/p>\n","protected":false},"excerpt":{"rendered":"Internal processing of critical Java errors in the Oracle database Continue Reading