Tracing Library Cache Locks 3: Object Names

In the previous installment I provided a gdb script for tracing library cache locks. That script, however, had a serious disadvantage – you had to stop the execution after each lock allocation, get the lock handle address from the CPU registry and then use this value to query the object name from the in-memory C program structure. The reason for this cumbersome implementation was that I didn’t know how to retrieve the object name directly from the shared memory (all this stuff is undocumented internals).

In the meantime, I closed the gap and wrote a fully-fledged script for tracing library cache locks with debugger:

set pagination off
set logging file library_cache_locks.log
set logging on

break kgllkal
commands 1
backtrace 10
echo kgllkhdl
p/z $rdx
echo kglnaobj
x/s $rdx+0x1c0
echo kgllkmod
p/x $rcx
continue
end

Besides the object name, the script also prints the lock handle address and the requested mode. These values are stored in the CPU registers rdx and rcx, respectively. The object name itself is stored at the offset 0x1c0 from the library cache lock handle, at least for the version 19.9.0.0.201020. I’ll explain later how you can find out this offset for any Oracle database software.

Find the sample output below:

...
Thread 2 hit Breakpoint 1, 0x0000000006a3f3e0 in kgllkal ()
#0  0x0000000006a3f3e0 in kgllkal ()
#1  0x0000000006a2349e in kglLock ()
#2  0x0000000006b2691c in kglget ()
#3  0x000000000f874c30 in kkdllks ()
#4  0x000000000f8a1367 in kkdllppac0 ()
#5  0x000000000de8c7a4 in aindrv ()
#6  0x000000000618a7c0 in opiexe ()
#7  0x0000000006715fff in opiosq0 ()
#8  0x00000000068ac4b5 in kpooprx ()
#9  0x00000000067d49f4 in kpoal8 ()
kgllkhdl$95 = 0x000000009d3174e0
kglnaobj0x9d3176a0:     "IX_TSYS"
kgllkmod$96 = 0x2

Thread 2 hit Breakpoint 1, 0x0000000006a3f3e0 in kgllkal ()
#0  0x0000000006a3f3e0 in kgllkal ()
#1  0x0000000006a2349e in kglLock ()
#2  0x0000000006b2691c in kglget ()
#3  0x0000000006aa8046 in kkdllk0 ()
#4  0x000000000f8a19b8 in kkdllppac0 ()
#5  0x000000000de8c7a4 in aindrv ()
#6  0x000000000618a7c0 in opiexe ()
#7  0x0000000006715fff in opiosq0 ()
#8  0x00000000068ac4b5 in kpooprx ()
#9  0x00000000067d49f4 in kpoal8 ()
kgllkhdl$97 = 0x00000000edd6cbc0
kglnaobj0xedd6cd80:     "TSYS"
kgllkmod$98 = 0x3
...

As we can see, the schema owner – SYS in this case – is appended to the object name (IX_T and T).

But how can you generally find out the offset from the library cache lock handle which stores the object name?

In a nutshell, you can stop the execution immediately after a library cache lock allocation for a known object and then observe the address space near the library cache lock handle (this entails some risks – better do it in lab only).

For that purpose, it’s useful to use an object name, whose hexadecimal representation is easy to spot while eyeballing the address space, for example:

create table aaaaaaaaaaaaaaaa (n1 integer,n2 integer);
create index ix_a on aaaaaaaaaaaaaaaa(n1,n2);

There will be many library cache lock allocations, for example, also during parsing. To skip the irrelevant part, we’ll first stop the execution on dictionary lookup when altering the index:

b kkdllk0

alter index ix_a invisible ;

Only after hitting the breakpoint on kkdllk0, set the breakpoint on the function which actually allocates the lock:

b kgllkal
continue

After hitting the breakpoint on kgllkal, get out of the function:

fin

Then query the library cache lock handle address for the table:

select kgllkhdl from x$kgllk where kglnaobj = 'AAAAAAAAAAAAAAAA' ;

KGLLKHDL
----------------
000000009FC1A4B0

You might need to reach this breakpoint twice to get the address above, because the first library cache lock allocation is for the index.

Now inspect the address space around the library cache lock handle address:

x/200x 0x000000009FC1A4B0

...
0x9fc1a660:       0x00027e97      0x00000000      0x00000000      0x00000000
0x9fc1a670:       0x41414141      0x41414141      0x41414141      0x41414141
0x9fc1a680:       0x00535953      0x00000000      0x103373c0      0x00000001
...

The object name is stored on the address 0x9fc1a670 (0x41 is the ASCII code for “A”).

Finally, the offset is calculated by substracting the address which stores the object name from the handle address.

@calc.sql 0x9fc1a670 - 0x000000009FC1A4B0

                                DEC                  HEX
----------------------------------- --------------------
                         448.000000                  1C0

In summary, my gdb script traces the library cache lock allocations. Since the offset which stores the object name might change between releases, I provided a general procedure to find it out.