What the heck are the /dev/shm/JOXSHM_EXT_x files on Linux?

There was an interesting question in Oracle-L about the JOXSHM_EXT_* files in /dev/shm directory on Linux. Basically something like this:

$ ls -l /dev/shm/* | head
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 /dev/shm/JOXSHM_EXT_0_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 /dev/shm/JOXSHM_EXT_100_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 /dev/shm/JOXSHM_EXT_101_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:23 /dev/shm/JOXSHM_EXT_102_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:23 /dev/shm/JOXSHM_EXT_103_LIN112_1409029
-rwxrwx--- 1 oracle dba 36864 Apr 18 10:23 /dev/shm/JOXSHM_EXT_104_LIN112_1409029
...

There are a few interesting MOS articles about these files and how/when to get rid of those (don’t remove any files before reading the notes!), but none of these articles explain why these JOXSHM (and PESHM) files are needed at all:

  • /dev/shm Filled Up With Files In Format JOXSHM_EXT_xxx_SID_xxx (Doc ID 752899.1)
  • Stale Native Code Files Are Being Cached with File Names Such as: JOXSHM_EXT*, PESHM_EXT*, PESLD* or SHMDJOXSHM_EXT* (Doc ID 1120143.1)
  • Ora-7445 [Ioc_pin_shared_executable_object()] (Doc ID 1316906.1)

Here’s an explanation, a bit more elaborated version of what I already posted in Oracle-L:

The JOX files are related to Oracle’s in-database JVM JIT compilation. So, instead of interpreting the JVM bytecode during runtime, Oracle compiles it to archictecture-specific native binary code – just like compiling C code with something like gcc would do. So the CPUs can execute that binary code directly without any interpretation layers in between.

Now the question is that how do we load that binary code into our own process address space – so that the CPUs could execute this stuff directly?

This is why the JOX files exist. When the JIT compilation is enabled (it’s on by default on Oracle 11g), then the java code you access in the database will be compiled to machine code and saved in to the JOX files. Each Java class or method gets its own file (I haven’t checked which is it exactly). And then your Oracle process maps those files into its address space with a mmap() system call. So, any time this compiled java code has to be executed, your Oracle process can just jump to the compiled method address (and return back later).

Let’s do a little test:

SQL> SHOW PARAMETER jit

PARAMETER_NAME                                               TYPE        VALUE
------------------------------------------------------------ ----------- -----
java_jit_enabled                                             boolean     TRUE

Java just-in-time compilation is enabled. Where the JOX files are put by default is OS-specific, but on Linux they will go to /dev/shm (the in-memory filesystem) unless you specify some other directory with the _ncomp_shared_objects_dir parameter (and you’re not hitting one of the related bugs).

So let’s run some Java code in the Database:

SQL> SELECT DBMS_JAVA.GETVERSION FROM dual;

GETVERSION
--------------------------------------------
11.2.0.4.0

After this execution a bunch of JOX files showed up in /dev/shm:

$ ls -l /dev/shm | head
total 77860
-rwxrwx--- 1 oracle dba         4096 May  9 22:13 JOXSHM_EXT_0_LIN112_229381
-rwxrwx--- 1 oracle dba        12288 May  9 22:13 JOXSHM_EXT_10_LIN112_229381
-rwxrwx--- 1 oracle dba         4096 May  9 22:13 JOXSHM_EXT_11_LIN112_229381
-rwxrwx--- 1 oracle dba         8192 May  9 22:13 JOXSHM_EXT_12_LIN112_229381
-rwxrwx--- 1 oracle dba         4096 May  9 22:13 JOXSHM_EXT_13_LIN112_229381
-rwxrwx--- 1 oracle dba         4096 May  9 22:13 JOXSHM_EXT_14_LIN112_229381
...

When I check that process’es address space with pmap, I see that some of these JOX files are also mapped into its address space:

oracle@oel6:~$ sudo pmap -x 33390
33390:   oracleLIN112 (LOCAL=NO)
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000008048000  157584   21268       0 r-x--  oracle
0000000011a2c000    1236     372      56 rw---  oracle
0000000011b61000     256     164     164 rw---    [ anon ]
0000000013463000     400     276     276 rw---    [ anon ]
0000000020000000    8192       0       0 rw-s-  SYSV00000000 (deleted)
0000000020800000  413696       0       0 rw-s-  SYSV00000000 (deleted)
0000000039c00000    2048       0       0 rw-s-  SYSV16117e54 (deleted)
00000000420db000     120     104       0 r-x--  ld-2.12.so
00000000420f9000       4       4       4 r----  ld-2.12.so
00000000420fa000       4       4       4 rw---  ld-2.12.so
00000000420fd000    1604     568       0 r-x--  libc-2.12.so
000000004228e000       8       8       8 r----  libc-2.12.so
0000000042290000       4       4       4 rw---  libc-2.12.so
0000000042291000      12      12      12 rw---    [ anon ]
0000000042296000      92      52       0 r-x--  libpthread-2.12.so
00000000422ad000       4       4       4 r----  libpthread-2.12.so
00000000422ae000       4       4       4 rw---  libpthread-2.12.so
00000000422af000       8       4       4 rw---    [ anon ]
00000000422b3000      12       8       0 r-x--  libdl-2.12.so
00000000422b6000       4       4       4 r----  libdl-2.12.so
00000000422b7000       4       4       4 rw---  libdl-2.12.so
00000000f63b9000       4       4       4 rwxs-  JOXSHM_EXT_88_LIN112_229381
00000000f63ba000      16      16      16 rwxs-  JOXSHM_EXT_91_LIN112_229381
00000000f63be000       4       4       4 rwxs-  JOXSHM_EXT_90_LIN112_229381
00000000f63bf000       4       4       4 rwxs-  JOXSHM_EXT_89_LIN112_229381

...

Note the X and S bits (in the rwxs-) for the JOX mapped segments, this means that the Linux virtual memory manager allows the contents of these mapped files to be directly executed by the CPU and the S means its a shared mapping (other processes can map this binary code into their address spaces as well).

Oracle can also load some of its binary libraries into its address space with the dynamic dlopen() system call, but I verified using strace that the JOX files are “loaded” into the address space with just a mmap() syscall:

33390 open("/dev/shm/JOXSHM_EXT_85_LIN112_229381", O_RDWR|O_NOFOLLOW|O_CLOEXEC) = 8
33390 mmap2(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_SHARED, 8, 0) = 0xfffffffff63c2000
33390 close(8)                          = 0
33390 open("/dev/shm/JOXSHM_EXT_87_LIN112_229381", O_RDWR|O_NOFOLLOW|O_CLOEXEC) = 8
33390 mmap2(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_SHARED, 8, 0) = 0xfffffffff63c0000
33390 close(8)                          = 0
33390 open("/dev/shm/JOXSHM_EXT_89_LIN112_229381", O_RDWR|O_NOFOLLOW|O_CLOEXEC) = 8
33390 mmap2(NULL, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_SHARED, 8, 0) = 0xfffffffff63bf000
33390 close(8)                          = 0
...

Just to check if these JOX files really are compiled machine-code instructions:

$ file /dev/shm/JOXSHM_EXT_88_LIN112_229381
/dev/shm/JOXSHM_EXT_88_LIN112_229381: data
$

Oops, the file command doesn’t recognize any specific file format from its contents as it’s a bare slice of machinecode and doesn’t contain the normal object module stuff the .o files would have… let’s try to disassemble this file and see if contains sensible instructions:

$ objdump -b binary -m i386 -D /dev/shm/JOXSHM_EXT_88_LIN112_229381 | head -30

/dev/shm/JOXSHM_EXT_88_LIN112_229381:     file format binary


Disassembly of section .data:

00000000 :
       0:	f7 15 1e 33 00 00    	notl   0x331e
       6:	00 00                	add    %al,(%eax)
       8:	55                   	push   %ebp
       9:	89 e5                	mov    %esp,%ebp
       b:	83 c4 c8             	add    $0xffffffc8,%esp
       e:	8b 45 10             	mov    0x10(%ebp),%eax
      11:	89 c1                	mov    %eax,%ecx
      13:	83 e1 f8             	and    $0xfffffff8,%ecx
      16:	89 4d e0             	mov    %ecx,-0x20(%ebp)
      19:	8b 4d 08             	mov    0x8(%ebp),%ecx
      1c:	89 45 f0             	mov    %eax,-0x10(%ebp)
      1f:	8b 45 0c             	mov    0xc(%ebp),%eax
      22:	89 5d ec             	mov    %ebx,-0x14(%ebp)
      25:	89 7d e8             	mov    %edi,-0x18(%ebp)
      28:	89 75 e4             	mov    %esi,-0x1c(%ebp)
      2b:	c7 45 f8 02 00 00 40 	movl   $0x40000002,-0x8(%ebp)
      32:	8b 91 1d 02 00 00    	mov    0x21d(%ecx),%edx
      38:	8b 7d 14             	mov    0x14(%ebp),%edi
      3b:	89 45 f4             	mov    %eax,-0xc(%ebp)
      3e:	8d 45 f0             	lea    -0x10(%ebp),%eax
      41:	83 c2 04             	add    $0x4,%edx
      44:	89 91 1d 02 00 00    	mov    %edx,0x21d(%ecx)
      4a:	39 d0                	cmp    %edx,%eax
...

This file contains machine code indeed!

So this is how Oracle approaches native JIT compilation for the in-database JVM. In 11g it’s similar to the PL/SQL native compilation too (you’d see various PESHM_ files in /dev/shm). Before 11g, Oracle actually generated intermediate C code for your PL/SQL and then invoked an OS C compiler on it, but in 11g it’s all self-contained in the database code. Pretty cool!

 

This entry was posted in Linux, Oracle. Bookmark the permalink.

4 Responses to What the heck are the /dev/shm/JOXSHM_EXT_x files on Linux?

  1. That was me, scratching my head as to why I had shared memory segments in /dev/shm when I wasn’t using AMM…. that one had me stumped for a while.

    Thanks for the detailed explanation, Tanel!

    -Mark

  2. Tanel Poder says:

    Yeah, for quite a while I just ignored these files, knowing only that the JOX module means something Java-specific in Oracle … I started looking into this also when changing one of my dev databases to SGA_TARGET and still saw those files there (initially I thought that maybe these files are part of java pool somehow, but it turned out to be different – it’s mapped to private memory of processes).

  3. Each Java class or method gets its own file (I haven’t checked which is it exactly).
    any idea how to check that ? i spent some time on that trying to find out more but with no success.

    • Tanel Poder says:

      I’d probably do something like creating a dummy class with 100 dummy methods in it and load it and see how many new mappings show up (but any dependencies would be mapped as well, so you’ve got to account for that).

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>