Monday, March 17, 2014

Mind Reading a JVM on Linux


I'm currently looking at what appears to be a memory leak in a Java process that uses native Tibco libraries.

To help me see what's going on, I discovered I could read the process's memory in Linux. I could do this on a machine running the 3.6.10 kernel but had trouble on the 2.6.16 kernel.

It appears that the memory of a process is available at:

/proc/PID/mem

So, running this program:

package com.phenry.memory;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;

public class AnalyserMain {

    public static void main(String[] args) {
        try {
            read(args);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private static void read(String[] args)
        throws FileNotFoundException,
        IOException {
        String              filename            = "/proc/" + args[0] + "/mem";
        RandomAccessFile    randomAccessFile    = new RandomAccessFile(new File(filename), "r");
        byte[]              b                   = new byte[1024 * 64];
        long                offset              = Long.parseLong(args[1], 16);
        randomAccessFile.seek(offset);
        int                 off                 = 0;
        
        randomAccessFile.read(b, off, b.length);
        System.out.println("output = " + new String(b));
    }

}

with the PID of my Java process (2801 in this case) as an argument and an address taken from pmap, so:

[henryp@corsair MyMemAnalyser]$ pmap 2801 | grep anon | head
000000000095a000    132K rw---    [ anon ]
00000006f0000000 120832K rw---    [ anon ]
00000006f7600000  28160K -----    [ anon ]
.
.

I could see this:

[henryp@corsair MyMemAnalyser]$ java -classpath bin com.phenry.memory.AnalyserMain 2801 000000000095a000 | strings 
output = 
/usr/java/jdk1.7.0_51/bin/java
org.eclipse.equinox.launcher.Main
/usr/java/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
/usr/java/jdk1.7.0_51/jre/lib/amd64/server
libjvm.so
/lib64/libm.so.6
/lib64
libm.so.6
/usr/java/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so
libm.so.6
u"V9
t"V9
DCmdFactory
SharedDecoderLock
-Djava.class.path=.
.
.

I've not found the cause of the memory leak yet but this takes me closer.

UPDATE 1: On some systems, you need to execute this as root:

echo 0 > /proc/sys/kernel/yama/ptrace_scope

as they have "hardened" kernels for security reasons (for instance, Ubuntu). Note: you are making your system more vulnerable if you have to do this.

UPDATE 2: The memory leak appeared to be coming from native Tibco libraries. Updating from 8.1 to 8.4 solved the problem.

No comments:

Post a Comment