FastThread vs. Spotify thread dump analyzer

I wrote about FastThread.io last week.

Today, I compared with Spotify’s engineer’s tool.

Group threads with the same stacks and show the most common stack traces and their names first.

GitHub – spotify/threaddump-analyzer: A JVM threaddump analyzer

Both tools can group thread with same stack traces and display it ordered by # of threads.     But I found there’s difference.   I took 2 screenshots , 1 from each . To the left is FastThread.io, right is Spotify.

image

Blue part shows “31 threads with common stack trace” and green one showing    “21 threads with ….”.     But in Spotify “26 threads with … “ is missing.    What is this?

The answer is below. 26 threads were “

Gang worker#N (Parallel GC Threads)”.

image

q-text-as-data to create histogram from java heap dump

When java heap is used by many java.lang.String or “Class [C”  , it is often difficult to get a high level picture. Below is a screen capture of firefox browser accessing ‘jhat’ .   I thought I would create histogram for numbers in red rectanble. i.e.) (NNNNNN bytes).

From quick browsing of this page, I knew that largest number is 2.3MB. So I decided to bin(bucket) every 10000 bytes and show sum().

image

There may be many ways to this. I used #qtextasdata.

$ curl --silent http://localhost:8001/class/0x703af77a8 |
  sed -rn 's/^<a .* \(([0-9]+) bytes\).*$/\1/p'  |
  sort -n|uniq -c |
  q 'select sum(c1*c2),c2/10000 from - group by c2/10000'
162529307 0
388236 1
158056 2
269652 3
[... continue ...]

Above doesn’t fill the vacant bin with ‘0’ but I think this is good enough to know the distribution and find memory leak suspects.

FastThread.io and Oracle JDBC

I had an opportunity to try FastThread.io for Java thread dump analysis. It is a great tool and I highly recommend it.

Their page shows Oracle sample.

at oracle/ons/ReceiverThread.run(ReceiverThread.java:295)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

Fig: Oracle Database connection stuck in SocketInputStream.socketRead0() API

Threads stuck in java.net.SocketInputStream.socketRead0 – Fast thread

But it says “ons” which stands for:

Oracle Notification System (ONS)

Oracle RAC Fast Application Notification

and is not a core part of JDBC driver.  I think better sample would look like:

java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:426)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:390)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:249)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:566)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:215)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:58)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:929)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1034)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3867)

Above is taken from Cloudera Manager