FastThread vs. Spotify thread dump analyzer

I wrote about last week.

Today, I compared with Spotify’s engineer’s tool.

Group threads with the same stacks and show the most common stack traces and their names first.

GitHub – spotify/threaddump-analyzer: A JVM threaddump analyzer

Both tools can group thread with same stack traces and display it ordered by # of threads.     But I found there’s difference.   I took 2 screenshots , 1 from each . To the left is, right is Spotify.


Blue part shows “31 threads with common stack trace” and green one showing    “21 threads with ….”.     But in Spotify “26 threads with … “ is missing.    What is this?

The answer is below. 26 threads were “

Gang worker#N (Parallel GC Threads)”.


q-text-as-data to create histogram from java heap dump

When java heap is used by many java.lang.String or “Class [C”  , it is often difficult to get a high level picture. Below is a screen capture of firefox browser accessing ‘jhat’ .   I thought I would create histogram for numbers in red rectanble. i.e.) (NNNNNN bytes).

From quick browsing of this page, I knew that largest number is 2.3MB. So I decided to bin(bucket) every 10000 bytes and show sum().


There may be many ways to this. I used #qtextasdata.

$ curl --silent http://localhost:8001/class/0x703af77a8 |
  sed -rn 's/^<a .* \(([0-9]+) bytes\).*$/\1/p'  |
  sort -n|uniq -c |
  q 'select sum(c1*c2),c2/10000 from - group by c2/10000'
162529307 0
388236 1
158056 2
269652 3
[... continue ...]

Above doesn’t fill the vacant bin with ‘0’ but I think this is good enough to know the distribution and find memory leak suspects. and Oracle JDBC

I had an opportunity to try for Java thread dump analysis. It is a great tool and I highly recommend it.

Their page shows Oracle sample.

at oracle/ons/
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

Fig: Oracle Database connection stuck in SocketInputStream.socketRead0() API

Threads stuck in – Fast thread

But it says “ons” which stands for:

Oracle Notification System (ONS)

Oracle RAC Fast Application Notification

and is not a core part of JDBC driver.  I think better sample would look like:

java.lang.Thread.State: RUNNABLE
at Method)
at Source)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(
at oracle.jdbc.driver.T4CTTIfun.receive(
at oracle.jdbc.driver.T4CTTIfun.doRPC(
at oracle.jdbc.driver.T4C8Oall.doOALL(
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(

Above is taken from Cloudera Manager