Dear Reader,<\/p>\n
In a production environment Java Profiling is not an option, we have seen multiple times that
\nour CPU has reached almost 100% or even 300% sometime. That is really a panic scenario especially
\nwhen you are handling production environment or at client place to check what went wrong.<\/p>\n
Fortunately, Java comes with some great debugging tools, co-operating those tools with Linux
\nbuilt-in tools will help you to know what is happening.<\/p>\n
I am going to explain below items here:
\n 1) An introduction about Java threads and its relation to Linux LWP (Light Weight Process).
\n 2) Step-by-step process to take thread dump and analyze CPU utilization.
\n 3) “jmap” – Memory Map (dump), Command to get this.
\n 4) “jmap” – Histo, Command to get this.
\n 5) Command to see list of open files in Unix.<\/p>\n
1) Introduction: Any java program starts when the JVM calls the main method, this creates a thread
\n called the main thread and any thread you create using java code will be derived from the main
\n thread. The same exact behavior occurs on the Linux level, the main thread for java means a Process
\n for the OS and every thread you create using java the OS will create a Light-weight-process
\n or LWP. In short: Java main thread = Linux process and Java supporting threads = Linux LWP.<\/p>\n
The idea here:
\n Ask Linux which LWP is eating the CPU.
\n Ask Java for a Thread Dump.
\n Map this LWP to a Java thread.
\n Get the part of code causing the issue. <\/p>\n
2) Step-by-step process:
\n Get the PID: the very first step is to know what is the Java process ID, we will use Linux commands
\n as below. Use either of the below commands (we use our grep “DAPPNAME”, you can use anything):
\n jps -v | cut -c -106 | grep DAPPNAME
\n jps -mvl | cut -c -106 | grep DAPPNAME
\n ps -eaf | cut -c -106 | grep DAPPNAME
\n ps -ef | cut -c -106 | grep DAPPNAME
\n ps -eaf | grep java<\/p>\n
Below are the sample output when you execute the command:
\n dmodi@MyDirectory:~$ jps -mlv | cut -c -106 | grep DAPPNAME
\n 8243 org.quickserver.net.server.QuickServer -load config\/DmodiServer.xml -DAPPNAME=CLIENT
\n 13712 org.quickserver.net.server.QuickServer -load .\/conf\/DmodiDNXServer.xml -DAPPNAME=SERVER
\n 12229 org.quickserver.net.server.QuickServer -load .\/config\/DmodiPOSServer.xml -DAPPNAME=SERVER2<\/p>\n
Explanation: “jps” – Java Virtual Machine Process Status Tool, a command in Unix. “106” shows 106
\n characters we want to display in console.<\/p>\n
The next step is to get the CPU usage per each LWP related to the main process, we can use below commands:
\n ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep PROCESS_ID | cut -c -106 > threadsList.txt<\/p>\n
The newly created file threadsList.txt will contain below things in similar way (The headers will not be
\n shown as below):
\n PID LWP NLWP RUSER %CPU STIME ELAPSED COMMAND
\n 8243 8243 3 dmodi 0.0 May13 1-19:20:18 java -Dprogram.name=run.sh -Xms64m -Xmx100m -Dsun.rmi.dgc
\n 8243 8244 3 dmodi 0.0 May13 1-19:20:18 java -Dprogram.name=run.sh -Xms64m -Xmx100m -Dsun.rmi.dgc
\n 8243 8245 3 dmodi 99.9 May13 1-19:20:18 java -Dprogram.name=run.sh -Xms64m -Xmx100m -Dsun.rmi.dgc<\/p>\n
To see headers too, just execute the below command:
\n ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args > threadsList.txt<\/p>\n
Explanation: PID is process Id.
\n LWP: is Light weight processes Lists for the above PID. These values are in Decimal.
\n NLWP: is number of LWP created for the above PID.
\n Rest other headers don’t require any explanation.
\n We can see LWP – 8245 is eating CPU. We need to convert this value into HEX value which will be “2035”.<\/p>\n
Now take the thread dump and kill the process Id: 8243. Open the thread dump file and search value “2035”.
\n See below command:
\n \/\/Taking thread dump
\n jstack -l 8243 > ThreadDump_15_May_2014_13_PM.txt<\/p>\n
\/\/Killing process
\n kill -9 8243<\/p>\n
3) “jmap” – Memory Map (dump): Prints shared object memory maps or heap memory details of a given JVM process.
\n dmodiUnixUser@productName79:~$ jmap -dump:file=deepak.bin 8243
\n Dumping heap to \/home\/dmodiUnixUser\/deepak.bin …
\n Heap dump file created
\n dmodiUnixUser@productName79:~$ ls
\n deepak.bin
\n This newly created file will be big in size (of 5-10 MB around). You can’t see this content using
\n “less” or “cat” command. You need tool to see this. We don’t use this generally, so not mentioning here.<\/p>\n
4) “jmap” – Histo: See below command:
\n dmodiUnixUser@productName79:~$ jmap -histo:live 8243 > deepak.txt
\n Contents of this file “deepak.txt” will have similar like above:<\/p>\n
num #instances #bytes class name 5) List of open files in Linux: dmodiUnixUser@productName79:~$ lsof | grep PROCESS_ID > help.txt Dear Reader,<\/p>\n In a production environment Java Profiling is not an option, we have seen multiple times that our CPU has reached almost 100% or even 300% sometime. That is really a panic scenario especially when you are handling production environment or at client place to check what went wrong.<\/p>\n Fortunately, Java comes with some […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18,4,21],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3441"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3441"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3441\/revisions"}],"predecessor-version":[{"id":3442,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3441\/revisions\/3442"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3441"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3441"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3441"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n ———————————————-
\n 1: 14452 2229096
\n 2: 14452 1740720
\n 3: 1004 1406296
\n 4: 1336 1270504 [B
\n 5: 25057 1060840
\n 6: 835 809368
\n 7: 1004 787096
\n lsof – list open files
\n dmodiUnixUser@productName79:~$ lsof | grep home\/dmodi\/productName\/dist\/
\n COMMAND PID USER FD TYPE DEVICE SIZE\/OFF NODE NAME
\n java 16460 dmodiUnixUser mem REG 9,2 25680 8127062 home\/dmodi\/productName\/dist\/sample1.jar
\n java 16460 dmodiUnixUser mem REG 9,2 66770 8127061 home\/dmodi\/productName\/dist\/sample2.jar<\/p>\n
\n dmodiUnixUser@productName79:~$ less help.txt
\n———————————-END—————————————–<\/p>\n","protected":false},"excerpt":{"rendered":"