{"id":7614,"date":"2018-06-21T20:40:59","date_gmt":"2018-06-21T12:40:59","guid":{"rendered":"http:\/\/rmohan.com\/?p=7614"},"modified":"2018-06-21T20:40:59","modified_gmt":"2018-06-21T12:40:59","slug":"jboss-too-many-files-open-error","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=7614","title":{"rendered":"JBoss Too Many Files Open Error"},"content":{"rendered":"<p>&nbsp;<\/p>\n<p>First of all you want to determine what file(s) remain open. I&#8217;m assuming your server runs linux, so once you know JBoss&#8217;es PID<\/p>\n<pre class=\"default prettyprint prettyprinted\"><code><span class=\"pln\">ps ax <\/span><span class=\"pun\">|<\/span><span class=\"pln\"> grep something<\/span><span class=\"pun\">-<\/span><span class=\"pln\">that<\/span><span class=\"pun\">-<\/span><span class=\"pln\">makes<\/span><span class=\"pun\">-<\/span><span class=\"pln\">your<\/span><span class=\"pun\">-<\/span><span class=\"pln\">jboss<\/span><span class=\"pun\">-<\/span><span class=\"pln\">process<\/span><span class=\"pun\">-<\/span><span class=\"pln\">unique<\/span><\/code><\/pre>\n<p>you can do<\/p>\n<pre class=\"default prettyprint prettyprinted\"><code><span class=\"pln\">ls <\/span><span class=\"pun\">-<\/span><span class=\"pln\">l <\/span><span class=\"pun\">\/<\/span><span class=\"pln\">proc<\/span><span class=\"pun\">\/<\/span><span class=\"pln\">jbosspid<\/span><span class=\"pun\">\/<\/span><span class=\"pln\">fd<\/span><\/code><\/pre>\n<p>to get a nice list of files that are open at that instant.<\/p>\n<p>What you&#8217;re going to do next depends a bit on what you see here:<\/p>\n<ol>\n<li>you may just need to up the number of files the server can open a bit with ulimit (also look at systemwide limits on your server)<\/li>\n<li>maybe you spot a number of files your application forgot to close<\/li>\n<li>&#8230;.<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<p>Also the max limit is high<\/p>\n<p>linux-server:~# cat \/proc\/sys\/fs\/file-max<br \/>\n202989<\/p>\n<p>and the max ever occupied is well below the limit:<br \/>\ncat \/proc\/sys\/fs\/file-nr<br \/>\n6304 0 202989<\/p>\n<p>all users return the same limit (including the jboss user who initiates the app server):<br \/>\njboss@linux-server:\/home$ ulimit -n<\/p>\n<p>A wat to have a look at this is to run the lsof command (only as root) &#8211; it will show you all the open file descriptors.<\/p>\n<p>In order to fix that, edit the file in \/etc\/security\/limits.conf and add the following lines and restart your jboss.<\/p>\n<pre class=\"default prettyprint prettyprinted\"><code><span class=\"pln\">jboss          soft    nofile          <\/span><span class=\"lit\">16384<\/span><span class=\"pln\">\r\njboss          hard    nofile          <\/span><span class=\"lit\">16384<\/span><\/code><\/pre>\n<p>(assuming your jboss is run by the &#8220;jboss&#8221; user)<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<section class=\"field_kcs_resolution_txt\">\n<ul>\n<li>Settings in\u00a0<code>\/etc\/security\/limits.conf<\/code>\u00a0take the following form:\n<div class=\"code-raw\">\n<div class=\"code-raw-toolbar\"><a class=\"code-raw-btn\" href=\"https:\/\/access.redhat.com\/solutions\/61334#\">Raw<\/a><\/div>\n<pre><code># vi \/etc\/security\/limits.conf\r\n#&lt;domain&gt;        &lt;type&gt;  &lt;item&gt;  &lt;value&gt;\r\n\r\n*               -       core             &lt;value&gt;\r\n*               -       data             &lt;value&gt;\r\n*               -       priority         &lt;value&gt;\r\n*               -       fsize            &lt;value&gt;\r\n*               soft    sigpending       &lt;value&gt; eg:57344\r\n*               hard    sigpending       &lt;value&gt; eg:57444\r\n*               -       memlock          &lt;value&gt;\r\n*               -       nofile           &lt;value&gt; eg:1024\r\n*               -       msgqueue         &lt;value&gt; eg:819200\r\n*               -       locks            &lt;value&gt;\r\n*               soft    core             &lt;value&gt;\r\n*               hard    nofile           &lt;value&gt;\r\n@&lt;group&gt;        hard    nproc            &lt;value&gt;\r\n&lt;user&gt;          soft    nproc            &lt;value&gt;\r\n%&lt;group&gt;        hard    nproc            &lt;value&gt;\r\n&lt;user&gt;          hard    nproc            &lt;value&gt;\r\n@&lt;group&gt;        -       maxlogins        &lt;value&gt;\r\n&lt;user&gt;          hard    cpu              &lt;value&gt;\r\n&lt;user&gt;          soft    cpu              &lt;value&gt;\r\n&lt;user&gt;          hard    locks            &lt;value&gt;\r\n<\/code><\/pre>\n<\/div>\n<ul>\n<li><code>&lt;domain&gt;<\/code>\u00a0can be:\n<ul>\n<li>an user name<\/li>\n<li>a group name, with @group syntax<\/li>\n<li>the wildcard *, for default entry<\/li>\n<li>the wildcard %, can be also used with %group syntax, for maxlogin limit<\/li>\n<\/ul>\n<\/li>\n<li><code>&lt;type&gt;<\/code>\u00a0can have the two values:\n<ul>\n<li>&#8220;soft&#8221; for enforcing the soft limits<\/li>\n<li>&#8220;hard&#8221; for enforcing hard limits<\/li>\n<\/ul>\n<\/li>\n<li><code>&lt;item&gt;<\/code>\u00a0can be one of the following:\n<ul>\n<li>core &#8211; limits the core file size (KB)<\/li>\n<li>data &#8211; max data size (KB)<\/li>\n<li>fsize &#8211; maximum filesize (KB)<\/li>\n<li>memlock &#8211; max locked-in-memory address space (KB)<\/li>\n<li>nofile &#8211; max number of open files<\/li>\n<li>rss &#8211; max resident set size (KB)<\/li>\n<li>stack &#8211; max stack size (KB)<\/li>\n<li>cpu &#8211; max CPU time (MIN)<\/li>\n<li>nproc &#8211; max number of processes<\/li>\n<li>as &#8211; address space limit (KB)<\/li>\n<li>maxlogins &#8211; max number of logins for this user<\/li>\n<li>maxsyslogins &#8211; max number of logins on the system<\/li>\n<li>priority &#8211; the priority to run user process with<\/li>\n<li>locks &#8211; max number of file locks the user can hold<\/li>\n<li>sigpending &#8211; max number of pending signals<\/li>\n<li>msgqueue &#8211; max memory used by POSIX message queues (bytes)<\/li>\n<li>nice &#8211; max nice priority allowed to raise to values: [-20, 19]<\/li>\n<li>rtprio &#8211; max realtime priority<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Exit and re-login from the terminal for the change to take effect.<\/li>\n<li>More details can be found from below command:<\/li>\n<\/ul>\n<div class=\"code-raw\">\n<div class=\"code-raw-toolbar\"><a class=\"code-raw-btn\" href=\"https:\/\/access.redhat.com\/solutions\/61334#\">Raw<\/a><\/div>\n<pre><code># man limits.conf\r\n<\/code><\/pre>\n<\/div>\n<\/section>\n<section class=\"field_kcs_diagnostic_txt\">\n<h2>Diagnostic Steps<\/h2>\n<ul>\n<li>To improve performance, we can safely set the limit of processes for the super-user root to be unlimited. Edit the .bashrc file vi\u00a0<code>\/root\/.bashrc<\/code>and add the following line:<\/li>\n<\/ul>\n<div class=\"code-raw\">\n<div class=\"code-raw-toolbar\"><a class=\"code-raw-btn\" href=\"https:\/\/access.redhat.com\/solutions\/61334#\">Raw<\/a><\/div>\n<pre><code># vi \/root\/.bashrc\r\nulimit -u unlimited<\/code><\/pre>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp;<\/p>\n<p>First of all you want to determine what file(s) remain open. I&#8217;m assuming your server runs linux, so once you know JBoss&#8217;es PID<\/p>\n<p> ps ax | grep something-that-makes-your-jboss-process-unique <\/p>\n<p>you can do<\/p>\n<p> ls -l \/proc\/jbosspid\/fd <\/p>\n<p>to get a nice list of files that are open at that instant.<\/p>\n<p>What you&#8217;re going to do next depends [&#8230;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7614"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7614"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7614\/revisions"}],"predecessor-version":[{"id":7615,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7614\/revisions\/7615"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7614"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7614"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7614"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}