{"id":3251,"date":"2014-06-24T01:54:18","date_gmt":"2014-06-23T17:54:18","guid":{"rendered":"http:\/\/rmohan.com\/?p=3251"},"modified":"2014-06-24T08:56:20","modified_gmt":"2014-06-24T00:56:20","slug":"maximum-number-of-file-descriptors","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=3251","title":{"rendered":"maximum-number-of-file-descriptors solaris"},"content":{"rendered":"

Before I get into details here is the bottom line. If you start MySQL on Solaris as a non-root (ie, mysql) user and for some reason you need to adjust the file descriptor resource limit for the parent shell, never use ‘ulimit -n’. This will set both the soft and hard limit and may cause MySQL to adjust the max_connections and table_open_cache configuration variables upon next startup.<\/p>\n

Use either:<\/p>\n

 ulimit -S -n 1024<\/pre>\n
or something like:<\/p>\n
\u00a0<\/span><\/span> prctl -n process.max-file-descriptor -t basic -v  1024 -r -i process $$\r\n\r\n\r\n\r\n<\/pre>\n
The Details<\/h2>\n
The default ‘basic’ privilege value for the resource control process.max-file-descriptor is 256. This control represents the soft ulimit for file descriptors per process. The default ‘privileged’ privilege is set to 65535, which represents the hard ulimit. A non-root user can adjust the soft limit down or up to the hard limit. Unless it has PRIV_SYS_RESOURCE a non-root user can only decrease the hard limit.<\/p>\n
Not being aware of the default values I started a sysbench run with 256 threads. It failed with the following message:<\/p>\n
FATAL: unable to connect to MySQL server, aborting...\r\nFATAL: error 2001: Can't create UNIX socket (24)\r\nFATAL: failed to connect to database server!\r\nFATAL: thread#252: failed to connect to database server, aborting...\r\n<\/pre>\n
Sysbench fails because the action for the basic privilege process.max-file-descriptors is deny. I was running sysbench logged in as mysql. I made the mistake of increasing the ulimit to what I thought was a reasonable 1024 via ‘ulimit -n 1024’. When restarting mysqld it inherited a soft and hard limit of 1024. This had the effect of causing mysqld to recalculate the values for max_connections and table_open_cache as displayed in the error log:<\/p>\n
090610  9:44:11 [Warning] Changed limits: max_open_files: 1024  max_connections: 886  table_cache: 64\r\n<\/pre>\n
This value is much too small for table_open_cache and is guaranteed to cause heavy contention on LOCK_open as concurrent threads increase.<\/p>\n
Why did MySQL change table_open_cache and max_connections?<\/h2>\n
Upon startup mysqld will calculate the number of open files it wants selecting the greatest of:<\/p>\n
    \n
  • myisam’s requirement for 2 file handles per connection\n
      \n
    • 10+max_connections+table_cache_size\\*2<\/li>\n<\/ul>\n<\/li>\n
    • max_connnections\\*5<\/li>\n
    • open_files_limit<\/li>\n<\/ul>\n
      For the purpose of this discussion let’s call this max_open_files.<\/p>\n
      It then compares max_open_files with the soft resource limit for file descriptors per process. If the soft limit is less than max_open_files both the soft and hard limit will be set to max_open_files via setrlimit. If setrlimit fails the requested max_open_files is modified to the old soft limit. If the returned max_open_files is less than myisam’s requirement for 2 file handles per connection and open_files_limit has not been specified, max_connections and table_open_cache will be recalculated to fit within the new boundaries.<\/p>\n
      Now take the case where I inadvertently set both the soft and hard limit via ‘ulimit -n 1024’. I had set\u00a0 open_table_cache=4096 and max_connections=2049, arbitrary large values so I could set them once and forget about them. In this case max_open_files=max_connections\\*5=10245. When trying to increase the limits to 10245, the setrlimit call fails because I started mysqld as the mysql user and it does not have the privilege to increase the hard limit from 1024 to 10245. In this case max_open_files is set to 1024, which causes mysqld to recalculate max_connections and table_open_cache:<\/p>\n
      090610  9:44:11 [Warning] Changed limits: max_open_files: 1024  max_connections: 886  table_cache: 64<\/pre>\n
      How to observe process.max-file-descriptor<\/h2>\n
      You can use either prctl or ulimit to observe the current values of process.max-file-descriptor:<\/p>\n
      $ prctl -n process.max-file-descriptor  -i process $$\r\nprocess: 20773: -ksh\r\nNAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT\r\nprocess.max-file-descriptor\r\n        basic             256       -   deny                             20773\r\n        privileged      65.5K       -   deny                                 -\r\n        system          2.15G     max   deny                                 -\r\n<\/pre>\n
      $ ulimit -S -n ### soft limit\r\n256\r\n$ ulimit -H -n ### hard limit\r\n65535\r\n\r\n<\/pre>\n
      On running processes:<\/p>\n

      \n# ps -ef | grep <username> | grep -v root \\
      \n| awk '{print \"prctl -n process.max-file-descriptor -v <value> -r -i process \" $2}' | sh -x
      \n<\/code><\/p>\n
      System wide via mdb(add to \/etc\/system to be presistent across reboots):<\/p>\n

      \n# mdb -kw
      \nrlim_fd_cur\/W <0tvalue>
      \nrlim_fd_max\/W <0tvalue>
      \n<\/code><\/p>\n
      This only works in the shell and not in .profile or .cshrc:<\/p>\n

      \n$ ulimit -Hn\u00a0<\/span>
      \n$ ulimit -Sn\u00a0<\/span>
      \n<\/code><\/p>\n
      The same with:<\/p>\n

      \n$ prctl -n process.max-file-descriptor -v <value> -r -i process $$<\/code><\/p>\n
      <\/pre>\n","protected":false},"excerpt":{"rendered":"
      Before I get into details here is the bottom line. If you start MySQL on Solaris as a non-root (ie, mysql) user and for some reason you need to adjust the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[39],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3251"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3251"}],"version-history":[{"count":4,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3251\/revisions"}],"predecessor-version":[{"id":3253,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3251\/revisions\/3253"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3251"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3251"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3251"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}