mod_mpmstats
mod_mpmstats provides three capabilities:
- check if the web server is at or near the maximum concurrent clients setting (MaxClients for Unix and Linux; ThreadsPerChild for Windows)
- optionally report the IHS worker thread usage at a specified interval
- optionally report the plug-in module where requests are currently stalled
Often this type of monitoring has been performed with mod_status, which requires that a request be sent to the web server from a browser or script, and then be viewed or parsed to record the web server activity. mod_mpmstats can eliminate the need to constantly monitor the mod_status report to determine the current thread usage.
The ability to report the plug-in module where requests are stalled allows the administrator to quickly determine what components to check when the web server is unresponsive.
The mpmstats module, to see the health of the IHS server, displaying the information on the assets and threads being used.
Activation
To activate the mpmstats module should load the module in the IHS server and set the monitoring interval
loadmodule mpmstats_module modules/mod_mpmstats.so # mod_mpmstats will write a report of thread usage every 30 seconds ReportInterval 30
Once the server is configured, it must be restarted for the changes to take effect.
Information
Once enabled statistics, you write about the error file one line at a configured interval as follows:
[Wed Aug 19 14:01:52 2004] [notice] mpmstats: rdy 712 bsy 312 rd 121 WRT 173 can log 0 0 0 DNS 18 cls [Wed Aug 19 14:01:54 2004] [notice] mpmstats: rdy 718 bsy 306 rd 120 WRT 169 0 log 0 0 DNS 17 cls [Wed Aug 19 14:01:56 2004] [notice] mpmstats: rdy 729 bsy 295 rd 123 WRT 135 can log 0 0 0 DNS 37 cls [Wed Aug 19 14:01:58 2004] [notice] mpmstats: rdy 759 bsy 265 rd 130 wr 67 can log 0 0 0 DNS 68 cls [Wed Aug 19 14:02:00 2004] [notice] mpmstats: rdy 809 bsy 215 rd 131 wr 44 can log 0 0 0 DNS 40 cls [Wed Aug 19 14:02:02 2004] [notice] mpmstats: rdy 781 bsy 243 rd 117 WRT 106 can log 0 0 0 DNS 20 cls [Wed Aug 19 14:02:04 2004] [notice] mpmstats: rdy 813 bsy 211 rd 118 wr 72 can log 0 0 0 DNS 21 cls [Wed Aug 19 14:02:07 2004] [notice] mpmstats: rdy 685 bsy 339 rd 177 WRT 122 can log 0 0 0 DNS 40 cls
Where the fields are displayed:
| field | description |
| rdy (ready) | Breed registry id thread initialized and ready to process client requests. |
| bsy (busy) | Table Number of threads that are currently processing requests. |
| rd (reading) | Number of threads (bsy) currently reading customer requests. |
| wr (writing) | Number of threads (bsy) who have already read the request but are processing the same (eg awaiting response from WebSphere Application Server) |
| also (keepalive) | Number of threads (bsy), which are not processing a request but are waiting in case the client sends another request on the same connection. |
| log (logging) | Number of threads (bsy) who are writing to log. |
| dns (dns lookup) | Number of threads (bsy) executing a DNS lookup. |
| cls (closing) | Number of threads (bsy) who are waiting for the ACK from the client indicating that it has processed the response and to close the connection. |
Important note for IBM HTTP Server 7.0 and later users
On any platform where mod_mpmstats is supported, it is now shipped as part of the product (modules/debug/*.so) starting in IBM HTTP Server 7.0 and later. On these releases, diagnostic modules should not be downloaded or installed from this mustgather.
Supported server versions:
mod_mpmstats is provided for AIX, Linux, Solaris, HP-UX, Windows, and z/OS.
There are two levels of support, based on the capabilities of your level of IHS.
Base level of support
- IHS 2.0.42.2 and above
With this level of support, the reports of busy conditions and the number of threads in each state can be generated. The additional module tracking capability enabled by the TrackModules directive is not available.
Module-tracking support
- IHS 2.0.42.2 with PK01070 or later cumulative e-fix
- IHS 2.0.47.1 with PK01070 or later cumulative e-fix
- IHS 6.0.2 or later
The additional information available is the name of the module of request processing, in addition to the information provided with the base level of support.
The Recommended Updates for IBM HTTP Server page has links to download the latest maintenance.
IBM recommendation
Enable this module if you need to track web server thread usage over time but you don’t need extensive details such as what requests are being processed.
If your level of IHS includes module-tracking support, the module tracking feature can result in quicker diagnosis of web server hangs.
Customers with third-party modules and prior problems with unresponsive behavior need to use the module tracking feature.
Note that mod_status provides more details, including actual requests being processed, so we recommend that mod_status be enabled as well.
Installation
IBM HTTP Server 6.1 and earlier: Copy mod_mpmstats.so for your platform to the modules directory in the web server installation location (e.g., to /opt/IBMIHS/modules).
IBM HTTP Server 7.0 and later: On supported platforms, this module is provided with the product in the modules/debug/ subdirectory.
Activation
- Add the following directive to the end of your configuration file:
loadmodule mpmstats_module modules/mod_mpmstats.so
- Add the ReportInterval directive if you want a report of thread usage written to the error log at intervals.
- Add the TrackModules directive if you want the count of requests by modules written to the error log along with the thread usage. Verify that mod_status is loaded and ExtendedStatus is set to On when this directive is used.
- Restart the server so that the updated configuration takes effect.
IBM HTTP Server 7.0 and later: See conf/httpd.conf.default for the LoadModule and NetTrace directives referencing this module from the modules/debug/ subdirectory.
Deactivation
- Comment out the
LoadModule,ReportInterval, andTrackModulesdirectives added as part of the activation step. - Restart the server so that the updated configuration takes effect.
Format of output and optional directives
Whenever enabled, mod_mpmstats will report to the error log when most or all of the web server threads are busy handling clients. Here is an example from Unix:
[Thu Aug 19 14:14:00 2004] [notice] mpmstats: approaching MaxClients (48/50) [Thu Aug 19 14:14:05 2004] [error] server reached MaxClients setting, consider raising the MaxClients setting
(The second message would appear even without mod_mpmstats.)
Here is an example from Windows:
[Wed Aug 17 08:47:29 2005] [notice] mpmstats: approaching ThreadsPerChild (23/25) [Wed Aug 17 08:47:42 2005] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
(The second message would appear even without mod_mpmstats.)
Without this module, the web server will only report the busy condition when it actually reaches the maximum clients setting, and it will only report it once per restart of the server. mod_mpmstats will check every second, and will report the busy condition as often as once every 90 seconds.
ReportInterval
The ReportInterval directive specifies the interval in seconds between reports of the number of threads in different states. If the ReportInterval directive is not specified, reports of thread states will not be written.
Example:
# mod_mpmstats will write a report of thread usage every 30 seconds ReportInterval 30
Here is an example section of the error log when mod_mpmstats has been configured to report the thread usage every two seconds with the ReportInterval 2 directive:
[Thu Aug 19 14:01:52 2004] [notice] mpmstats: rdy 712 bsy 312 rd 121 wr 173 ka 0 log 0 dns 0 cls 18 [Thu Aug 19 14:01:54 2004] [notice] mpmstats: rdy 718 bsy 306 rd 120 wr 169 ka 0 log 0 dns 0 cls 17 [Thu Aug 19 14:01:56 2004] [notice] mpmstats: rdy 729 bsy 295 rd 123 wr 135 ka 0 log 0 dns 0 cls 37 [Thu Aug 19 14:01:58 2004] [notice] mpmstats: rdy 759 bsy 265 rd 130 wr 67 ka 0 log 0 dns 0 cls 68 [Thu Aug 19 14:02:00 2004] [notice] mpmstats: rdy 809 bsy 215 rd 131 wr 44 ka 0 log 0 dns 0 cls 40 [Thu Aug 19 14:02:02 2004] [notice] mpmstats: rdy 781 bsy 243 rd 117 wr 106 ka 0 log 0 dns 0 cls 20 [Thu Aug 19 14:02:04 2004] [notice] mpmstats: rdy 813 bsy 211 rd 118 wr 72 ka 0 log 0 dns 0 cls 21 [Thu Aug 19 14:02:07 2004] [notice] mpmstats: rdy 685 bsy 339 rd 177 wr 122 ka 0 log 0 dns 0 cls 40 [Thu Aug 19 14:02:09 2004] [notice] mpmstats: rdy 707 bsy 317 rd 193 wr 97 ka 0 log 0 dns 0 cls 27 [Thu Aug 19 14:02:11 2004] [notice] mpmstats: rdy 731 bsy 293 rd 196 wr 39 ka 0 log 0 dns 0 cls 58 [Thu Aug 19 14:02:11 2004] [error] [client 9.27.164.15] File does not exist: /home/trawick/testihsbuild/install/htdocs/en_US/jakdsljadslkfjalkfdsasf [Thu Aug 19 14:02:13 2004] [notice] mpmstats: rdy 747 bsy 277 rd 186 wr 71 ka 0 log 0 dns 0 cls 20 [Thu Aug 19 14:02:15 2004] [notice] mpmstats: rdy 749 bsy 275 rd 162 wr 89 ka 0 log 0 dns 0 cls 24 [Thu Aug 19 14:02:17 2004] [notice] mpmstats: rdy 764 bsy 260 rd 156 wr 83 ka 0 log 0 dns 0 cls 21 [Thu Aug 19 14:02:19 2004] [notice] mpmstats: rdy 784 bsy 240 rd 158 wr 33 ka 0 log 0 dns 0 cls 49 [Thu Aug 19 14:02:21 2004] [notice] mpmstats: rdy 788 bsy 236 rd 159 wr 59 ka 0 log 0 dns 0 cls 18 [Thu Aug 19 14:02:23 2004] [notice] mpmstats: rdy 735 bsy 289 rd 161 wr 117 ka 0 log 0 dns 0 cls 11
Note: The report will not be written when the web server is not actively processing any requests. This information does not require ExtendedStatus to be enabled.
The fields logged are described in the table below:
| field | description |
| rdy (ready) | the number of web server threads started and ready to process new client connections |
| bsy (busy) | the number of web server threads already processing a client connection |
| rd (reading) | the number of busy web server threads currently reading the request from the client |
| wr (writing) | the number of busy web server threads that have read the request from the client but are either processing the request (e.g., waiting on a response from WebSphere Application Server) or are writing the response back to the client |
| ka (keepalive) | the number of busy web server threads that are not processing a request but instead are waiting to see if the client will send another request on the same connection; refer to the KeepAliveTimeout directive to decrease the amount of time that a web server thread remains in this state |
| log (logging) | the number of busy web server threads that are writing to the access log |
| dns (dns lookup) | the number of busy web server threads that are performing a dns lookup |
| cls (closing) | the number of busy web server threads that are waiting for the client to acknowledge that the entire response has been received so that the connection can be closed |
Here’s a comparison of displays by mod_mpmstats vs. mod_status for the same server state:
mod_mpmstats would simply write
[Thu Aug 19 15:22:43 2004] [notice] mpmstats: rdy 5 bsy 45 rd 10 wr 3 ka 0 log 0 dns 0 cls 31
The mod_status report would look like this:
Apache Server Status for b80-2.raleigh.ibm.com
- Server Version: IBM_HTTP_Server/2.0.47.1-PQ90698 Apache/2.0.47 (Unix)
- Server Built: Jun 28 2004 10:32:12
- Current Time: Thursday, 19-Aug-2004 15:22:43 EDT
- Restart Time: Thursday, 19-Aug-2004 15:22:03 EDT
- Parent Server Generation: 0
- Server uptime: 40 seconds
- Total accesses: 372 – Total Traffic: 5.7 MB
- CPU Usage: u44.02 s8.66 cu0 cs0 – 132% CPU load
- 9.3 requests/sec – 146.9 kB/second – 15.8 kB/request
- 45 requests currently being processed, 5 idle workers
CCCCCCCCRRCCRR___RWCCCCCCCCCCW_CCRCWRC_CCCCCRCCRRC.............. ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................ ................................................................
Scoreboard Key:
“_” Waiting for Connection, “S” Starting up, “R” Reading Request,
“W” Sending Reply, “K” Keepalive (read), “D” DNS Lookup,
“C” Closing connection, “L” Logging, “G” Gracefully finishing,
“I” Idle cleanup of worker, “.” Open slot with no current process
If ExtendedStatus On were specified in httpd.conf, the actual requests being processed would be listed in the mod_status report.
The mod_status report is easier to read but it is not in a form that is easy to save at intervals and easily see the thread use.
TrackModules
The TrackModules directive is used to enable the reporting of how many requests are currently in each plug-in module. This directive has the following requirements:
- ReportInterval directive must be specified
- mod_status must be loaded and ExtendedStatus must be set to On
- a recent level of IHS must be used (see this information on support levels)
Example:
# mod_mpmstats will write a report of thread usage and counts by # module every 15 seconds ReportInterval 15 TrackModules On
Here is an example section of the error log when mod_mpmstats has been configured to report the thread and module usage every 15 seconds with the prior example:
[Sat Mar 19 07:47:12 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 50 ka 0 log 0 dns 0 cls 0 [Sat Mar 19 07:47:12 2005] [notice] mpmstats: bsy: 50 in mod_cgid.c [Sat Mar 19 07:47:27 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 45 ka 5 log 0 dns 0 cls 0 [Sat Mar 19 07:47:27 2005] [notice] mpmstats: bsy: 45 in mod_cgid.c [Sat Mar 19 07:47:42 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 44 ka 6 log 0 dns 0 cls 0 [Sat Mar 19 07:47:42 2005] [notice] mpmstats: bsy: 44 in mod_cgid.c [Sat Mar 19 07:47:57 2005] [notice] mpmstats: rdy 250 bsy 50 rd 0 wr 44 ka 6 log 0 dns 0 cls 0 [Sat Mar 19 07:47:57 2005] [notice] mpmstats: bsy: 44 in mod_cgid.c
In this example, the predominant type of request is a CGI request, which is handled by mod_cgid. Usually, only a subset of active threads will be attributed to a particular plug-in module. Threads not attributed to a plug-in module are busy performing I/O with the client or are handling basic request processing.
Interpreting the module counts
When web server requests are handled with reasonable response time, the module counts are not so important, and will reflect the types of processing which is being performed. If there is a meaningful number of CGI requests, then some number of IHS threads will typically be busy in mod_cgid.c. If there is a meaningful number of WebSphere requests, then some number of IHS threads will typically be busy in mod_was_ap20_http.c. If SiteMinder is handling authentication, then some number of IHS threads will typically be busy in mod_sm.c.
When response time starts to deteriorate or the web server stops responding altogether, the module counts are more important. Check the reports of module counts for the time leading up to the deteriorated or unresponsive behavior. If a particular module becomes the predominant module where IHS threads are busy, that is the likely cause of the response problem. For example, if a problem occurs with a WebSphere application which prevents it from responding as expected, the number of IHS threads busy in mod_was_ap20_http.c will begin increasing. If there are enough users making requests to that application, all IHS threads could eventually become busy in mod_was_ap20_http.c. Another example is with authentication requests. If SiteMinder is used and a problem occurs with a policy or LDAP server it uses, the number of IHS threads busy in mod_sm.c will begin increasing. If enough requests arrive before the server problem is resolved, all IHS threads could eventually become busy in mod_sm.c.
Recent Comments