{"id":6546,"date":"2017-03-12T21:35:20","date_gmt":"2017-03-12T13:35:20","guid":{"rendered":"http:\/\/rmohan.com\/?p=6546"},"modified":"2017-03-12T23:53:13","modified_gmt":"2017-03-12T15:53:13","slug":"monit-process","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=6546","title":{"rendered":"Monit process"},"content":{"rendered":"\n\n\n
# mysql<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
check process mysqld with pidfile \/var\/lib\/mysql\/ns388683.pid<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
group database<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
start program = “\/etc\/init.d\/mysql start”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
stop program = “\/etc\/init.d\/mysql stop”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if failed host 127.0.0.1 port 3306 then restart<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if 5 restarts within 5 cycles then timeout<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
# nginx<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
check process nginx with pidfile \/opt\/nginx\/logs\/nginx.pid<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
start program = “\/etc\/init.d\/nginx start”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
stop program = “\/etc\/init.d\/nginx stop”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if failed host 127.0.0.1 port 80 then restart<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if cpu is greater than 40% for 2 cycles then alert<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if cpu > 60% for 5 cycles then restart<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if 10 restarts within 10 cycles then timeout<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
# redis<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
check process redis with pidfile \/var\/run\/redis.pid<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
start program = “\/etc\/init.d\/redis-server start”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
stop program = “\/etc\/init.d\/redis-server stop”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
group redis<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
check file dump.rdb with path \/var\/lib\/redis\/dump.rdb<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if size > 100 MB then alert<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
# tomcat<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
check process tomcat with pidfile \/var\/run\/tomcat\/tomcat.pid<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
start program = “\/etc\/init.d\/tomcat start”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
as uid solr gid solr<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
stop program = “\/etc\/init.d\/tomcat stop”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
as uid solr gid solr<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if failed port 8080 then alert<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
if failed port 8080 for 5 cycles then restart<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Now for the tomcat part – this is based on tomcat being in \/usr\/local\/tomcat where our typical setup script puts everything.
\nTomcat, method a (recomended):<\/p>\n

Simply run the below snippet to enable monit monitoring of tomcat. This method requires the least work and changes to configurations. Typically monit prefers a bid file to monitor a service as described in method b, but this way works just as well so long as the http port connector is enabled.<\/p>\n

echo “check host tomcat with address localhost
\nstop program = “\/etc\/init.d\/tomcat stop”
\nstart program = “\/etc\/init.d\/tomcat restart”
\nif failed port 8080 and protocol http
\nthen start
\n” > \/etc\/monit.d\/tomcat
\n\/etc\/init.d\/monit restart<\/p>\n

Tomcat, method b:<\/p>\n

Use this method if you dont have a suitable http connector enabled for your tomcat instance, but be aware that pid files can be left in an inconsistent state in some cases. Which may then require manual intervention anyway. Add a tomcat instance into your config for monit that looks like this (change gid\/uid tomcat runs )<\/p>\n

check process tomcat with pidfile “\/var\/run\/tomcat\/tomcat.pid”
\nstart program = “\/usr\/local\/tomcat\/bin\/startup.sh”
\nas uid tomcat gid tomcat
\nstop program = “\/usr\/local\/tomcat\/bin\/shutdown.sh”
\nas uid tomcat gid tomcat
\nif failed port 8080 then alert
\nif failed port 8080 for 5 cycles then restart<\/p>\n

Then edit your catalina.sh and set<\/p>\n

CATALINA_PID to be \/var\/run\/tomcat\/tomcat.pid
\nJAVA_HOME=\/usr\/java\/jdk<\/p>\n

Add in at the top<\/p>\n

Then of course create the pid directory<\/p>\n

mkdir \/var\/run\/tomcat\/
\nchown tomcat.tomcat \/var\/run\/tomcat\/<\/p>\n

MySQL<\/p>\n

Add this to your my.cnf under [mysqld]<\/p>\n

pid-file=\/var\/run\/mysqld\/mysqld.pid<\/p>\n

and this to your monit config<\/p>\n

check process mysql with pidfile \/var\/run\/mysqld\/mysqld.pid
\ngroup database
\nstart program = “\/etc\/init.d\/mysql start”
\nstop program = “\/etc\/init.d\/mysql stop”
\nif failed host 127.0.0.1 port 3306 protocol mysql then restart
\nif 5 restarts within 5 cycles then timeout
\ndepends on mysql_rc<\/p>\n

check file mysql_rc with path \/etc\/init.d\/mysql
\ngroup database
\nif failed checksum then unmonitor
\nif failed permission 755 then unmonitor
\nif failed uid root then unmonitor
\nif failed gid root then unmonitor<\/p>\n

Disk Space<\/p>\n

Add the following to your monit config<\/p>\n

check filesystem with path \/dev\/xvda1
\nif space usage > 95% then alert
\nif inode usage > 95% then alert<\/p>\n

SSH<\/p>\n

Add this to your monit config (change the port if yours is different)<\/p>\n

check process sshd with pidfile \/var\/run\/sshd.pid
\nstart program “\/etc\/init.d\/sshd start”
\nstop program “\/etc\/init.d\/sshd stop”
\nif failed port 22 protocol ssh then restart
\nif 5 restarts within 5 cycles then timeout<\/p>\n

Shell script<\/p>\n

 <\/p>\n

if failed port 443 protocol https request \/ with timeout 5 seconds for 2 cycles then exec \"\/var\/lib\/monit\/scripts\/notifyAndExecute.sh\" else if succeeded then exec \"\/etc\/monit\/pagerduty-resolve authentication\"<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"

# mysql check process mysqld with pidfile \/var\/lib\/mysql\/ns388683.pid group database start program = “\/etc\/init.d\/mysql start” stop program = “\/etc\/init.d\/mysql stop” if failed host 127.0.0.1 port 3306 then restart if 5 restarts within 5 cycles then timeout # nginx check process nginx with pidfile \/opt\/nginx\/logs\/nginx.pid start program = “\/etc\/init.d\/nginx start” stop program = “\/etc\/init.d\/nginx stop” […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,73],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6546"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6546"}],"version-history":[{"count":4,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6546\/revisions"}],"predecessor-version":[{"id":6550,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6546\/revisions\/6550"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6546"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6546"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6546"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}