August 2025
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

Categories

August 2025
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

Mitigate DDoS attack with ngx_http_limit_req_module and fail2ban

Mitigate DDoS attack with ngx_http_limit_req_module and fail2ban

The fail2ban do have comprehensive collection of scripts that scan log files and ban IPs that match malicious activities.

But we are going to look on how to use ngx_http_limit_req_module logs to ban IPs that shows sign of Distributed Denial of Service (DDoS) attack on your website.
It is assumed in this tutorial that Nginx server is installed in your server.

The following procedures are tested on  running Centos 7 64-bit Linux distribution.

Enable ngx_http_limit_req_module by adding the following script in your Nginx configuration:

http {
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
…
server {
…
limit_req zone=one burst=5;
}
}

Restart Nginx server:

systemctl restart  nginx.service

You will see entry something like this in Nginx error log if there’s abuse detected:

2015/08/27 02:18:05 [error] 21235#21235: *326 limiting requests, excess: 5.297 by zone “one”, client: 91.214.169.44, server: www.webfoobar.com, request: “GET /node/8 HTTP/1.1”, host: “www.webfoobar.com”, referrer: “https://www.webfoobar.com/archive/201502”

We will use this sample log entry for our fail2ban filter script.
Install fail2ban:

yum install -y fail2ban

Create fail2ban filter script based on the Nginx error log entry:

vi /etc/fail2ban/filter.d/nginx-ddos.conf

The content of this filter file:

[Definition]
failregex = limiting requests, excess:.* by zone.*client:
ignoreregex =

We will use the /etc/hosts.deny to block the IP of the DDoS attacker so we will need to create new fail2ban action script:

vi /etc/fail2ban/action.d/hostsdeny.conf

Add the following script as its content:

[Definition]
actionstart =
actionstop =
actioncheck =
actionban = IP= &&
printf %%b “: $IP\n” >>
actionunban = IP= && sed -i.old /ALL:\ $IP/d

[Init]
file = /etc/hosts.deny
daemon_list = ALL

Enable the newly created fail2ban filter:

vi /etc/fail2ban/jail.local

Append the following script:

[nginx-ddos]
enabled = true
port    = http,https
banaction = hostsdeny
findtime = 120
bantime  = 7200
maxretry = 30
logpath = %(nginx_error_log)s

Start the fail2ban service:

systemctl start fail2ban
systemctl enable fail2ban.service
systemctl list-unit-files | grep fail2ban

To check the status of this fail2ban filter:

fail2ban-client status nginx-ddos

You will see something like this:

Status for the jail: nginx-ddos
|- Filter
|  |- Currently failed: 18
|  |- Total failed:     770
|  `- File list:        /var/log/nginx/nginx_error_log
`- Actions
|- Currently banned: 1
|- Total banned:     8
`- Banned IP list:   91.214.169.44

To test if the fail2ban nginx-ddos filter working:

fail2ban-regex /var/log/nginx/nginx_error_log /etc/fail2ban/filter.d/nginx-ddos.conf

You can use apache-bench to test the whole system:

ab -n 20 -c 10 http://www.example.com

Execute the following command to monitor the fail2ban log:

watch -n 1 tail -n 20 /var/log/fail2ban.log

And you will something like this while testing with apache-bench:

May 17th, 2016 | Category: Centos RHEL 7 | Leave a comment

Nginx as reverse proxy for Apache

Setup Nginx as reverse proxy for Apache with Virtualmin support
We know that Nginx is more faster than Apache and most of us prefer to replace Apache with Nginx as their web server.
Nginx is known to serve faster static content and run with less RAM. As of this writing, Virtualmin supports Apache as its web server.

To take advantage of Nginx, we will install it as reverse proxy for Apache and continue using Virtualmin to manage your domains. Nginx configurations for virtual host are tailored for Drupal site and microcache is used here.

The following procedures are tested on my Linode server running Centos 7 64-bit Linux distribution.

Install Nginx

If you need to install Nginx with Pagespeed module please follow the steps here instead and jump to configure Nginx section.
In able to install the latest Nginx server we will need to register Nginx repository:

vi /etc/yum.repos.d/nginx.repo

Have the following codes as its content:

name=nginx repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=0
priority=1
enabled=0

Note: if just in case the nginx does not install try to hard code the $releasever with value of 7
Install Nginx using yum:

yum –enablerepo=nginx -y install nginx

Make Nginx auto-start upon reboot:

chkconfig nginx on

Configure Nginx

Edit the main Nginx file “/etc/nginx/nginx.conf” to match the following:

user nginx;
# This number should be, at maximum, the number of CPU cores on your system.
# (since nginx doesn’t benefit from more than one worker per CPU.)
worker_processes auto;
error_log /var/log/nginx/error.log error;
pid /var/run/nginx.pid;
# Number of file descriptors used for Nginx. This is set in the OS with ‘ulimit -n 200000’
# or using /etc/security/limits.conf
worker_rlimit_nofile 200000;
events {
# Determines how many clients will be served by each worker process.
# (Max clients = worker_connections * worker_processes)
worker_connections 1024;
# Accept as many connections as possible,
# after nginx gets notification about a new connection.
# May flood worker_connections, if that option is set too low.
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main ‘$remote_addr – $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” “$http_x_forwarded_for”‘;
## Use sendfile() syscall to speed up I/O operations and speed up
## static file serving.
# Sendfile copies data between one FD and other from within the kernel.
# More efficient than read() + write(), since the requires transferring
# data to and from the user space.
sendfile on;
## Handling of IPs in proxied and load balancing situations.
set_real_ip_from 0.0.0.0/32; # all addresses get a real IP.
real_ip_header X-Forwarded-For; # the ip is forwarded from the load balancer/proxy
## If you are using CloudFlare, uncomment the lines below
## CloudFlare IPs https://www.cloudflare.com/ips
#set_real_ip_from 199.27.128.0/21;
#set_real_ip_from 173.245.48.0/20;
#set_real_ip_from 103.21.244.0/22;
#set_real_ip_from 103.22.200.0/22;
#set_real_ip_from 103.31.4.0/22;
#set_real_ip_from 141.101.64.0/18;
#set_real_ip_from 108.162.192.0/18;
#set_real_ip_from 190.93.240.0/20;
#set_real_ip_from 188.114.96.0/20;
#set_real_ip_from 197.234.240.0/22;
#set_real_ip_from 198.41.128.0/17;
#set_real_ip_from 162.158.0.0/15;
#set_real_ip_from 104.16.0.0/12;
#set_real_ip_from 172.64.0.0/13;
#set_real_ip_from 2400:cb00::/32;
#set_real_ip_from 2606:4700::/32;
#set_real_ip_from 2803:f800::/32;
#set_real_ip_from 2405:b500::/32;
#set_real_ip_from 2405:8100::/32;
#real_ip_header CF-Connecting-IP;
## Timeouts.
client_body_timeout 60;
client_header_timeout 60;
# Timeout for keep-alive connections. Server will close connections after this time.
keepalive_timeout 10 10;
send_timeout 60;
## Reset lingering timed out connections. Deflect DDoS.
reset_timedout_connection on;
## Body size.
client_max_body_size 10m;
## TCP options.
# don’t buffer data-sends (disable Nagle algorithm).
# Good for sending frequent small bursts of data in real time.
tcp_nodelay on;
## Optimization of socket handling when using sendfile.
# Tcp_nopush causes nginx to attempt to send its HTTP response head in one packet,
# instead of using partial frames. This is useful for prepending headers
# before calling sendfile, or for throughput optimization.
tcp_nopush on;
## Compression.
# Reduces the amount of data that needs to be transferred over the network
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 1;
gzip_http_version 1.1;
gzip_min_length 10;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-javascript application/x-font-ttf text/x-js;
gzip_vary on;
gzip_proxied any; # Compression for all requests.
gzip_disable “MSIE [1-6]\.(?!.*SV1)”;
## Hide the Nginx version number.
server_tokens off;
## Use a SSL/TLS cache for SSL session resume. This needs to be
## here (in this context, for session resumption to work. See this
## thread on the Nginx mailing list:
## http://nginx.org/pipermail/nginx/2010-November/023736.html.
ssl_session_cache shared:SSL:30m;
ssl_session_timeout 1d;
## The server dictates the choice of cipher suites.
ssl_prefer_server_ciphers on;
## No SSL2 support.
## No SSLv3 support (SSLv3 POODLE Vulnerability)
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
## Pregenerated Diffie-Hellman parameters.
ssl_dhparam /etc/nginx/dh_param.pem;
## Curve to use for ECDH.
ssl_ecdh_curve secp521r1;
## Enable OCSP stapling. A better way to revocate server certificates.
ssl_stapling on;
## Enable verification of OCSP stapling responses by the server.
ssl_stapling_verify on;
## Use Google’s DNS
resolver 8.8.4.4 8.8.8.8;
## Enable the builtin cross-site scripting (XSS) filter available
## in modern browsers. Usually enabled by default we just
## reinstate in case it has been somehow disabled for this
## particular server instance.
## https://www.owasp.org/index.php/List_of_useful_HTTP_headers.
add_header X-XSS-Protection ‘1; mode=block’;
## Enable this if using HTTPS
#add_header Strict-Transport-Security “max-age=7200″;
## Block MIME type sniffing on IE.
add_header X-Content-Options nosniff;
## Add a cache miss/hit status header
add_header X-Micro-Cache $upstream_cache_status;
## Block HTTP methods.
map $request_method $not_allowed_method {
default 1;
GET 0;
HEAD 0;
POST 0;
}
## Add as many servers as needed.
## Cf. http://wiki.nginx.org/HttpUpstreamModule.
## Note that this configuration assumes by default that keepalive
## upstream connections are supported and that you have a Nginx
## version with the fair load balancer.
upstream phpapache {
## Use the least connection algorithm for load balancing
least_conn;
server 127.0.0.1:8000;
keepalive 5;
}
## Configuration for reverse proxy. Passing the necessary headers to
## the backend. Nginx doesn’t tunnel the connection, it opens a new
## one. Hence whe need to send these headers to the backend so that
## the client(s) IP is available to them. The host is also sent.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
## Hide the Drupal headers
proxy_hide_header ‘X-Drupal-Cache’;
proxy_hide_header ‘X-Generator’;
## Include blacklist for bad bot and referer blocking.
include blacklist.conf;
## Include the caching setup. Needed for using Drupal with an external cache.
include apps/drupal/drupal_map.conf;
## Defining the proxy cache zone for the microcache as presented at:
## http://fennb.com/microcaching-speed-your-app-up-250x-with-no-n.
proxy_cache_path /var/cache/nginx/microcache levels=1:2 keys_zone=microcache:5M max_size=1G loader_threshold=2592000000 loader_sleep=1 loader_files=100000;
## To build optimal server_names_hash
server_names_hash_bucket_size 72;
## Include all vhosts.
include /etc/nginx/sites-enabled/*;
}

Create the file /etc/nginx/dh_param.pem and add the following to it:

—–BEGIN DH PARAMETERS—–
MIIBCAKCAQEAkD39jm2I+Sr1j1+YPB5TbgUvIWUv3Gzj1s1rtpuZJUhCQ8MElafR
XrjrNXtgN8yjX6J5+Nuj0G9SytrvtKU9T3pLDVjZiV2l0m+/pvzaW3qCSlegpA/S
bkIQPg4n7CP/dhs7JcQD0Ny6TX9iYioDz5/kGfrBHTfAW8A6gPinAiC/+8Osz6mP
UghuQPkFVxJmleIdGU7ll3tAKARJpe8HyHNMNoRGbWTCH1mc8Z/la0E7xjs5R2mh
rYxofg/TMFJyvnnjtTLRQ9edvdA+K9JNsF23t8qvY78ppHNEP7u1PA7ORtePagJk
vcSF5yMYeDzUQLWpOuK5B0yHtltZzANH6wIBAg==
—–END DH PARAMETERS—–

Create the file /etc/nginx/blacklist.conf and add the following to it:

## Add here all user agents that are to be blocked.
map $http_user_agent $bad_bot {
default 0;
~*^Lynx 0; # Let Lynx go through
libwww-perl 1;
~(?i)(httrack|htmlparser|libwww) 1;
}
## Add here all referrers that are to blocked.
map $http_referer $bad_referer {
default 0;
~(?i)(adult|babes|click|diamond|forsale|girl|jewelry|love|nudit|organic|poker|porn|poweroversoftware|sex|teen|webcam|zippo|casino|replica) 1;
}
## Add here all hosts that should be spared any referrer checking.
geo $bad_referer {
127.0.0.1 0;
192.168.1.0/24 0;
}

Create the file /etc/nginx/drupal_map.conf and add the following to it:

## Let Ajax calls go through
map $uri $no_cache_ajax {
default 0;
/system/ajax 1;
}
## Check session cookie being present
map $http_cookie $no_cache_cookie {
default 0;
~SESS 1; # PHP session cookie
}
## Combine both results to get the cache bypassing mapping
map $no_cache_ajax$no_cache_cookie $no_cache {
default 1;
00 0;
}
## Cache bypassing mapping (auth).
map $no_cache_ajax $no_auth_cache {
default 0;
1 1;
}
## Set a cache_uid variable for authenticated users.
map $http_cookie $cache_uid {
default nil;
~SESS[[:alnum:]]+=(?[[:graph:]]+) $session_id;
}

Create the file /etc/nginx/drupal.conf and add the following to it:

location / {
## Let Drupal handle 404
error_page 404 /index.php;
## Regular private file serving (i.e. handled by Drupal).
location ^~ /system/files/ {
proxy_pass http://phpapache;
proxy_http_version 1.1; # keep alive to the Apache upstream
proxy_set_header Connection ”;
## Rewrite the ‘Host’ header to the value in the client request,
## or primary server name
proxy_set_header Host $host;
## For not signaling a 404 in the error log whenever the
## system/files directory is accessed add the line below.
## Note that the 404 is the intended behavior.
log_not_found off;
}
## Trying to access private files directly returns a 404.
location ^~ /sites/[\.\-[:alnum:]]+/files/private/ {
internal;
}
## Support for the file_force module
## http://drupal.org/project/file_force.
location ^~ /system/files_force/ {
proxy_pass http://phpapache;
proxy_http_version 1.1; # keep alive to the Apache upstream
proxy_set_header Connection ”;
## Rewrite the ‘Host’ header to the value in the client request,
## or primary server name
proxy_set_header Host $host;
## For not signaling a 404 in the error log whenever the
## system/files directory is accessed add the line below.
## Note that the 404 is the intended behavior.
log_not_found off;
}
## If accessing an image generated by Drupal imagecache, serve it
## directly if available, if not relay the request to Drupal to (re)generate
## the image.
location ~* /imagecache/ {
## Image hotlinking protection. If you want hotlinking
## protection for your images uncomment the following line.
include hotlinking_protection.conf;
access_log off;
expires 30d;
try_files $uri $uri/ @drupal-noexp;
}
## Drupal generated image handling, i.e., imagecache in core. See:
## http://drupal.org/node/371374.
location ~* /files/styles/ {
## Image hotlinking protection. If you want hotlinking
## protection for your images uncomment the following line.
include hotlinking_protection.conf;
access_log off;
expires 30d;
try_files $uri $uri/ @drupal-noexp;
}
## Advanced Aggregation module CSS/JS
## support. http://drupal.org/project/advagg.
location ~ ^/sites/[\.\-[:alnum:]]+/files/advagg_(?:css|js)/ {
expires max;
gzip_static on;
add_header ETag ”;
add_header Accept-Ranges ”;
# Set a far future Cache-Control header to 52 weeks.
add_header Cache-Control ‘max-age=31449600, no-transform, public’;
location ~* (?:css|js)[_\-[:alnum:]]+\.(?:css|js)(\.gz)?$ {
access_log off;
try_files $uri $uri/ @drupal-noexp;
}
}
## All static files will be served directly.
location ~* ^.+\.(?:css|cur|js|jpe?g|gif|htc|ico|png|htm|html|xml|txt|otf|ttf|eot|woff|svg|webp|webm|zip|gz|tar|rar)$ {
access_log off;
expires 30d;
## No need to bleed constant updates. Send the all shebang in one
## fell swoop.
tcp_nodelay off;
## Set the OS file cache.
open_file_cache max=3000 inactive=120s;
open_file_cache_valid 45s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
try_files $uri $uri/ @drupal-noexp;
}
## PDFs and powerpoint files handling.
location ~* ^.+\.(?:pdf|pptx?)$ {
access_log off;
expires 30d;
## No need to bleed constant updates. Send the all shebang in one
## fell swoop.
tcp_nodelay off;
try_files $uri $uri/ @drupal-noexp;
}
## MP3 and Ogg/Vorbis files are served using AIO when supported. Your OS must support it.
location ~ ^/sites/[\.\-[:alnum:]]+/files/audio/mp3 {
location ~* .*\.mp3$ {
access_log off;
directio 4k; # for XFS
## If you’re using ext3 or similar uncomment the line below and comment the above.
#directio 512; # for ext3 or similar (block alignments)
tcp_nopush off;
aio on;
output_buffers 1 2M;
try_files $uri $uri/ @drupal;
}
}
location ~ ^/sites/[\.\-[:alnum:]]+/files/audio/ogg {
location ~* .*\.ogg$ {
access_log off;
directio 4k; # for XFS
## If you’re using ext3 or similar uncomment the line below and comment the above.
#directio 512; # for ext3 or similar (block alignments)
tcp_nopush off;
aio on;
output_buffers 1 2M;
try_files $uri $uri/ @drupal;
}
}
## Pseudo streaming of FLV files:
## http://wiki.nginx.org/HttpFlvStreamModule.
## If pseudo streaming isn’t working, try to comment
## out in nginx.conf line with:
## add_header X-Frame-Options SAMEORIGIN;
location ~ ^/sites/[\.\-[:alnum:]]+/files/video/flv {
location ~* .*\.flv$ {
access_log off;
flv;
try_files $uri $uri/ @drupal;
}
}
## Pseudo streaming of H264/AAC files. This requires an Nginx
## version greater or equal to 1.0.7 for the stable branch and
## greater or equal to 1.1.3 for the development branch.
## Cf. http://nginx.org/en/docs/http/ngx_http_mp4_module.html.
location ~ ^/sites/[\.\-[:alnum:]]+/files/video/mp4 { # videos
location ~* .*\.(?:mp4|mov)$ {
access_log off;
mp4;
mp4_buffer_size 1M;
mp4_max_buffer_size 5M;
try_files $uri $uri/ @drupal;
}
}
location ~ ^/sites/[\.\-[:alnum:]]+/files/audio/m4a { # audios
location ~* .*\.m4a$ {
access_log off;
mp4;
mp4_buffer_size 1M;
mp4_max_buffer_size 5M;
try_files $uri $uri/ @drupal;
}
}
## Advanced Help module makes each module provided README available.
location ^~ /help/ {
location ~* ^/help/[^/]*/README\.txt$ {
access_log off;
proxy_pass http://phpapache;
proxy_http_version 1.1; # keep alive to the Apache upstream
proxy_set_header Connection ”;
## Rewrite the ‘Host’ header to the value in the client request,
## or primary server name
proxy_set_header Host $host;
}
}
## Replicate the Apache  directive of Drupal standard
## .htaccess. Disable access to any code files. Return a 404 to curtail
## information disclosure. Hide also the text files.
location ~* ^(?:.+\.(?:htaccess|make|txt|engine|inc|info|install|module|profile|po|pot|sh|.*sql|test|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Repository|/Root|/Tag|/Template)$ {
return 404;
}
## First we try the URI and relay to the upstream server if not found.
try_files $uri $uri/ @drupal;
}
## Restrict access to the strictly necessary PHP files. Reducing the
## scope for exploits. Handling of PHP code and the Drupal event loop.
location @drupal {
proxy_pass http://phpapache;
proxy_http_version 1.1; # keep alive to the Apache upstream
proxy_set_header Connection ”;
## Rewrite the ‘Host’ header to the value in the client request,
## or primary server name
proxy_set_header Host $host;
## Proxy microcache
include microcache_proxy.conf;
## The Cache-Control and Expires headers should be delivered untouched
## from the upstream to the client.
proxy_ignore_headers Cache-Control Expires;
## To avoid any interaction with the cache control headers we expire
## everything on this location immediately.
expires epoch;
}
## Restrict access to the strictly necessary PHP files. Reducing the
## scope for exploits. Handling of PHP code and the Drupal event loop.
location @drupal-noexp {
proxy_pass http://phpapache;
proxy_http_version 1.1; # keep alive to the Apache upstream
proxy_set_header Connection ”;
## Rewrite the ‘Host’ header to the value in the client request,
## or primary server name
proxy_set_header Host $host;
## Proxy microcache.
include microcache_proxy.conf;
}
## Disallow access to .bzr, .git, .hg, .svn, .cvs directories
## Return 404 as not to disclose information.
location ^~ /.bzr {
return 404;
}
location ^~ /.git {
return 404;
}
location ^~ /.hg {
return 404;
}
location ^~ /.svn {
return 404;
}
location ^~ /.cvs {
return 404;
}
## Disallow access to patches directory.
location ^~ /patches {
return 404;
}
## Disallow access to drush backup directory.
location ^~ /backup {
return 404;
}
## Disable access logs for robots.txt.
location = /robots.txt {
access_log off;
## Add support for the robotstxt module
## http://drupal.org/project/robotstxt.
try_files $uri $uri/ @drupal;
}
## RSS feed support.
location = /rss.xml {
try_files $uri $uri/ @drupal;
}
## XML Sitemap support.
location = /sitemap.xml {
try_files $uri $uri/ @drupal;
}
## Support for favicon.
## Return an 1×1 transparent GIF if it doesn’t exist.
location = /favicon.ico {
expires 30d;
try_files /favicon.ico @empty;
}
## Return an in memory 1×1 transparent GIF.
location @empty {
expires 30d;
empty_gif;
}
## Any other attempt to access PHP files returns a 404.
location ~* ^.+\.php$ {
return 404;
}

Create the file /etc/nginx/microcache_proxy.conf and add the following to it:

## The cache zone referenced.
proxy_cache microcache;
## The cache key.
proxy_cache_key $cache_uid@$scheme$host$request_uri;
## For 200 and 301 make the cache valid for 5 seconds.
proxy_cache_valid 200 301 5s;
## For 302 make it valid for 1 minute.
proxy_cache_valid 302 1m;
## For 404 make it valid 1 second.
proxy_cache_valid 404 1s;
## If there are any upstream errors or the item has expired use
## whatever it is available.
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504 off;
proxy_pass_header Set-Cookie;
proxy_pass_header Cookie;
## Bypass the cache.
proxy_cache_bypass $no_auth_cache;
proxy_no_cache $no_auth_cache;
## Add a cache miss/hit status header.
add_header X-Micro-Cache $upstream_cache_status;
## Block MIME type sniffing on IE.
add_header X-Content-Options nosniff;
## Cache locking mechanism for protecting the backendof too many
## simultaneous requests.
proxy_cache_lock on;

Create the file /etc/nginx/hotlinking_protection.conf and add the following to it:

## Hotlinking protection for images. Include it in any context you
## want. Adjust the list of allowed referers to your liking.
valid_referers none blocked
www.yahoo.com
www.google.com.ph
www.google.com;
if ($invalid_referer) {
return 200 “No hotlinking allowed\n”;
}

Create the folders /etc/nginx/sites-available and /etc/nginx/sites-enabled:

mkdir /etc/nginx/sites-available /etc/nginx/sites-enabled
chown nginx. /etc/nginx/sites-available /etc/nginx/sites-enabled

In /etc/nginx/sites-available contains the physical file Nginx configurations for your virtual hosts and to enable a virtual host Nginx configuration just create a soft link if this configuration file from /etc/nginx/sites-available to /etc/nginx/sites-enabled. This is easy and good approach to disable and enable a virtual host.
Restart Nginx:

systemctl restart nginx.service

Setup Nginx requirements

In this tutorial the Apache will use port 8000 and lets open this port to become accessible:

iptables -I INPUT -p tcp -m tcp –dport 8000 -j ACCEPT
iptables –line -vnL
service iptables save
service iptables restart

Create the Nginx cache path folder:

mkdir /var/cache/nginx/microcache
chown nginx:root /var/cache/nginx/microcache
chmod 700 /var/cache/nginx/microcache

Create Nginx logrotate script:

vi /etc/logrotate.d/websites_nginx_logs.conf

Content:

/var/log/virtualmin/*nginx_access_log /var/log/virtualmin/*nginx_error_log {
rotate 10
missingok
daily
compress
postrotate
service httpd graceful ; sleep 5
endscript
sharedscripts
}

Configure Apache

Since Nginx is reverse proxy to Apache, the IP address that Apache will get is the IP of the server and we need to correct that. Apache 2.4 and above do have mod_remoteip and we will use that module. Open mod_remoteip’s configuration file:

vi /etc/httpd/conf.d/remoteip.conf

Add the following codes:

# mod_remoteip settings
RemoteIPHeader X-Real-IP
RemoteIPInternalProxy 127.0.0.1
RemoteIPInternalProxy 188.8.8.8

Note: change 188.8.8.8 to your server’s IP address.
Change the port of Apache:

vi /etc/httpd/conf/httpd.conf

Look for:

Listen 80

… and change to:

Listen 8000

Restart Apache:

systemctl restart httpd.service

Configure Virtualmin

Set the virtual server template to listen to 8000. Login to Virtualmin, go to “System Settings” -> “Server Templates” -> “Default Settings” and select from the dropdown “Apache Website”. Change the “Port number for virtual hosts” from 80 to 8000. Restart webmin:

systemctl restart webmin.service

Lets build the necessary scripts that will automate the creation of Nginx virtual host file each time Virtualmin created a new server. First the Nginx virtual host template:

vi /etc/nginx/sites-available/template.conf

The content:

## Configuration for {DOM}.
server {
## Replace XXX.XXX.XXX.XXX with your server’s IPv4 address
listen XXX.XXX.XXX.XXX:80;
## Replace XXXX:XXXX::XXXX:XXXX:XXXX:XXXX with your server’s IPv6 address
listen [XXXX:XXXX::XXXX:XXXX:XXXX:XXXX]:80;
server_name {DOM};
## Redirect permanently to domain with www
return 301 $scheme://www.{DOM}$request_uri;
}
server {
## Replace XXX.XXX.XXX.XXX with your server’s IPv4 address
listen XXX.XXX.XXX.XXX:80;
## Replace XXXX:XXXX::XXXX:XXXX:XXXX:XXXX with your server’s IPv6 address
listen [XXXX:XXXX::XXXX:XXXX:XXXX:XXXX]:80;
server_name www.{DOM};
## Access and error logs.
access_log /var/log/virtualmin/{DOM}_nginx_access_log;
error_log /var/log/virtualmin/{DOM}_nginx_error_log error;
## Root of the site and index.
root {HOME}/public_html;
index index.php;
## Deny access based on the User-Agent header.
if ($bad_bot) {
return 444;
}
## Deny access based on the Referer header.
if ($bad_referer) {
return 444;
}
## Protection against illegal HTTP methods. Only HEAD,
## GET and POST are allowed.
if ($not_allowed_method) {
return 405;
}
## Configuration for Drupal site
include drupal.conf;
}

vi /usr/local/bin/virtualmin.sh

#!/bin/sh
NGINX_CONF_FILE=”/etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf ”

if [ “$VIRTUALSERVER_ACTION” = “CREATE_DOMAIN” ]; then
if [ “${VIRTUALSERVER_WEB}” = “1” ];
then
cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
perl -pi -e “s#{DOM}#$VIRTUALSERVER_DOM#g” $NGINX_CONF_FILE
perl -pi -e “s#{SITE_IP}#$VIRTUALSERVER_IP#g” $NGINX_CONF_FILE
perl -pi -e “s#{HOME}#$VIRTUALSERVER_HOME#g” $NGINX_CONF_FILE
ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
nginx -s reload
fi
elif [ “$VIRTUALSERVER_ACTION” = “DELETE_DOMAIN” ]; then
if [ “${VIRTUALSERVER_WEB}” = “1” ];
then
rm /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
rm /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf
rm /var/log/virtualmin/${VIRTUALSERVER_DOM}_nginx_*
nginx -s reload
fi
elif [ “$VIRTUALSERVER_ACTION” = “MODIFY_DOMAIN” ]; then
if [ “${VIRTUALSERVER_WEB}” = “1” ];
then
if [ ! -f $NGINX_CONF_FILE ]; then
cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
perl -pi -e “s#{DOM}#$VIRTUALSERVER_DOM#g” $NGINX_CONF_FILE
perl -pi -e “s#{SITE_IP}#$VIRTUALSERVER_IP#g” $NGINX_CONF_FILE
perl -pi -e “s#{HOME}#$VIRTUALSERVER_HOME#g” $NGINX_CONF_FILE
ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
fi
fi
if [ “$VIRTUALSERVER_DOM” != “$VIRTUALSERVER_OLDSERVER_DOM” ]; then
if [ “${VIRTUALSERVER_WEB}” = “1” ];
then
OLD_NGINX_CONF_FILE=/etc/nginx/sites-available/${VIRTUALSERVER_OLDSERVER_DOM}.conf
mv $OLD_NGINX_CONF_FILE $NGINX_CONF_FILE
rm /etc/nginx/sites-enabled/${VIRTUALSERVER_OLDSERVER_DOM}.conf
perl -pi -e “s#$VIRTUALSERVER_OLDSERVER_DOM#$VIRTUALSERVER_DOM#g” $NGINX_CONF_FILE
perl -pi -e “s#$VIRTUALSERVER_OLDSERVER_IP#$VIRTUALSERVER_IP#g” $NGINX_CONF_FILE
perl -pi -e “s#$VIRTUALSERVER_OLDSERVER_HOME#$VIRTUALSERVER_HOME#g” $NGINX_CONF_FILE
ln -s /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
fi
fi
if [ “${VIRTUALSERVER_WEB}” = “1” ];
then
nginx -s reload
fi
fi

Make the script executable:

chmod u+x /usr/local/bin/virtualmin.sh

Let Virtualmin know about the virtualmin.sh. Login to Virtualmin, go to “System Settings” -> “Virtualmin Configuration” and select from dropdown “Actions upon server and user creation”. Populate the “Command to run after making changes to a server” field with:

/usr/local/bin/virtualmin.sh

May 17th, 2016 | Category: Nginx | Leave a comment

Banning visitors from a specific country using Fail2ban CENTOS 7

This article, we will take a look on how to exempt from banning visitors from a specific country using Fail2ban and geoip.
It is assumed that Fail2ban is already installed and configured in your server.
Lets install first the geoip:

yum install geoip

Create Fail2ban action script:

vi /etc/fail2ban/action.d/geohostsdeny.conf

Copy the following script:

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart =

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop =

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck =

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
#          Excludes PH|Philippines from banning.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = IP=<ip> &&
COUNTRY=$(geoiplookup $IP | egrep “<country_list>”) && [ “$COUNTRY” ] ||
(printf %%b “<daemon_list>: $IP\n” >> <file>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = IP=<ip> && sed -i.old /ALL:\ $IP/d <file>

[Init]

# Option:  country_list
# Notes.:  List of exempted countries separated by pipe “|”
# Values:  STR  Default:
#
country_list = PH|Philippines

# Option:  file
# Notes.:  hosts.deny file path.
# Values:  STR  Default:  /etc/hosts.deny
#
file = /etc/hosts.deny

# Option:  daemon_list
# Notes:   The list of services that this action will deny. See the man page
#          for hosts.deny/hosts_access. Default is all services.
# Values:  STR  Default: ALL
daemon_list = ALL

The script above will exempt from banning the visitors from Philippines which defined in “country_list”.
To enable our action script in Fail2Ban:

vi /etc/fail2ban/jail.local

Copy the following line:

banaction = geohostsdeny

Restart Fail2Ban:

systemctl restart fail2ban

May 17th, 2016 | Category: Centos RHEL 7 | Leave a comment

MariaDB Galera Cluster on Centos 7

MariaDB is a branch of Mysql, it has been widely used in open source projects, such as hot openstack, therefore, in order to ensure high availability of the service,
while increasing the load capacity of the system, the cluster deployment is essential.

MariaDB Galera Cluster Introduction

MariaDB Galera Cluster is an open source MariaDB synchronous multi-master cluster. It supports only XtraDB / InnoDB storage engine
(although experimental support for MyISAM – see wsrep_replicate_myisam system variable).

Using MariaDB, you may say, “MariaDB Galera Cluster” to the redundancy of the database on CENTOS 7.2

Database redundancy (cluster), although construction was just cumbersome MariaDB Galera Cluster is very simple.

In addition, in order to operate as all the nodes where you want to participate in the Galera Cluster master, you can do the update work of the database at any node.

In this post, we will introduce how to build a MariaDB Galera Cluster.

The main function:

Replication
True multi-master, that is, all nodes can read and write database
Automatic control node membership, the failed node automatically cleared
The new node is added automatically copied data
True parallel copy, row-level
Users can connect directly to a cluster, the use of feelings entirely consistent with MySQL

Advantage:

Because it is multi-master, there is no Slavelag (delayed)
Lost affairs absence
Both read and write scalability
Smaller clients delay
Data is synchronized between nodes, and Master / Slave mode is asynchronous, binlog on different slave may be different

galera_channel_failover_galera

galera_channel_failover_slave

galera_overview

galera_replication1

galera-cluster

Technology:

Galera Cluster replication based Galeralibrary achieve, in order to allow communication with the MySQL library Galera, developed specifically for MySQL wsrep API.

Galera Cluster Synchronization Plug-assurance data, maintain data consistency, can rely on a certified copy, works as follows:

When the client sends a commit instruction, before the transaction is committed, all changes to the database will be write-set collected, and the write-set to send the contents of the record to other nodes.

write-set the certification testing at each node, the node test results determine whether to apply write-set change data.

If the authentication test fails, the node will discard write-set ; if the authentication test is successful, the transaction commits.
1 Prepare the installation environment

Install MariaDB server cluster requires at least three (if only two words requires special configuration, please refer to the official documentation )

Here, I list the test machine configuration:

Work environment

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.71 apache1.rmohan.com apache1
192.168.1.72 apache2.rmohan.com apache2
192.168.1.74 apache3.rmohan.com apache3

Run the three nodes, respectively:

systemctl stop firewalld

You are an expert Firewalld, you do not need to be invalidated. Please open the three ports of 3306,4444,4567.
# systemctl stop firewalld
# systemctl disable firewalld

Disable the Selinux. You do only the following command, but we recommend that you restart.

Then /etc/sysconfig/selinux the selinux set to disabled , this initialize the environment is completed.

setenforce 0
sed -i “s/^SELINUX\=enforcing/SELINUX\=disabled/g” /etc/selinux/config

yum -y install epel-release

vi /etc/yum.repos.d/mariadb.repo
# MariaDB 10.1 CentOS repository list – created 2016-05-16 11:29 UTC
# http://mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

Installation of 2 MariaDB Galera Cluster

Lets install some Pre-requisites and other interesting tools.

yum install rsync nmap lsof perl-DBI nc

yum install -y MariaDB-server MariaDB-client galera rsync MariaDB-Galera-server MariaDB-client

3. Install MariaDB-server and start the Mariadb service

yum install MariaDB-server MariaDB-client MariaDB-compat galera socat jemalloc

Launch the mysql_secure_installation command to set up the username and password. At this point you can start MariaDB normally by using the mysql command and create a database

Complete!
[root@apache1 ~]# ls -ltr /usr/lib64/galera/
total 32832
-rwxr-xr-x 1 root root 33616730 Mar 15 01:52 libgalera_smm.so

[root@apache1 ~]# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we’ll need the current
password for the root user. If you’ve just installed MariaDB, and
you haven’t set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on…

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
… Success!

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] n
… skipping.

Normally, root should only be allowed to connect from ‘localhost’. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
… skipping.

By default, MariaDB comes with a database named ‘test’ that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] n
… skipping.

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
… Success!

Cleaning up…

All done! If you’ve completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

After the normal procedure to set up the first database on apache1, you can create a database and dump it, then import the database on apache2 and apache3.

You have now 3 independant databases on 3 different servers. I would adivce you to dump the whole database with the command

mysqldump –all-databases > db.sql

6. We will now start setting up the Galera clustering. On apache1, edit the /etc/my.cnf.d/server.cnf file on apache1 and configure it as follows. The server apache1 is the being setup as the first primary cluster. So every other cluster is going to be set up in the network that is galera2 and galera3 is going to replicate itself from apache1. Add the following parameter under the [galera] option.
vim /etc/my.cnf.d/server.cnf

[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=’gcomm://’
wsrep_cluster_name=’galera’
wsrep_node_address=’apache1′
wsrep_node_name=’apache1′
wsrep_sst_method=rsync
wsrep_slave_threads=4
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
innodb_locks_unsafe_for_binlog=1
#wsrep_provider_options=”socket.ssl_key=/etc/pki/galera/galera.key; socket.ssl_cert=/etc/pki/galera/galera.crt;”

Tip: If you do not have a way ssl certification, please put wsrep_provider_options commented.

Copy this file to apache2, apache3, attention should wsrep_node_name and wsrep_node_address into corresponding node hostname and ip

[root@apache1 ~]# service mysql start –wsrep_cluster_address=gcomm://
Starting mysql (via systemctl): [ OK ]
[root@apache1 ~]#

[root@apache2 systemd]# vi /etc/my.cnf.d/server.cnf
[root@apache2 systemd]# service mysql start –wsrep_cluster_address=gcomm://
Starting mysql (via systemctl): [ OK ]
[root@apache2 systemd]#
[root@apache3 ~]# vi /etc/my.cnf.d/server.cnf
[root@apache3 ~]# service mysql start –wsrep_cluster_address=gcomm://
Starting mysql (via systemctl): [ OK ]
[root@apache3 ~]#

May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377978452096 [Note] InnoDB: 128 rollback segment(s) are active.
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377978452096 [Note] InnoDB: Waiting for purge to start
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377978452096 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 1616819
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377978452096 [Note] Plugin ‘FEEDBACK’ is disabled.
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377252136704 [Note] InnoDB: Dumping buffer pool(s) not yet started
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377978452096 [Note] Server socket created on IP: ‘::’.
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: 2016-05-14 18:47:34 140377978452096 [Note] /usr/sbin/mysqld: ready for connections.
May 14 18:47:34 apache2.rmohan.com mysqld[19048]: Version: ‘10.1.14-MariaDB’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 MariaDB Server

Create a new user to Galera Server, and set the password.

“New User: root”,: and then to the “password test123”.

# mysql -e “grant all privileges on *.* to root@’%’ identified by ‘test123’ with grant option;”
# mysql -e “grant all privileges on *.* to root@localhost identified by ‘test123′ with grant option;”

MariaDB [(none)]> grant all privileges on *.* to root@’%’ identified by ‘test123’ with grant option;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> grant all privileges on *.* to root@localhost identified by ‘test123′ with grant option;
Query OK, 0 rows affected (0.00 sec)

Create a new user to Galera Server, and set the password.

“New User: root”,: and then to the “password test123″.

7. Once this is added, you can now start the first Galera cluster with the command. Make sure that the mysql service is stopped first. as at step5 we have started mysql to create a test database

service mysql start

[root@apache2 ~]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2605/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1085/sshd
tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN 2605/mysqld

Now you would like to make galera2 joined the primary cluster i.e apache1.

Simple set up the apache2 /etc/my.cnf.d/server.cnf file and under [galera] enter the following parameter.
The only difference is the wsrep_node_name, wsrep_node_address and the gcomm value where you need to add the first cluster.
do not forget to adjust the wsrep_node_address and wsrep_node_name variables.

[root@apache1 ~]#
#Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=’gcomm://apache1.rmohan.com,apache2.rmohan.com,apache3.rmohan.com’
wsrep_cluster_name=’galera’
wsrep_node_address=’192.168.1.71′
wsrep_node_name=’apache1.rmohan.com’
wsrep_sst_method=rsync
wsrep_slave_threads=4
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
innodb_locks_unsafe_for_binlog=1
#wsrep_provider_options=”socket.ssl_key=/etc/pki/galera/galera.key; socket.ssl_cert=/etc/pki/galera/galera.crt;”

[root@apache1 ~]#

[root@apache2 ~]#
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=’gcomm://apache1.rmohan.com,apache2.rmohan.com,apache3.rmohan.com’
wsrep_cluster_name=’galera’
wsrep_node_address=’192.168.1.72′
wsrep_node_name=’apache2.rmohan.com’
wsrep_sst_method=rsync
wsrep_slave_threads=4
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
innodb_locks_unsafe_for_binlog=1
#wsrep_provider_options=”socket.ssl_key=/etc/pki/galera/galera.key; socket.ssl_cert=/etc/pki/galera/galera.crt;”
[root@apache2 ~]#
do not forget to adjust the wsrep_node_address and wsrep_node_name variables.

[root@apache3 ~]#
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=’gcomm://apache1.rmohan.com,apache2.rmohan.com,apache3.rmohan.com’
wsrep_cluster_name=’galera’
wsrep_node_address=’192.168.1.74′
wsrep_node_name=’apache3.rmohan.com’
wsrep_sst_method=rsync
wsrep_slave_threads=4
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
innodb_locks_unsafe_for_binlog=1
#wsrep_provider_options=”socket.ssl_key=/etc/pki/galera/galera.key; socket.ssl_cert=/etc/pki/galera/galera.crt;”

[root@apache3 ~]#
do not forget to adjust the wsrep_node_address and wsrep_node_name variables.

[root@apache2 ~]# less /var/log/messages
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: wsrep_sst_grab()
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: Start replication
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: Setting initial position to 60149e36-1b72-11e6-910f-029ebfb18fbb:0
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: protonet asio version 0
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: Using CRC-32C for message checksums.
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: backend: asio
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: restore pc from disk failed
May 16 22:33:24 apache2 mysqld: 2016-05-16 22:33:24 139713247639680 [Note] WSREP: GMCast version 0
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Warning] WSREP: Failed to resolve tcp://apache3.rmohan.com:4567
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Note] WSREP: (258ce90a, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Note] WSREP: (258ce90a, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Note] WSREP: EVS version 0
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Note] WSREP: gcomm: connecting to group ‘galera’, peer ‘apache1.rmohan.com:,apache2.rmohan.com:,apache3.rmohan.com:’
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Warning] WSREP: (258ce90a, ‘tcp://0.0.0.0:4567’) address ‘tcp://192.168.1.72:4567’ points to own listening address, black
listing
May 16 22:33:25 apache2 mysqld: 2016-05-16 22:33:25 139713247639680 [Note] WSREP: (258ce90a, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://192.168.1.74:4567
May 16 22:33:27 apache2 mysqld: 2016-05-16 22:33:27 139713247639680 [Note] WSREP: declaring 16c80092 at tcp://192.168.1.74:4567 stable
May 16 22:33:27 apache2 mysqld: 2016-05-16 22:33:27 139713247639680 [Note] WSREP: declaring 1f8d7692 at tcp://192.168.1.71:4567 stable
May 16 22:33:27 apache2 mysqld: 2016-05-16 22:33:27 139713247639680 [Note] WSREP: Node 16c80092 state prim
May 16 22:33:27 apache2 mysqld: 2016-05-16 22:33:27 139713247639680 [Note] WSREP: view(view_id(PRIM,16c80092,7) memb {
May 16 22:33:27 apache2 mysqld: 16c80092,0
May 16 22:33:27 apache2 mysqld: 1f8d7692,0
May 16 22:33:27 apache2 mysqld: 258ce90a,0
May 16 22:33:27 apache2 mysqld: } joined {
May 16 22:33:27 apache2 mysqld: } left {
May 16 22:33:27 apache2 mysqld: } partitioned {
May 16 22:33:27 apache2 mysqld: })
May 16 22:33:27 apache2 mysqld: 2016-05-16 22:33:27 139713247639680 [Note] WSREP: save pc into disk
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247639680 [Note] WSREP: gcomm: connected
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247639680 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247639680 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247639680 [Note] WSREP: Opened channel ‘galera’
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247639680 [Note] WSREP: Waiting for SST to complete.
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: STATE EXCHANGE: sent state msg: 27845dea-1b73-11e6-82bb-cbc3208d7287
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: STATE EXCHANGE: got state msg: 27845dea-1b73-11e6-82bb-cbc3208d7287 from 0 (apache3.rmohan.com)
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: STATE EXCHANGE: got state msg: 27845dea-1b73-11e6-82bb-cbc3208d7287 from 1 (apache1.rmohan.com)
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: STATE EXCHANGE: got state msg: 27845dea-1b73-11e6-82bb-cbc3208d7287 from 2 (apache2.rmohan.com)
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: Quorum results:
May 16 22:33:28 apache2 mysqld: version = 3,
May 16 22:33:28 apache2 mysqld: component = PRIMARY,
May 16 22:33:28 apache2 mysqld: conf_id = 6,
May 16 22:33:28 apache2 mysqld: members = 3/3 (joined/total),
May 16 22:33:28 apache2 mysqld: act_id = 0,
May 16 22:33:28 apache2 mysqld: last_appl. = -1,
May 16 22:33:28 apache2 mysqld: protocols = 0/7/3 (gcs/repl/appl),
May 16 22:33:28 apache2 mysqld: group UUID = 60149e36-1b72-11e6-910f-029ebfb18fbb
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: Flow-control interval: [28, 28]
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: Restored state OPEN -> JOINED (0)
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247234816 [Note] WSREP: New cluster view: global state: 60149e36-1b72-11e6-910f-029ebfb18fbb:0, view# 7: Primary, number of nodes: 3, my index: 2, protocol version 3
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139713247639680 [Note] WSREP: SST complete, seqno: 0
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: Member 2.0 (apache2.rmohan.com) synced with group.
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 139712949905152 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
May 16 22:33:28 apache2 mysqld: 2016-05-16 22:33:28 7f11867ef880 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.

Initialize the first cluster node

Start MariaDB with the special ‘-wsrep-new-cluster’ option , Do it on node db1 only so the primary node of the cluster is initialized:
[root@apache1 ~]# /etc/init.d/mysql start –wsrep-new-cluster
Starting mysql (via systemctl): [ OK ]
[root@apache1

[root@apache1 ~]# mysql -u root -p -e”show status like ‘wsrep%'”
Enter password:
+——————————+——————————————————-+
| Variable_name | Value |
+——————————+——————————————————-+
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_cert_index_size | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_cluster_conf_id | 7 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 60149e36-1b72-11e6-910f-029ebfb18fbb |
| wsrep_cluster_status | Primary |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_connected | ON |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_gcomm_uuid | 1f8d7692-1b73-11e6-8496-8af69a3bf125 |
| wsrep_incoming_addresses | 192.168.1.74:3306,192.168.1.71:3306,192.168.1.72:3306 |
| wsrep_last_committed | 0 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cached_downto | 18446744073709551615 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 1 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_local_state_uuid | 60149e36-1b72-11e6-910f-029ebfb18fbb |
| wsrep_protocol_version | 7 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 25.3.15(r3578) |
| wsrep_ready | ON |
| wsrep_received | 4 |
| wsrep_received_bytes | 856 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_thread_count | 5 |
+——————————+——————————————————-+
[root@apache1 ~]#

Some important information in the output are the following lines:

wsrep_local_state_comment | Synced wsrep_incoming_addresses | 192.168.1.74:3306,192.168.1.71:3306,192.168.1.72:3306 wsrep_cluster_size | 1 wsrep_ready | ON all-database.sql

And then restoring it on the node to be connected with
[root@apache1 ~]# mysql -u root -p -e ‘CREATE DATABASE apachetest;’
Enter password:
[root@apache1 ~]# mysql -u root -p -e ‘CREATE TABLE apachetest.mycluster ( id INT NOT NULL AUTO_INCREMENT, name VARCHAR(50), ipaddress VARCHAR(20), PRIMARY KEY(id));’
Enter password:
[root@apache1 ~]# mysql -u root -p -e ‘INSERT INTO apachetest.mycluster (name, ipaddress) VALUES (“apache1”, “192.168.1.71”);’
Enter password:
[root@apache1 ~]# mysql -u root -p -e ‘SELECT * FROM apachetest.mycluster;’
Enter password:
+—-+———+————–+
| id | name | ipaddress |
+—-+———+————–+
| 2 | apache1 | 192.168.1.71 |
+—-+———+————–+
[root@apache1 ~]# mysql -u root -p -e ‘INSERT INTO apachetest.mycluster (name, ipaddress) VALUES (“apache2”, “192.168.1.72”);’
Enter password:
[root@apache1 ~]#

galera-cluster-new-features-deep-dive-webinar-27-638

itfish_40768_0.jpg

Troubleshooting
Name Resolution

If the node trying to connect to an establish cluster is reporting the error:

130307 15:14:52 [Warning] IP address ‘192.168.1.71’ could not be resolved: Name or service not known

And the node it is trying to connect to is displaying the error:

‘wsrep_sst_mysqldump –user ‘root’ –password ‘password’ –host ‘192.168.1.72’ –port ‘3306’ –local-port ‘3306’ –socket ‘/var/lib/mysql/mysql.sock’ –gtid ‘4c754641-e45a-11e2-0800-425dfc14f8f4:390” failed: 1 (Operation not permitted)

The try adding the following option to the nodes configuration files:

skip-name-resolve

Be aware that by doing this, MariaDB will no longer use credentials with host names. That means you’ll have to configure the password on root@127.0.0.1 instead of root@localhost.

Credentials
If the node trying to connect to an establish cluster is reporting the error:

WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.

And the node it is trying to connect to is displaying the error:

[ERROR] WSREP: Try 1/3: ‘wsrep_sst_mysqldump –user ‘root’ –password ‘password’ –host ‘192.168.1.71’ –port ‘3306’ –local-port ‘3306’ –socket ‘/var/lib/mysql/mysql.sock’ –gtid ‘4c754641-e45a-11e2-0800-425dfc14f8f4:420” failed: 1 (Operation not permitted)
ERROR 1045 (28000): Access denied for user ‘root’@’192.168.1.72’ (using password: YES)

Make sure the account on the node trying to connect to the cluster is correct. The established node that is being connected to is trying to send the database state to the connecting node, and is failing to do so because of an authentication error.

Rsync

If you see the error on the established node of
rsync: failed to connect to 192.168.1.72: No route to host (113)

It means that the default state transfer method is Rsync, not mysqldump. To use mysqldump, add the setting
wsrep_sst_method=mysqldump

Or you can open firewall port 4444.

Rsync Wan
If you see the error

sh: wsrep_sst_rsync_wan: command not found
in the log files of the node trying to connect to a cluster, run
cd /usr/bin

ln -s wsrep_sst_rsync wsrep_sst_rsync_wan

File Access
If you get the error File ‘/var/lib/mysql/aria_log_control’ not found make sure that the file is owned my the user mysql.
I ran into this issue running mysql_install_db as root, which meant that the file was owned by root.

Tips and Tricks

Initial Setup

You can find out how large your SQL database is with the command:

SELECT table_schema,
sum(data_length) / 1024 / 1024 “data”,
sum(index_length) / 1024 / 1024 “index”,
sum( data_length + index_length ) / 1024 / 1024 “total”
FROM information_schema.TABLES
GROUP BY table_schema;

This value includes all the indexes, and represents the amount of data MariaDB will transfer to an empty node when it joins the cluster.

To save yourself some time, initialise the nodes that will join the cluster by dumping the SQL data into a file with the command:

mysqldump -u root -ppassword –all-databases > all-database.sql

And then restoring it on the node to be connected with

mysql -u root -ppassword < all-database.sql

Because the backup will not contain the indexes, and can be compressed, you’ll find that you’ll be able to get a remote node up to a relatively recent initial state far more quickly than attempting to sync the entire database through the MariaDB state transfer. In my case a 5GB database could be backed up into a 600 MB compressed SQL file.

It doesn’t matter if the database changes between when you backed it up and when the node connects, because the state transfer will take care of that for you.

Save Bandwidth

Use the setting
wsrep_sst_method=rsync_wan
To save some bandwidth for state transfers over a WAN connection.

mysql -u root -ppassword < all-database.sql

Because the backup will not contain the indexes, and can be compressed, you’ll find that you’ll be able to get a remote node up to a relatively recent initial state far more quickly than attempting to sync the entire database through the MariaDB state transfer. In my case a 5GB database could be backed up into a 600 MB compressed SQL file.

It doesn’t matter if the database changes between when you backed it up and when the node connects, because the state transfer will take care of that for you.

Save Bandwidth

Use the setting
wsrep_sst_method=rsync_wan
To save some bandwidth for state transfers over a WAN connection.

May 16th, 2016 | Category: Centos RHEL 7 | Leave a comment

Pacemaker Apache on High Availability CENTOS 7

Pacemaker Apache on High Availability CENTOS 7

 

 

Red Hat, Inc. introduces new Open Source Software in their every release. Red Hat Enterprise Linux 7 High Availability Add-On introduces a new suite of technologies that underlying high-availability technology based on Pacemaker and Corosync that completely replaces the CMAN and RGManager technologies from previous releases of High Availability Add-On.

HA Add On from RHEL 5 to 7

RHCS-5-7

Pacemaker is a High Availability cluster Software for Linux like Operating System.Pacemaker is known as ‘Cluster Resource Manager‘,

It provides maximum availability of the cluster resources by doing fail over of resources between the cluster nodes.

Pacemaker use corosync for heartbeat and internal communication among cluster components , Corosync also take care of Quorum in cluster.

 

 

haserver_cluster4

In this article we will demonstrate the installation and configuration of two Node Apache (httpd) Web Server Clustering using Pacemaker on CentOS 7.

In my setup i will use two Virtual Machines and Shared Storage from Centos 7 server
two disks will be shared where one disk will be used as fencing device and other disk will used as shared storage for web server )

192.168.1.71 apache1.rmohan.com apache1 -CENTOS7.2 8 GB 2 CPU
192.168.1.72 apache2.rmohan.com apache2 -CENTOS7.2 8 GB 2 CPU

192.168.1.73 storage.rmohan.com storage

[root@apache1 ~]#

Step:1 Update ‘/etc/hosts’ file

Add the following lines in /etc/hosts file in both the nodes.

[root@apache1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.71 apache1.rmohan.com apache1
192.168.1.72 apache2.rmohan.com apache2
192.168.1.73 storage.rmohan.com storage

Step:2 Install the time server

yum install chrony -y

Nothing to do
[root@apache2 ~]# systemctl start crond.service
[root@apache2 ~]# systemctl status crond.service
? crond.service – Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2016-05-14 20:13:14 SGT; 1h 48min ago
Main PID: 848 (crond)
CGroup: /system.slice/crond.service
??848 /usr/sbin/crond -n

May 14 20:13:14 apache2.rmohan.com systemd[1]: Started Command Scheduler.
May 14 20:13:14 apache2.rmohan.com systemd[1]: Starting Command Scheduler…
May 14 20:13:14 apache2.rmohan.com crond[848]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 57% if used.)
May 14 20:13:14 apache2.rmohan.com crond[848]: (CRON) INFO (running with inotify support)
May 14 22:00:15 apache2.rmohan.com systemd[1]: Started Command Scheduler.
[root@apache2 ~]# systemctl enable crond.service
[root@apache2 ~]#

STEP3 Install the Cluster and other required packages.

Use th below yum command on both the nodes to install cluster package (pcs ), fence-agents & web server (httpd)

[root@apache1 ~]# yum -y install pcs fence-agents-all iscsi-initiator-utils httpd

[root@apache2 ~]# yum -y install pcs fence-agents-all iscsi-initiator-utils httpd

STEP4
Set the password to ‘hacluster’ user
It is recommended to use the same password of ‘hacluster’ user on both the nodes.

[root@apache1 ~]# echo p@ssw0rd | passwd –stdin hacluster
Changing password for user hacluster.
passwd: all authentication tokens updated successfully.
[root@apache1 ~]#

[root@apache2 ~]# echo p@ssw0rd | passwd –stdin hacluster
Changing password for user hacluster.
passwd: all authentication tokens updated successfully.
[root@apache2 ~]#

STEP5
Allow High Availability ports in firewall.
Use ‘firewall-cmd‘ command on both the nodes to open High Availability ports in OS firewall.

firewall-cmd –permanent –add-service=high-availability
success

firewall-cmd –reload
success

STEP6
Start the Cluster Service and authorize nodes to join the cluster.

Lets start the cluster service on both the nodes,

[root@apache1 ~]# systemctl start pcsd.service ; systemctl enable pcsd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.
[root@apache1 ~]#

[root@apache2 ~]# systemctl start pcsd.service ; systemctl enable pcsd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.
[root@apache2 ~]#

se below command on either of the node to authorize the nodes to join cluster.

[root@apache1 ~]# pcs cluster auth apache1 apache2
Username: hacluster
Password:
apache1: Authorized
apache2: Authorized
[root@apache1 ~]#

Create the Cluster & enable the Cluster Service

Use below pcs commands on any of the cluster nodes to create a cluster with the name ‘apachecluster‘ and apache1 & apache2 are the cluster nodes.
[root@apache1 ~]# pcs cluster setup –start –name apachecluster apache1 apache2
Shutting down pacemaker/corosync services…
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services…
Removing all cluster configuration files…
apache1: Succeeded
apache2: Succeeded
Starting cluster on nodes: apache1, apache2…
apache2: Starting Cluster…
apache1: Starting Cluster…
Synchronizing pcsd certificates on nodes apache1, apache2…
apache1: Success
apache2: Success

Restaring pcsd on the nodes in order to reload the certificates…
apache1: Success
apache2: Success

Enable the Cluster Service using below pcs command :

[root@apache1 ~]# pcs cluster enable –all
apache1: Cluster Enabled
apache2: Cluster Enabled
[root@apache1 ~]#

Now Verify the cluster Service

[root@apache1 ~]# pcs cluster status
Cluster Status:
Last updated: Sat May 14 12:33:25 2016 Last change: Sat May 14 12:32:36 2016 by hacluster via crmd on apache2
Stack: corosync
Current DC: apache2 (version 1.1.13-10.el7_2.2-44eb2dd) – partition with quorum
2 nodes and 0 resources configured
Online: [ apache1 apache2 ]

PCSD Status:
apache1: Online
apache2: Online
[root@apache1 ~]#

set8
Setup iscsi shared Storage on Fedora Server for both the nodes.

[root@storage ~]# vi /etc/hosts
[root@storage ~]# fdisk -c /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xf218266b.

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-41943039, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition ‘Linux’ to ‘Linux LVM’

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@storage ~]# partprobe /dev/sdb
[root@storage ~]# pvcreate /dev/sdb1
Physical volume “/dev/sdb1” successfully created
[root@storage ~]# vgcreate apachedate_vg /dev/sdb1
Volume group “apachedate_vg” successfully created
[root@storage ~]# fdisk -c /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xba406434.

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-41943039, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition ‘Linux’ to ‘Linux LVM’

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@storage ~]# partprobe /dev/sdc
[root@storage ~]# pvcreate /dev/sdc1
Physical volume “/dev/sdc1” successfully created
[root@storage ~]# vgcreate apachefence_vg /dev/sdc1
Volume group “apachefence_vg” successfully created
[root@storage ~]#
[root@storage ~]#
[root@storage ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 centos lvm2 a– 49.51g 44.00m
/dev/sdb1 apachedate_vg lvm2 a– 20.00g 20.00g
/dev/sdc1 apachefence_vg lvm2 a– 20.00g 20.00g

[root@storage ~]# lvcreate -n apachedata_lvs -l 100%FREE apachedate_vg
Logical volume “apachedata_lvs” created.
[root@storage ~]# lvcreate -n apachefence_lvs -l 100%FREE apachefence_vg
Logical volume “apachefence_lvs” created.
[root@storage ~]#

[root@storage ~]# yum -y install targetcli

[root@storage ~]# targetcli
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type ‘help’.

/backstores/block> ls
o- block ………………………………………………………………………………………… [Storage Objects: 0]
/backstores/block> ls
o- block ………………………………………………………………………………………… [Storage Objects: 0]
/backstores/block> create apachedata /dev/
/dev/apachedate_vg/ /dev/apachefence_vg/ /dev/block/ /dev/bsg/ /dev/bus/ /dev/cdrom
/dev/centos/ /dev/char/ /dev/cpu/ /dev/disk/ /dev/dm-0 /dev/dm-1
/dev/dm-2 /dev/dm-3 /dev/dri/ /dev/fd/ /dev/hugepages/ /dev/input/
/dev/mapper/ /dev/mqueue/ /dev/net/ /dev/pts/ /dev/raw/ /dev/sda
/dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1
/dev/shm/ /dev/snd/ /dev/sr0 /dev/vfio/
/backstores/block> create apachedata /dev/apachedate_vg/apachedata_lvs
Created block storage object apachedata using /dev/apachedate_vg/apachedata_lvs.
/backstores/block> create apachefence /dev/apache
/dev/apachedate_vg/ /dev/apachefence_vg/
/backstores/block> create apachefence /dev/apachefence_vg/apachefence_lvs
Created block storage object apachefence using /dev/apachefence_vg/apachefence_lvs.
/backstores/block> cd /iscsi
/iscsi> ls
o- iscsi ………………………………………………………………………. [Targets: 0]
/iscsi> create
Created target iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> ls
o- iscsi ……………………………………………………………………………………………….. [Targets: 1]
o- iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b ………………………………………………….. [TPGs: 1]
o- tpg1 ……………………………………………………………………………………. [no-gen-acls, no-auth]
o- acls ……………………………………………………………………………………………… [ACLs: 0]
o- luns ……………………………………………………………………………………………… [LUNs: 0]
o- portals ………………………………………………………………………………………… [Portals: 1]
o- 0.0.0.0:3260 …………………………………………………………………………………………. [OK]
/iscsi> cd luns
No such path /iscsi/luns
/iscsi> ls
o- iscsi ……………………………………………………………………………………………….. [Targets: 1]
o- iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b ………………………………………………….. [TPGs: 1]
o- tpg1 ……………………………………………………………………………………. [no-gen-acls, no-auth]
o- acls ……………………………………………………………………………………………… [ACLs: 0]
o- luns ……………………………………………………………………………………………… [LUNs: 0]
o- portals ………………………………………………………………………………………… [Portals: 1]
o- 0.0.0.0:3260 …………………………………………………………………………………………. [OK]
/iscsi> cd iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b/
/iscsi/iqn.20….94eff7fe336b> ls
o- iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b ……………………………………………………. [TPGs: 1]
o- tpg1 ……………………………………………………………………………………… [no-gen-acls, no-auth]
o- acls ……………………………………………………………………………………………….. [ACLs: 0]
o- luns ……………………………………………………………………………………………….. [LUNs: 0]
o- portals ………………………………………………………………………………………….. [Portals: 1]
o- 0.0.0.0:3260 …………………………………………………………………………………………… [OK]
/iscsi/iqn.20….94eff7fe336b> cd luns
No such path /iscsi/iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b/luns
/iscsi/iqn.20….94eff7fe336b> cd tpg1/luns
/iscsi/iqn.20…36b/tpg1/luns> ls
o- luns …………………………………………………………………………………………………… [LUNs: 0]
/iscsi/iqn.20…36b/tpg1/luns> cd luns
No such path /iscsi/iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b/tpg1/luns/luns
/iscsi/iqn.20…36b/tpg1/luns> ls
o- luns ………………………………………………………………………….. [LUNs: 0]
/iscsi/iqn.20…36b/tpg1/luns> [wd
/iscsi/iqn.20…36b/tpg1/luns> pwd
/iscsi/iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b/tpg1/luns
/iscsi/iqn.20…36b/tpg1/luns> create iqn.1994-05.com.redhat:b26f647eddb
storage object or path not valid
/iscsi/iqn.20…36b/tpg1/luns> create iqn.1994-05.com.redhat:b26f647eddb
storage object or path not valid
/iscsi/iqn.20…36b/tpg1/luns> create /backstores/block/apachedata
Created LUN 0.
/iscsi/iqn.20…36b/tpg1/luns> create /backstores/block/apachefence
Created LUN 1.
/iscsi/iqn.20…36b/tpg1/luns> cd ..
/iscsi/iqn.20…f7fe336b/tpg1> ls
o- tpg1 ………………………………………………………………. [no-gen-acls, no-auth]
o- acls ………………………………………………………………………… [ACLs: 0]
o- luns ………………………………………………………………………… [LUNs: 2]
| o- lun0 ………………………………. [block/apachedata (/dev/apachedate_vg/apachedata_lvs)]
| o- lun1 ……………………………. [block/apachefence (/dev/apachefence_vg/apachefence_lvs)]
o- portals …………………………………………………………………… [Portals: 1]
o- 0.0.0.0:3260 ……………………………………………………………………. [OK]
/iscsi/iqn.20…f7fe336b/tpg1> cd acls
/iscsi/iqn.20…36b/tpg1/acls> ls
o- acls ………………………………………………………………………….. [ACLs: 0]
/iscsi/iqn.20…36b/tpg1/acls> create iqn.1994-05.com.redhat:b26f647eddb
Created Node ACL for iqn.1994-05.com.redhat:b26f647eddb
Created mapped LUN 1.
Created mapped LUN 0.
/iscsi/iqn.20…36b/tpg1/acls> create iqn.1994-05.com.redhat:1b9a01e1275
Created Node ACL for iqn.1994-05.com.redhat:1b9a01e1275
Created mapped LUN 1.
Created mapped LUN 0.
/iscsi/iqn.20…36b/tpg1/acls> cd .
/iscsi/iqn.20…36b/tpg1/acls> cd /
/> saveconfig
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
[root@storage ~]# systemctl start target.service
[root@storage ~]#
[root@storage ~]# systemctl enable target.service
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.

Now Scan the iscsi storage on both the nodes :

Run below commands on both the nodes

iscsiadm –mode discovery –type sendtargets –portal 192.168.1.73
iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage.x8664:sn.94eff7fe336b -l -p 192.168.1.73:3260

[root@apache1 ~]# systemctl start iscsi.service
[root@apache1 ~]# systemctl enable iscsi.service
[root@apache1 ~]# systemctl enable iscsid.service
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.
[root@apache1 ~]#

root@apache1 ~]# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 May 14 2016 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0
lrwxrwxrwx 1 root root 10 May 14 2016 dm-name-centos-home -> ../../dm-2
lrwxrwxrwx 1 root root 10 May 14 2016 dm-name-centos-root -> ../../dm-0
lrwxrwxrwx 1 root root 10 May 14 2016 dm-name-centos-swap -> ../../dm-1
lrwxrwxrwx 1 root root 10 May 14 2016 dm-uuid-LVM-YEJwQZi9JXl6MBbZN7XdhQaR09fqbpQH2nyITpeRJVbMnYzojU1b9qSDNbJr0eLn -> ../../dm-0
lrwxrwxrwx 1 root root 10 May 14 2016 dm-uuid-LVM-YEJwQZi9JXl6MBbZN7XdhQaR09fqbpQHhB2KiZ6jcZwYY8OJpwA4l11wnghcdTtJ -> ../../dm-2
lrwxrwxrwx 1 root root 10 May 14 2016 dm-uuid-LVM-YEJwQZi9JXl6MBbZN7XdhQaR09fqbpQHW6P6EC6fmlWdGYY5o41uhw9vKBmWKV0o -> ../../dm-1
lrwxrwxrwx 1 root root 10 May 14 2016 lvm-pv-uuid-YXOIJV-EPlD-dXwg-ePQX-D7av-jPdr-Grb4rp -> ../../sda2
lrwxrwxrwx 1 root root 9 May 14 15:30 scsi-3600140562a971495dce49a581f20d1ea -> ../../sdc
lrwxrwxrwx 1 root root 9 May 14 15:30 scsi-360014059dec3b96b8944a29a6cbe1d5e -> ../../sdb
lrwxrwxrwx 1 root root 9 May 14 15:30 wwn-0x600140562a971495dce49a581f20d1ea -> ../../sdc
lrwxrwxrwx 1 root root 9 May 14 15:30 wwn-0x60014059dec3b96b8944a29a6cbe1d5e -> ../../sdb

Step:9 Create the Cluster Resources.

Define stonith (Shoot The Other Node In The Head) fencing device for the cluster. It is a method to isolate the node from cluster when node become unresponsive.

I am using 1 GB iscsi storage (/dev/sdc ) for fencing.

Run the following commands on either of the node :

[root@apache1 ~]# pcs stonith create scsi_fecing_device fence_scsi pcmk_host_list=”apache1 apache2″ pcmk_monitor_action=”metadata” pcmk_reboot_action=”off” devices=”/dev/disk/by-id/wwn-0x600140562a971495dce49a581f20d1ea” meta provides=”unfencing”
[root@apache1 ~]#

[root@apache1 ~]# fdisk

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xab2385e3.

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (8192-41934847, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-41934847, default 41934847):
Using default value 41934847
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@apache1 ~]# fdisk -l

Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a55d1

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 125829119 62401536 8e Linux LVM

Disk /dev/mapper/centos-root: 40.1 GB, 40093351936 bytes, 78307328 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-swap: 4160 MB, 4160749568 bytes, 8126464 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-home: 19.6 GB, 19574816768 bytes, 38232064 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdb: 21.5 GB, 21470642176 bytes, 41934848 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes

Disk /dev/sdc: 21.5 GB, 21470642176 bytes, 41934848 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
Disk label type: dos
Disk identifier: 0xab2385e3

Device Boot Start End Blocks Id System
/dev/sdc1 8192 41934847 20963328 83 Linux

[root@apache1 ~]# pcs stonith show
scsi_fecing_device (stonith:fence_scsi): Started apache1
[root@apache1 ~]#

Mount the new file system temporary on /var/www and create sub-folders

[root@apache1 ~]# ls -ltr /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 May 14 15:30 wwn-0x60014059dec3b96b8944a29a6cbe1d5e -> ../../sdb
lrwxrwxrwx 1 root root 9 May 14 15:30 scsi-360014059dec3b96b8944a29a6cbe1d5e -> ../../sdb
lrwxrwxrwx 1 root root 9 May 14 19:36 wwn-0x600140562a971495dce49a581f20d1ea -> ../../sdc
lrwxrwxrwx 1 root root 9 May 14 19:36 scsi-3600140562a971495dce49a581f20d1ea -> ../../sdc
lrwxrwxrwx 1 root root 10 May 14 19:36 wwn-0x600140562a971495dce49a581f20d1ea-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 May 14 19:36 scsi-3600140562a971495dce49a581f20d1ea-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 May 14 2016 lvm-pv-uuid-YXOIJV-EPlD-dXwg-ePQX-D7av-jPdr-Grb4rp -> ../../sda2
lrwxrwxrwx 1 root root 9 May 14 2016 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0
lrwxrwxrwx 1 root root 10 May 14 2016 dm-name-centos-root -> ../../dm-0
lrwxrwxrwx 1 root root 10 May 14 2016 dm-uuid-LVM-YEJwQZi9JXl6MBbZN7XdhQaR09fqbpQH2nyITpeRJVbMnYzojU1b9qSDNbJr0eLn -> ../../dm-0
lrwxrwxrwx 1 root root 10 May 14 2016 dm-uuid-LVM-YEJwQZi9JXl6MBbZN7XdhQaR09fqbpQHW6P6EC6fmlWdGYY5o41uhw9vKBmWKV0o -> ../../dm-1
lrwxrwxrwx 1 root root 10 May 14 2016 dm-name-centos-swap -> ../../dm-1
lrwxrwxrwx 1 root root 10 May 14 2016 dm-uuid-LVM-YEJwQZi9JXl6MBbZN7XdhQaR09fqbpQHhB2KiZ6jcZwYY8OJpwA4l11wnghcdTtJ -> ../../dm-2
lrwxrwxrwx 1 root root 10 May 14 2016 dm-name-centos-home -> ../../dm-2
[root@apache1 ~]# fdisk /dev/disk/by-id/wwn-0x60014059dec3b96b8944a29a6cbe1d5e
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x26a39fc5.

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): w
Partition number (1-4, default 1):
First sector (8192-41934847, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-41934847, default 41934847):
Using default value 41934847
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@apache1 ~]# mkfs.ext4 /dev/disk/by-id/wwn-0x60014059dec3b96b8944a29a6cbe1d5e
mke2fs 1.42.9 (28-Dec-2013)
/dev/disk/by-id/wwn-0x60014059dec3b96b8944a29a6cbe1d5e is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=1024 blocks
1310720 inodes, 5241856 blocks
262092 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2153775104
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

mkfs.ext4 /dev/disk/by-id/wwn-0x60014059dec3b96b8944a29a6cbe1d5e
mount /dev/disk/by-id/wwn-0x60014059dec3b96b8944a29a6cbe1d5e /var/www/
df -TH
mkdir /var/www/html
mkdir /var/www/cgi-bin
mkdir /var/www/error
echo “Apache Web Sever Pacemaker Cluster” > /var/www/html/index.html
umount /var/www/

[root@apache1 ~]# pcs resource create webserver_fs Filesystem device=”/dev/disk/by-id/wwn-0x60014059dec3b96b8944a29a6cbe1d5e” directory=”/var/www” fstype=”ext4″ –group webgroup

Create Virtual IP (IPaddr2) Cluster Resource using below command. Execute the following command on any of the node.

[root@apache1 ~]# pcs resource create vip_res IPaddr2 ip=192.168.1.70 cidr_netmask=24 –group webgroup

Add the following lines in ‘/etc/httpd/conf/httpd.conf’ file on both the nodes.
SetHandler server-status
Order deny,allow
Deny from all
Allow from 127.0.0.1

[root@apache1 ~]# pcs resource create apache_res apache configfile=”/etc/httpd/conf/httpd.conf” statusurl=”http://127.0.0.1/server-status” –group webgroup

Pacemaker and pcs on Linux example, configuring cluster resource

Once the cluster and nodes stonith devices configuration is done, then we can start create resource in the cluster.

In this example, there are 3 SAN based storage LUNs, all accessible by the nodes in the cluster, we create filesystem resources to let cluster to manage them, any of them fail to access the resource, the filesystem will float to other nodes.
Resource Creation

Create a filesystem based resource, resource id is fs11

#pcs resource create fs11 ocf:heartbeat:Filesystem params device=/dev/mapper/LUN11 directory=/lun11 fstype=”xfs”

Normally we want gracefully stop the filesystem, and kill the processes that accessing the filesystems when stopping the resource. And have filesystem monitor enabled.

#pcs resource create fs11 ocf:heartbeat:Filesystem params device=/dev/mapper/LUN11 directory=/lun11 fstype=”xfs” fast_stop=”no” force_unmount=”safe” op stop on-fail=stop timeout=200 op monitor on-fail=stop timeout=200 OCF_CHECK_LEVEL=10

From the left, the options are:
– the name of the filesystem resource, also called resource id(fs11)
– the resource agent to use (ocf:heartbeat:Filesystem)
– the block device for the filesystem (e.g. /dev/mapper/lun11)
– the mount point for the filesystem (e.g. /lun11)
– the filesystem type (xfs)
– fast_stop=”no” force_unmount=”safe” are to help the filesystem stop and unmount filesystem
– op monitor on-fail=stop timeout=200 OCF_CHECK_LEVEL=10: similar to stop, also the check level does a read test to see if the filesystem is readable for its monitor probe

To List the created resource fs11

# pcs resource
fs11 (ocf::heartbeat:Filesystem): Started

# pcs resource show fs11
Resource: fs11 (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/mapper/LUN11 directory=/lun11 fstype=xfs fast_stop=no force_unmount=safe
Operations: start interval=0s timeout=60 (fs11-start-timeout-60)
stop on-fail=stop interval=0s timeout=200 (fs11-stop-on-fail-stop-timeout-200)
monitor on-fail=stop interval=60s timeout=200 OCF_CHECK_LEVEL=10 (fs11-monitor-on-fail-stop-timeout-200)

There are 4 properties to identify a resource,

id: the id in the cluster to identify a particular service — > fs11

Rest 3 just after resource id in the command line, seperated by : ocf:heartbeat:Filesystem
ocf: Rresource standard
heartbeat: Provider
Filesystem: type

To check all available resources provided by pacemaker by categories.

#pcs resource list # list all available resources
#pcs resource standards # list resource standards
#pcs resource providers # list all available resource providers
#pcs resource list string # it works as a filter, for example, you want to list resource Filesystem
#pcs resource list Filesystem
ocf:heartbeat:Filesystem – Manages filesystem mounts

Delete resource

Want delete a resource ? here is it.

#pcs resource delete fs11

Resource-Specific Parameters

To check resource type Filesystem parameters, all the parameters can be set and updated

# pcs resource describe Filesystem
ocf:heartbeat:Filesystem – Manages filesystem mounts

Resource script for Filesystem. It manages a Filesystem on a shared storage medium. The standard monitor operation of depth 0 (also known as probe) checks if the filesystem is mounted. If you want deeper tests, set OCF_CHECK_LEVEL to one of the following values: 10: read first 16 blocks of the device (raw read) This doesn’t exercise the filesystem at all, but the device on which the filesystem lives. This is noop for non-block devices such as NFS, SMBFS, or bind mounts. 20: test if a status file can be written and read The status file must be writable by root. This is not always the case with an NFS mount, as NFS exports usually have the “root_squash” option set. In such a setup, you must either use read-only monitoring (depth=10), export with “no_root_squash” on your NFS server, or grant world write permissions on the directory where the status file is to be placed.

Resource options:
device (required): The name of block device for the filesystem, or -U, -L options for mount,
or NFS mount specification.
directory (required): The mount point for the filesystem.
fstype (required): The type of filesystem to be mounted.
options: Any extra options to be given as -o options to mount. For bind mounts, add “bind”
here and set fstype to “none”. We will do the right thing for options such as “bind,ro”.
statusfile_prefix: The prefix to be used for a status file for resource monitoring with depth 20. If you don’t specify
this parameter, all status files will be created in a separate directory.
run_fsck: Specify how to decide whether to run fsck or not. “auto” : decide to run fsck depending on the fstype(default)
“force” : always run fsck regardless of the fstype “no” : do not run fsck ever.
fast_stop: Normally, we expect no users of the filesystem and the stop operation to finish quickly. If you cannot
control the filesystem users easily and want to prevent the stop action from failing, then set this parameter
to “no” and add an appropriate timeout for the stop operation.
force_clones: The use of a clone setup for local filesystems is forbidden by default. For special setups like glusterfs,
cloning a mount of a local device with a filesystem like ext4 or xfs independently on several nodes is a
valid use case. Only set this to “true” if you know what you are doing!
force_unmount: This option allows specifying how to handle processes that are currently accessing the mount directory.
“true” : Default value, kill processes accessing mount point “safe” : Kill processes accessing mount
point using methods that avoid functions that could potentially block during process detection “false” :
Do not kill any processes. The ‘safe’ option uses shell logic to walk the /procs/ directory for pids
using the mount point while the default option uses the fuser cli tool. fuser is known to perform
operations that can potentially block if unresponsive nfs mounts are in use on the system.

Resource Meta Options

The resource meta options can be updated anytime, for example, by now, the fs resource can be started on any node, if you prefer to have resource has a preferred node, then

#pcs status | grep fs11
fs11 (ocf::heartbeat:Filesystem): Started nodeC
#pcs resource meta fs11 stickness=500

Set nodeC to standby, the resource fs11 will float to other nodes

#pcs cluster standby nodeC
#pcs status | grep fs11
fs11 (ocf::heartbeat:Filesystem): Started nodeA

Set nodeC to unstandby, the resource fs11 will float back

pcs cluster unstandby nodeC
#pcs status | grep fs11
fs11 (ocf::heartbeat:Filesystem): Started nodeC

Resource Operations

You can either add operations to a resource when resource creation, or later add the operations, but

#pcs resource op add

Displaying Configured Resources

To list all resources4

# pcs resource show
fs11 (ocf::heartbeat:Filesystem): Started
fs12 (ocf::heartbeat:Filesystem): Started

To list a resource and its full attributes, meta and operations configurations

# pcs resource show fs11
Resource: fs11 (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/mapper/LUN11 directory=/lun11 fstype=xfs fast_stop=no force_unmount=safe
Meta Attrs: stickness=500
Operations: start interval=0s timeout=60 (fs11-start-timeout-60)
stop on-fail=stop interval=0s timeout=200 (fs11-stop-on-fail-stop-timeout-200)
monitor on-fail=stop interval=60s timeout=200 OCF_CHECK_LEVEL=10 (fs11-monitor-on-fail-stop-timeout-200)

To show all resource in full list mode

#pcs resource show –full

Enabling and Disabling Cluster Resources

To disable a resource on a node and don’t want this resource start on other nodes

#pcs resource disable fs11
# pcs resource
fs11 (ocf::heartbeat:Filesystem): Stopped

To enable the resource

#pcs resource enable fs11
# pcs resource
fs11 (ocf::heartbeat:Filesystem): Started

Cluster Resources Cleanup

When a resource messed up, showing some error on start, stop or other situation, to clean up the mess,

#pcs resource cleanup

May 16th, 2016 | Category: Centos RHEL 7 | Leave a comment

Chrony time synch

Alternatively, you can install the new Chrony service that is quicker to synchronize clocks in mobile and virtual systems.

Install the Chrony service:
# yum install -y chrony

Activate the Chrony service at boot:
# systemctl enable chronyd

Start the Chrony service:
# systemctl start chronyd

[root@apache2 ~]# yum install chrony -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.vodien.com
* extras: mirror.vodien.com
* updates: mirror.vodien.com
Package chrony-2.1.1-1.el7.centos.x86_64 already installed and latest version
Nothing to do
[root@apache2 ~]# systemctl start crond.service
[root@apache2 ~]#
[root@apache2 ~]#
[root@apache2 ~]#
[root@apache2 ~]#
[root@apache2 ~]# systemctl status crond.service
? crond.service – Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2016-05-14 20:13:14 SGT; 1h 48min ago
Main PID: 848 (crond)
CGroup: /system.slice/crond.service
??848 /usr/sbin/crond -n

May 14 20:13:14 apache2.rmohan.com systemd[1]: Started Command Scheduler.
May 14 20:13:14 apache2.rmohan.com systemd[1]: Starting Command Scheduler…
May 14 20:13:14 apache2.rmohan.com crond[848]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 57% if used.)
May 14 20:13:14 apache2.rmohan.com crond[848]: (CRON) INFO (running with inotify support)
May 14 22:00:15 apache2.rmohan.com systemd[1]: Started Command Scheduler.
[root@apache2 ~]# systemctl enable crond.service
[root@apache2 ~]# chr
chronyc chronyd chroot chrt
[root@apache2 ~]# chronyd
[root@apache2 ~]# chrony
chronyc chronyd
[root@apache2 ~]# chronyc
chrony version 2.1.1
Copyright (C) 1997-2003, 2007, 2009-2015 Richard P. Curnow and others
chrony comes with ABSOLUTELY NO WARRANTY. This is free software, and
you are welcome to redistribute it under certain conditions. See the
GNU General Public License version 2 for details.

chronyc> quit

The Chrony configuration is in the /etc/chrony.conf file:

[root@apache2 ~]# cat /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

# Ignore stratum in source selection.
stratumweight 0

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Enable kernel RTC synchronization.
rtcsync

# In first three updates step the system clock instead of slew
# if the adjustment is larger than 10 seconds.
makestep 10 3

# Allow NTP client access from local network.
#allow 192.168/16

# Listen for commands only on localhost.
bindcmdaddress 127.0.0.1
bindcmdaddress ::1

# Serve time even if not synchronized to any NTP server.
#local stratum 10

keyfile /etc/chrony.keys

# Specify the key used as password for chronyc.
commandkey 1

# Generate command key if missing.
generatecommandkey

# Disable logging of client accesses.
noclientlog

# Send a message to syslog if a clock adjustment is larger than 0.5 seconds.
logchange 0.5

logdir /var/log/chrony
#log measurements statistics tracking
[root@apache2 ~]#

Note: For basic configuration purpose, only the server directives could need a change to point at a different set of master time servers than the defaults specified.

To get information about the main time reference, type:

#log measurements statistics tracking
[root@apache2 ~]# chronyc tracking
Reference ID : 202.156.0.34 (time2.maxonline.com.sg)
Stratum : 2
Ref time (UTC) : Sat May 14 04:01:50 2016
System time : 0.001108132 seconds fast of NTP time
Last offset : +0.002678378 seconds
RMS offset : 0.002678378 seconds
Frequency : 37.078 ppm fast
Residual freq : +0.000 ppm
Skew : 315.347 ppm
Root delay : 0.015427 seconds
Root dispersion : 0.011265 seconds
Update interval : 64.9 seconds
Leap status : Normal
[root@apache2 ~]#

To get equivalent information to the ntpq command, type:
[root@apache2 ~]# chronyc sources -v
210 Number of sources = 4

.– Source mode ‘^’ = server, ‘=’ = peer, ‘#’ = local clock.
/ .- Source state ‘*’ = current synced, ‘+’ = combined , ‘-‘ = not combined,
| / ‘?’ = unreachable, ‘x’ = time may be in error, ‘~’ = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) –. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- shim.active-apps.net 2 6 37 58 +14ms[ +14ms] +/- 61ms
^- time1.maxonline.com.sg 1 6 77 64 -2046us[-2046us] +/- 3682us
^* time2.maxonline.com.sg 1 6 77 4 -1368us[-6926us] +/- 4931us
^- tor-exit.katzen.me 2 6 77 9 +4244us[+4244us] +/- 49ms

Time synchronization can also be forced with the below chronyc commands.

[root@apache2 ~]# chronyc -a ‘burst 4/4’
200 OK
200 OK
[root@apache2 ~]# chronyc -a makestep
200 OK
200 OK
[root@apache2 ~]#

Chronyc can also be used to view detailed NTP specific information as demonstrated below.
[root@apache2 ~]# chronyc tracking
Reference ID : 202.156.0.34 (time2.maxonline.com.sg)
Stratum : 2
Ref time (UTC) : Sat May 14 04:08:53 2016
System time : 0.000000000 seconds slow of NTP time
Last offset : -0.000051153 seconds
RMS offset : 0.001930560 seconds
Frequency : 2.401 ppm slow
Residual freq : -0.003 ppm
Skew : 2.714 ppm
Root delay : 0.006534 seconds
Root dispersion : 0.000577 seconds
Update interval : 2.1 seconds
Leap status : Normal
[root@apache2 ~]# date
Sat May 14 12:09:54 SGT 2016

We can also view all NTP servers that chronyc is synchronizing with.
[root@apache1 ~]# chronyc sources
210 Number of sources = 4
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ 103.224.117.98 4 6 17 26 -5985us[-6099us] +/- 135ms
^+ frontier.innolan.net 2 6 17 26 +2607us[+2478us] +/- 71ms
^* tor-exit.katzen.me 2 6 17 25 -499us[ -644us] +/- 57ms
^- gandhari.thirdeye.it 2 6 17 25 +3135us[+3135us] +/- 364ms
[root@apache1 ~]# chronyc -a ‘burst 4/4’

Further information on these sources can then be viewed.
[root@centos7 ~]# chronyc sourcestats

[root@apache2 ~]# chronyc sourcestats
210 Number of sources = 4
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
shim.active-apps.net 18 9 689 +1.049 15.882 +2220us 4358us
time1.maxonline.com.sg 12 6 618 +0.582 4.374 +164us 648us
time2.maxonline.com.sg 16 14 624 -0.487 2.781 -185us 563us
tor-exit.katzen.me 12 7 617 -0.846 9.713 +4003us 1389us
[root@apache2 ~]#

May 14th, 2016 | Category: Centos RHEL 7 | Leave a comment

iSCSI Configuration on RHEL 7 / CentOS 7

iSCSI Configuration on RHEL 7 / CentOS 7

Step 1: First you need to create partition

[root@server1 ~]# fdisk -c /dev/sdb

Press ‘p’ to print partition table

Press ‘n’ to create a new partition

Press ‘p’ to create primary partition

Type Partition Number : 1

First Sector : PRESS ENTER

Last Sector : +1G

Press ‘p’ to print partition tables again

Press ‘t’ to change partition ID

Type your partition Number :1

Type Partition code : 8e

Press ‘p’ to print partition tables again

Press ‘w’ to save and exit
Step 2: if required, use partprobe command to update partition table entry into kernel.

[root@server1 ~]# partprobe /dev/sdb

Step 3: Now create a Logical Volume using /dev/sdb1 partition

[root@server1 ~]# pvcreate /dev/sdb1

[root@server1 ~]# vgcreate iSCSI_vg /dev/sdb1

[root@server1 ~]# lvcreate -n iscsi_lv1 -l 100%FREE iSCSI_vg

Step 4: First you need to install “targetcli” package

[root@server1 ~]# yum install targetcli -y

Step 5: Now run targetcli with no options to enter into interactive mode:

[root@server1 ~]# targetcli

/> ls

Now Configure the existing /dev/iSCSI_vg/iscsi_lv1 logical volume as a block-type backing store using the name of “server1.disk1”.

/> cd backstores/

/backstores> ls

/backstores> cd block

/backstores/block> ls

/backstores/block> create server1.disk1 /dev/iSCSI_vg/iscsi_lv1

/backstores/block> ls

Now Create a unique iSCSI Qualified Name (IQN) for the target.

/backstores/block> cd /iscsi

/iscsi> create iqn.2014-10.com.example.server1:iscsi-1

Now an ACL for client node (initiator). the initiator will be connecting with it’s initiator name.

/iscsi> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-1/tpg1/acls

/iscsi/iqn.20…i-1/tpg1/acls> create iqn.2014-10.com.example.com.server1:server2
Now set username and password into ACL to access this LUN

/> cd iqn.2014-10.com.example.com.server1:server2

/> set auth userid=user1

/> set auth password=password

Now Create a LUN under the target, The LUN should use the previously defined backing storage device named “server1.disk1″

/iscsi/iqn.20…i-1/tpg1/acls> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-1/tpg1/luns

/iscsi/iqn.20…i-1/tpg1/luns> create /backstores/block/server1.disk1

Now Configure a portal for the target to listen on 192.168.0.254

/iscsi/iqn.20…i-1/tpg1/luns> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-1/tpg1/portals

/iscsi/iqn.20…/tpg1/portals> create 192.168.0.254

Now view, verify and save the target server configuration

/iscsi/iqn.20…/tpg1/portals> cd /

/> ls

Now Save this configuration

/> saveconfig

/> exit

NOTE-: this configuration will be saved to ” ~]# cat /etc/target/saveconfig.json”

Step 6: Now Enable and Start target service

[root@server1 ~]# systemctl enable target.service

[root@server1 ~]# systemctl restart target.service
[root@server1 ~]# systemctl status target.service

Step 7: Now Configure firewall to allow target service
[root@server1 ~]# firewall-cmd –permanent –add-port=3260/tcp
[root@server1 ~]# firewall-cmd –reload

Accessing iSCSI Storage with CHAP Authentication

Step 1: First you need to install iSCSI initiator package

[root@server2 ~]# yum install iscsi-initiator-utils -y

Step 2: Now Create a unique iSCSI IQN name for the client initiator. Otherwise you will not able to connect/login into IQN

[root@server2 ~]# vim /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.2014-10.com.example.server1:server2
:wq (save and exit)

Step 3: Now you need to modify “/etc/iscsi/iscsid.conf” to provide username and password for chap authentication

[root@server2 ~]# vim /etc/iscsi/iscsid.conf

# line 54: uncomment

node.session.auth.authmethod = CHAP

# line 58,59: uncomment and specify the username and password you set on the iSCSI target server

node.session.auth.username = user1

node.session.auth.password = password
:wq (save and exit)

Step 4: Now Enable and start iscsi client service
[root@server2 ~]# systemctl restart iscsid.service

[root@server2 ~]# systemctl enable iscsid.service

Step 5: Now discover target using the following command:

[root@server2 ~]# iscsiadm -m discovery -t st -p 192.168.0.254

Step 6: Confirm status after discovery

[root@server2 ~]# iscsiadm -m node -o show

Step 7: Now connect/login the discovered target into system

[root@server2 ~]# iscsiadm -m node -T iqn.2014-10.com.example.server1:tgt1 -p 192.168.0.254 -l
Step 8: Confirm the established session

[root@server2 ~]# iscsiadm -m session -o show

Step 9: Confirm the partitions

[root@server2 ~]# cat /proc/partitions

Step 10: Create label, create a new primary partition, format it using xfs file system and the mount it on /mnt directory.

[root@server2 ~]# parted –script /dev/sdb “mklabel msdos”

[root@server2 ~]# parted –script /dev/sdb “mkpart primary 0% 100%”

[root@server2 ~]# mkfs.xfs -i size=1024 -s size=4096 /dev/sdb1

[root@server2 ~]# mount /dev/sdb1 /mnt

[root@server2 ~]# df -hT
Step 11: Now make it persistent entry to mount at booting
[root@server2 ~]# blkid

Now Copy the UUID of /deb/sdb1 and paste it into /etc/fstab as following:

[root@server2 ~]# vim /etc/fstab

UUID=”be41aa12-1e30-4678-8c19-da3506df1d84″ /mnt xfs _netdev 0 0
:wq (save and exit)

[root@server2 ~]# umount /mnt/

[root@server2 ~]# mount -a

[root@server2 ~]# df -h Step 12: Now unmount the iSCSI Storage

[root@server2 ~]# cd

[root@server2 ~]# umount /mnt/

[root@server2 ~]# vim /etc/fstab

Remove the following entry form this file

UUID=”be41aa12-1e30-4678-8c19-da3506df1d84″ /mnt xfs _netdev 0 0
:wq (save and exit)

To Disconnect iSCSI storage

[root@server2 ~]# iscsiadm -m node -T iqn.2014-10.com.example.server1:tgt1 -p 192.168.0.254 -u

To delete cache as well

[root@server2 ~]# iscsiadm -m node -T iqn.2014-10.com.example.server1:tgt1 -p 192.168.0.254 -o delete

Now if you want to connect it again, you need to discover it again.

Configuring iSCSI Targets without CHAP Authentication

Step 1: First you need to create partition

[root@server1 ~]# fdisk -c /dev/sdc

Press ‘p’ to print partition table

Press ‘n’ to create a new partition

Press ‘p’ to create primary partition

Type Partition Number : 1

First Sector : PRESS ENTER

Last Sector : +1G

Press ‘p’ to print partition tables again

Press ‘t’ to change partition ID

Type your partition Number :1

Type Partition code : 8e

Press ‘p’ to print partition tables again

Press ‘w’ to save and exit

Step 2: if required, use partprobe command to update partition table entry into kernel.

[root@server1 ~]# partprobe /dev/sdc

Step 3: Now create a Logical Volume using /dev/sdc1 partition
[root@server1 ~]# pvcreate /dev/sdc1

[root@server1 ~]# vgcreate iSCSI_vg2 /dev/sdc1

[root@server1 ~]# lvcreate -n iscsi_lv2 -l 100%FREE iSCSI_vg2

Step 4: First you need to install “targetcli” package

[root@server1 ~]# yum install targetcli -y

Step 5: Now run targetcli with no options to enter into interactive mode:

[root@server1 ~]# targetcli

/> ls

Now Configure the existing /dev/iSCSI_vg2/iscsi_lv2 logical volume as a block-type backing store using the name of “server1.disk2”.

/> cd backstores/

/backstores> ls

/backstores> cd block

/backstores/block> ls

/backstores/block> create server1.disk2 /dev/iSCSI_vg2/iscsi_lv2

/backstores/block> ls

Now Create a unique iSCSI Qualified Name (IQN) for the target.

/backstores/block> cd /iscsi

/iscsi> create iqn.2014-10.com.example.server1:iscsi-2

Now an ACL for client node (initiator). the initiator will be connecting with it’s initiator name.

/iscsi> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-2/tpg1/acls

/iscsi/iqn.20…i-1/tpg1/acls> create iqn.2014-10.com.example.com.server1:tgt1

By default authentication is enabled. To disable it:

/> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-2/tgp1/

/iscsi/iqn.20…i-1/tpg1> set attribute authentication=0

/iscsi/iqn.20…i-1/tpg1> set attribute generate_node_acls=1

Now Create a LUN under the target, The LUN should use the previously defined backing storage device named “server1.disk2″

/iscsi/iqn.20…i-1/tpg1/acls> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-2/tpg1/luns

/iscsi/iqn.20…i-1/tpg1/luns> create /backstores/block/server1.disk2

Now Configure a portal for the target to listen on 192.168.0.254

/iscsi/iqn.20…i-1/tpg1/luns> cd /iscsi/iqn.2014-10.com.example.server1:iscsi-2/tpg1/portals

/iscsi/iqn.20…/tpg1/portals> create 192.168.0.254

Now view, verify and save the target server configuration

/iscsi/iqn.20…/tpg1/portals> cd /

/> ls

Now Save this configuration

/> saveconfig

/> exit

NOTE-: this configuration will be saved to ” ~]# cat /etc/target/saveconfig.json”

Step 6: Now Enable and Start target service

[root@server1 ~]# systemctl enable target.service

[root@server1 ~]# systemctl restart target.service
[root@server1 ~]# systemctl status target.service

Step 7: Now Configure firewall to allow target service

[root@server1 ~]# firewall-cmd –permanent –add-port=3260/tcp
[root@server1 ~]# firewall-cmd –reload

Accessing iSCSI Storage without CHAP Authentication

Step 1: First you need to install iSCSI initiator package
[root@server2 ~]# yum install iscsi-initiator-utils -y

Step 2: Now Create a unique iSCSI IQN name for the client initiator. Otherwise you will not able to connect/login into IQN

[root@server2 ~]# vim /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.2014-10.com.example.server1:tgt1

:wq (save and exit)

Step 3: Now Enable and start iscsi client service

[root@server2 ~]# systemctl restart iscsid.service

[root@server2 ~]# systemctl enable iscsid.service

Step 4: Now discover target using the following command:
[root@server2 ~]# iscsiadm -m discovery -t st -p 192.168.0.254

Step 5: Now you need to connect iscsi storage into system

[root@server2 ~]# iscsiadm -m node -T iqn.2014-10.com.example.server1:iscsi-2 -p 192.168.0.254 -l

[root@server2 ~]# fdisk -l

Step 6: Create label, create a new primary partition, format it using xfs file system and the mount it on /iscsi2 directory.

[root@server2 ~]# parted –script /dev/sdb “mklabel msdos”

[root@server2 ~]# parted –script /dev/sdb “mkpart primary 0% 100%”

[root@server2 ~]# mkfs.xfs -i size=1024 -s size=4096 /dev/sdb1

[root@server2 ~]# mount /dev/sdb1 /iscsi2

[root@server2 ~]# df -hT

Step 7: Now make it persistent entry to mount at booting

[root@server2 ~]# blkid

Now Copy the UUID of /deb/sdc1 and paste it into /etc/fstab as following:

[root@server2 ~]# vim /etc/fstab

UUID=”be41aa12-1e30-4678-8c19-da3506df1d84″ /iscsi2 xfs _netdev 0 0

:wq (save and exit)

[root@server2 ~]# umount /mnt/

[root@server2 ~]# mount -a

[root@server2 ~]# df -h

Step 8: Now unmount the iSCSI Storage

[root@server2 ~]# cd

[root@server2 ~]# umount /iscsi2

[root@server2 ~]# vim /etc/fstab

Remove the following entry form this file

UUID=”be41aa12-1e30-4678-8c19-da3506df1d84″ /mnt xfs _netdev 0 0

:wq (save and exit)

To Disconnect iSCSI storage

root@server2 ~]# iscsiadm -m node -T iqn.2014-10.com.example.server1:iscsi-2 -p 192.168.0.254 -u

To delete cache as well

[root@server2 ~]# iscsiadm -m node -T iqn.2014-10.com.example.server1:iscsi-2 -p 192.168.0.254 -o delete

Now if you want to connect it again, you need to discover it again.
NOTE-: iSCSI store caching in /var/lib/iscsi/ directory sometimes when we try to add another iscsi targets, system takes some information
from cache. so you have two option to address it.

1. reboot your system

2. remove iscsi cache

To remove nodes cache
[root@server2 ~]# rm -rf /var/lib/iscsi/nodes/*

To remove send_targets cache

[root@server2 ~]# rm -rf /var/lib/iscsi/send_targets/*

May 14th, 2016 | Category: Centos RHEL 7 | Leave a comment

How to create OpenSSH rpm package and its upgrade

How to create OpenSSH rpm package and its upgrade

In this tutorial we will create rpm package of OpenSSH version 6.7 stable version and will do OpenSSH upgrade. You may be wondering why it is required to create rpm package of OpenSSH version. The answer is because of CVE-2014-2532 .

We recently notified with the CVE-2014-2532 , which is a openssh AcceptEnv environment restriction bypass flaw .

Description of CVE-2014-2532

It was found that OpenSSH did not properly handle certain AcceptEnv parameter values with wildcard characters. A remote attacker could use this flaw to bypass intended environment variable restrictions.
Fixed In Version: openssh 6.6

Practical Overview :

1. Creating rpm package from OpenSSH 6.7 tar ball .
2. Upgrading OpenSSH to new version 6.7 .

Precaution while upgrading openssh to new version

1. Take the backup of ssh configuration file that is /etc/ssh
2. Take the backup of pam file that is /etc/pam.d/sshd
3. If working remotely via command line only then install telnet server as a second option for login. (Read the tutorial on how to install telnet server)

Create OpenSSH rpm package

At the time of writing this post we have not found OpenSSH 6.7 version rpm package available from reliable repo. Hence, decided to create our own rpm package.

1. Install required packages for creating OpenSSH rpm package

yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel

2. Create directories for building rpm

mkdir -p /usr/src/redhat/{SOURCES,SPECS}
It will create new directories
(a)/usr/src/redhat/
(b)/usr/src/redhat/SOURCES
(c)/usr/src/redhat/SPECS

3. Download latest OpenSSH package inside /usr/src/redhat/SOURCES/
Download link : http://mirror.team-cymru.org/pub/OpenBSD/OpenSSH/portable/

cd /usr/src/redhat/SOURCES/
wget http://mirror.team-cymru.org/pub/OpenBSD/OpenSSH/portable/openssh-6.7p1.tar.gz
wget http://mirror.team-cymru.org/pub/OpenBSD/OpenSSH/portable/openssh-6.7p1.tar.gz.asc

4. Extract spec file and move to /usr/src/redhat/SPECS directory.

cd /usr/src/redhat/SOURCES/
tar xfz openssh-6.7p1.tar.gz openssh-6.7p1/contrib/redhat/openssh.spec
mv openssh-6.7p1/contrib/redhat/openssh.spec ../SPECS/

5. Change ownership and group of extracted spec file

chown sshd:sshd /usr/src/redhat/SPECS/openssh.spec
6. By using sed command we will disable the ask-pass and replace the deprecated
BuildPreReq with BuildRequires command in spec file.

sed -i -e “s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g” /usr/src/redhat/SPECS/openssh.spec
sed -i -e “s/%define no_x11_askpass 0/%define no_x11_askpass 1/g” /usr/src/redhat/SPECS/openssh.spec
sed -i -e “s/BuildPreReq/BuildRequires/g” /usr/src/redhat/SPECS/openssh.spec
7. Now run rpmbuild command .

cd /usr/src/redhat/SPECS/
rpmbuild -ba openssh.spec
Actually we got one message after running this command that “error: File /root/rpmbuild/SOURCES/openssh-6.7p1.tar.gz: No such file or directory” .

Hence, we will copy the openssh-6.7p1.tar.gz to /root/rpmbuild/SOURCES/

cp -v /usr/src/redhat/SOURCES/openssh-6.7p1.tar.gz* /root/rpmbuild/SOURCES/
8. Now re-run the command once again. I hope this time, it will be successful.

cd /usr/src/redhat/SPECS/
rpmbuild -ba openssh.spec
9. You can find the rpm package in /root/rpmbuild/RPMS/x86_64

cd /root/rpmbuild/RPMS/x86_64
ls -lhrt
Below given is reference from our system

[root@localhost x86_64]# ls
openssh-6.7p1-1.x86_64.rpm openssh-clients-6.7p1-1.x86_64.rpm openssh-debuginfo-6.7p1-1.x86_64.rpm openssh-server-6.7p1-1.x86_64.rpm
[root@localhost x86_64]# pwd
/root/rpmbuild/RPMS/x86_64
[root@localhost x86_64]#
Now you have got the rpm files built from Openssh tar ball.

Next Step : Take backup of ssh

This section is very important, we will take backup of ssh. Because when we upgrade the current OpenSSH to new version 6.7 , PAM configuration files related to ssh will be changes.

1. Take backup of ssh configuration directory
cd ~
tar -cvzf etc_ssh.tar.gx /etc/ssh

2. Take backup of pam.d/sshd file . (Very very important, do not forget)

cp -p /etc/pam.d/sshd ~/sshd.orig.`date +%F`

Upgrade OpenSSH server to version 6.7

Before upgrading to OpenSSH 6.7 version, it is very very important to take ssh backup(Read above section for SSH backup)

Reason: After upgrade the /etc/pam.d/sshd file parameters is incompatible.

Note: We suggest you to follow this procedure in some test machine.So that you will be aware about the issue which may come after upgrade.

1. To upgrade to OpenSSH version 6.7 , run the below given command .

ls -l /root/rpmbuild/RPMS/x86_64/
rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm

2. Take backup copy of after upgrade sshd pam file

cp -p /etc/pam.d/sshd /root/sshd.afterupgrade

3. Now restore original sshd pam file inside /etc/pam.d

cp /root/sshd.orig /etc/pam.d/sshd

4. Now edit /etc/ssh/sshd_config file and enable UsePAM . (Read this post for reason)

vi /etc/ssh/sshd_config
..
UsePAM yes
..
5. We will regenerate new ssh keys by removing old keys. (Read this post for reason)

rm /etc/ssh/ssh*key

6. Now restart the ssh service

On CentOS 7 / RHEL 7

systemctl restart sshd

On CentOS 6 / Amazon Linux
service sshd restart

7. Now try to connect from remote machine via ssh .

You can download already made OpenSSH rpm files from our Github repo.

May 14th, 2016 | Category: Centos, Centos RHEL 7 | Leave a comment

default network name to old “eth0” on RHEL 7 / Fedora 19 disable ipv 6

Tags: biosdevname=0, default network device, GRUB_CMDLINE_LINUX, how to change network name to old eth0, net.ifnames=0, net.ifnames=0 biosdevname=0, Red Hat Enterprise Network name

Step 1) add kernel boot args & regenerate the grub config

The following kernel boot arguments need to be added:
biosdevname=0
net.ifnames=0

Open /etc/default/grub with your favorite editor and add those two options to the line starting with GRUB_CMDLINE_LINUX:

[root@apache2 ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=”console”
GRUB_CMDLINE_LINUX=”crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet”
GRUB_DISABLE_RECOVERY=”true”
[root@apache2 ~]#

Will look like this:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=”console”
GRUB_CMDLINE_LINUX=”crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet net.ifnames=0 biosdevname=0″
GRUB_DISABLE_RECOVERY=”true”

root@apache2 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file …
Found linux image: /boot/vmlinuz-3.10.0-327.18.2.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.18.2.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-327.13.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.13.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-8ac53b429efd489eb5771be2120e30d7
Found initrd image: /boot/initramfs-0-rescue-8ac53b429efd489eb5771be2120e30d7.img
done
[root@apache2 ~]#
[root@apache2 ~]# cd /etc/sysconfig/network-scripts/

[root@apache2 network-scripts]# mv ifcfg-eno16777736 ifcfg-eth0
[root@apache2 network-scripts]# vi ifcfg-eth0

BOOTPROTO=”static”
IPV4_FAILURE_FATAL=”no”
DEVICE=”eth0″
ONBOOT=”yes”
IPADDR=”192.168.1.72″
PREFIX=”24″
NETMASK=”255.255.255.0″
GATEWAY=”192.168.1.254″
DNS1=”8.8.8.8″

root@apache2 network-scripts]# systemctl stop NetworkManager
[root@apache2 network-scripts]# systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
[root@apache2 network-scripts]#

Step 2) add a udev symlink, just to make sure

Basically adding the biosdevname=0 and net.ifnames=0 arguments to grub should be enough. But here’s another way just in case:

[root@test ~]# ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules

[root@apache2 ~]# ifconfig -a
eth0: flags=4163 mtu 1500
inet 192.168.1.72 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::20c:29ff:fed4:db04 prefixlen 64 scopeid 0x20 ether 00:0c:29:d4:db:04 txqueuelen 1000 (Ethernet)
RX packets 191 bytes 19824 (19.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 125 bytes 18952 (18.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Disable IPv6:

First check that IPv6 is enabled or not:
lsmod | grep -i ipv6

Search for the line “GRUB_CMDLINE_LINUX” and add the following at the beginning: “ipv6.disable=1?

Will look like this:
GRUB_CMDLINE_LINUX=”ipv6.disable=1 rd.lvm.lv=rootvg/usrlv…

Create a new configuration based on the currently running system using grub2-mkconfig command:
grub2-mkconfig -o /boot/grub2/grub.cfg

May 14th, 2016 | Category: Centos RHEL 7 | Leave a comment

The keystone CLI is deprecated in favor of python-openstackclient.

The keystone CLI is deprecated in favor of python-openstackclient.

UPDATE: It turns out that installing the new client can cause issues with Keystone. I found this out the hard way yesterday when it failed during a demo, preventing authentication from the command line. After a few hours troubleshooting it turns out Apache (httpd.service) and Keystone (openstack-keystone.service) were clashing. I was unable to fix this regardless of updating each of these services config files to separate them out. Finally guessed it might be the last package I installed that was the cause. After removing python-openstackclient and rebooting the controller node the issue was fixed.

Original post
===============
In OpenStack Kilo the Depreciation message for the Keystone CLI will be displayed whenever using invoking the keystone command. “DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.”

To move to the new python-openstackclient, simply install it. On RHEL7.1:
yum install -y python-openstackclient.noarch

After that it will be available as the command “openstack”. It can be invoked in interactive mode just by typing “openstack” or directly from the command line to get information. For example, to list users:
Old Keystone CLI: “keystone user-list”
New Openstack CLI: “openstack user list”

To be more similar to the output of the old command issue “openstack user list –long” to get the extra fields.

You may also want to update the script “openstack-status” so it uses the new client. To do so, please:
1. Edit /usr/bin/openstack-status with your favorite editor
2. Replace the old command with the new one (around line 227) like so:

#keystone user-list
openstack user list –long

The new CLI can do a lot more of course. For a full list of commands please refer to the below (executed with “openstack” + command):

aggregate add host ip fixed remove server rescue
aggregate create ip floating add server resize
aggregate delete ip floating create server resume
aggregate list ip floating delete server set
aggregate remove host ip floating list server show
aggregate set ip floating pool list server ssh
aggregate show ip floating remove server suspend
availability zone list keypair create server unlock
backup create keypair delete server unpause
backup delete keypair list server unrescue
backup list keypair show server unset
backup restore limits show service create
backup show module list service delete
catalog list network create service list
catalog show network delete service show
command list network list snapshot create
complete network set snapshot delete
compute agent create network show snapshot list
compute agent delete object create snapshot set
compute agent list object delete snapshot show
compute agent set object list snapshot unset
compute service list object save token issue
compute service set object show token revoke
console log show project create usage list
console url show project delete usage show
container create project list user create
container delete project set user delete
container list project show user list
container save project usage list user role list
container show quota set user set
ec2 credentials create quota show user show
ec2 credentials delete role add volume create
ec2 credentials list role create volume delete
ec2 credentials show role delete volume list
endpoint create role list volume set
endpoint delete role remove volume show
endpoint list role show volume type create
endpoint show security group create volume type delete
extension list security group delete volume type list
flavor create security group list volume type set
flavor delete security group rule create volume type unset
flavor list security group rule delete volume unset
flavor set security group rule list
flavor show security group set
flavor unset security group show
help server add security group
host list server add volume
host show server create
hypervisor list server delete
hypervisor show server image create
hypervisor stats show server list
image create server lock
image delete server migrate
image list server pause
image save server reboot
image set server rebuild
image show server remove security group
ip fixed add server remove volume

May 14th, 2016 | Category: Openstack | Leave a comment
« Newer Entries  
  Older Entries »