CentOS 6.7 DRBD installation and configuration notes<\/p>\n
DRDB or Distributed Replicated Block Device is nice for HA situations of distributed block devices. This can be used for NAS, SAN, and a number of other use cases.<\/p>\n
** NOTE THIS SETUP IS FOR ACTIVE\/PASSIVE, NOT ACTIVE<\/p>\n
Primary: 192.168.1.60 (cluster1.rmohan.com)<\/p>\n
Secondary: 192.168.1.61 (cluster2.rmohan.com)
\nRequirements<\/p>\n
\u2013 Two disks (preferably same size)
\n\u2013 Networking between machines (node1 & node2)
\n\u2013 Working DNS resolution (\/etc\/hosts file)
\n\u2013 NTP synchronized times on both nodes
\n\u2013 Selinux Permissive
\n\u2013 Iptables ports (7788) allowed
\n(Primary) configured as a master server only
\n(Secondary) is configured as a server-side only
\n(Primary, Secondary) based server configuration from the server together
\nI. Prepare the environment: (Primary, Secondary)
\nPartition DRBD on both machines<\/p>\n
[root@cluster1 ~]# fdisk \/dev\/sdb
\nDevice contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
\nBuilding a new DOS disklabel with disk identifier 0xe8577511.
\nChanges will remain in memory only, until you decide to write them.
\nAfter that, of course, the previous content won’t be recoverable.<\/p>\n
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)<\/p>\n
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
\nswitch off the mode (command ‘c’) and change display units to
\nsectors (command ‘u’).<\/p>\n
Command (m for help): n
\nCommand action
\ne extended
\np primary partition (1-4)
\np
\nPartition number (1-4): 1
\nFirst cylinder (1-6527, default 1):
\nUsing default value 1
\nLast cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527):
\nUsing default value 6527<\/p>\n
Command (m for help): t
\nSelected partition 1
\nHex code (type L to list codes): 8e
\nChanged system type of partition 1 to 8e (Linux LVM)<\/p>\n
Command (m for help): w
\nThe partition table has been altered!<\/p>\n
Calling ioctl() to re-read partition table.
\nSyncing disks.<\/p>\n
root@cluster1 ~]# iptables -L
\nChain INPUT (policy ACCEPT)
\ntarget prot opt source destination
\nACCEPT all — anywhere anywhere state RELATED,ESTABLISHED
\nACCEPT icmp — anywhere anywhere
\nACCEPT all — anywhere anywhere
\nACCEPT tcp — anywhere anywhere state NEW tcp dpt:ssh
\nREJECT all — anywhere anywhere reject-with icmp-host-prohibited<\/p>\n
Chain FORWARD (policy ACCEPT)
\ntarget prot opt source destination
\nREJECT all — anywhere anywhere reject-with icmp-host-prohibited<\/p>\n
Chain OUTPUT (policy ACCEPT)
\ntarget prot opt source destination
\n[root@cluster1 ~]# \/etc\/init.d\/iptables stop
\niptables: Setting chains to policy ACCEPT: filter [ OK ]
\niptables: Flushing firewall rules: [ OK ]
\niptables: Unloading modules: [ OK ]
\n[root@cluster1 ~]# \/etc\/init.d\/iptables stop
\n[root@cluster1 ~]# iptables -F
\n[root@cluster1 ~]# iptables-save > \/etc\/sysconfig\/iptables<\/p>\n
iptables-save
\n*filter
\n:INPUT ACCEPT [0:0]
\n:FORWARD ACCEPT [0:0]
\n:OUTPUT ACCEPT [14:1480]
\n-A INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT
\n-A INPUT -p icmp -j ACCEPT
\n-A INPUT -i lo -j ACCEPT
\n-A INPUT -p tcp -m state –state NEW -m tcp –dport 22 -j ACCEPT
\n-A INPUT -p tcp -m state –state NEW -m tcp –dport 7788:7799 -j ACCEPT
\n-A INPUT -j REJECT –reject-with icmp-host-prohibited
\n-A FORWARD -j REJECT –reject-with icmp-host-prohibited
\n-A OUTPUT -p tcp -m tcp –dport 7788:7799 -j ACCEPT
\nCOMMIT<\/p>\n
[root@cluster2 ~]# fdisk \/dev\/sdb
\nDevice contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
\nBuilding a new DOS disklabel with disk identifier 0xe1b1fb03.
\nChanges will remain in memory only, until you decide to write them.
\nAfter that, of course, the previous content won’t be recoverable.<\/p>\n
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)<\/p>\n
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
\nswitch off the mode (command ‘c’) and change display units to
\nsectors (command ‘u’).<\/p>\n
Command (m for help): n
\nCommand action
\ne extended
\np primary partition (1-4)
\np
\nPartition number (1-4): 1
\nFirst cylinder (1-6527, default 1):
\nUsing default value 1
\nLast cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527):
\nUsing default value 6527<\/p>\n
Command (m for help): t
\nSelected partition 1
\nHex code (type L to list codes): 8e
\nChanged system type of partition 1 to 8e (Linux LVM)<\/p>\n
Command (m for help): w
\nThe partition table has been altered!<\/p>\n
Calling ioctl() to re-read partition table.
\nSyncing disks.
\n[root@cluster2 ~]# iptables -L
\nChain INPUT (policy ACCEPT)
\ntarget prot opt source destination
\nACCEPT all — anywhere anywhere state RELATED,ESTABLISHED
\nACCEPT icmp — anywhere anywhere
\nACCEPT all — anywhere anywhere
\nACCEPT tcp — anywhere anywhere state NEW tcp dpt:ssh
\nREJECT all — anywhere anywhere reject-with icmp-host-prohibited<\/p>\n
Chain FORWARD (policy ACCEPT)
\ntarget prot opt source destination
\nREJECT all — anywhere anywhere reject-with icmp-host-prohibited<\/p>\n
Chain OUTPUT (policy ACCEPT)
\ntarget prot opt source destination
\n[root@cluster2 ~]# \/etc\/init.d\/iptables stop
\niptables: Setting chains to policy ACCEPT: filter [ OK ]
\niptables: Flushing firewall rules: [ OK ]
\niptables: Unloading modules: [ OK ]
\n[root@cluster2 ~]# iptables -F
\n[root@cluster2 ~]# iptables-save > \/etc\/sysconfig\/iptables
\n[root@cluster2 ~]#
\nCreate mysql dir
\n[root@cluster1 ~]# mkdir \/mysql
\n[root@cluster2 ~]# mkdir \/mysql
\nNtp update<\/p>\n
[root@cluster1 ~]# chkconfig –level 345 ntpd on
\n[root@cluster1 ~]# \/etc\/init.d\/ntpd restart
\nShutting down ntpd: [FAILED]
\nStarting ntpd: [ OK ]
\n[root@cluster1 ~]# ntpdate -u 0.centos.pool.ntp.org<\/p>\n
[root@cluster2 ~]# chkconfig –level 345 ntpd on
\n[root@cluster2 ~]# \/etc\/init.d\/ntpd restart
\nShutting down ntpd: [FAILED]
\nStarting ntpd: [ OK ]
\n[root@cluster2 ~]# ntpdate -u 0.centos.pool.ntp.org
\n1. Install dependencies: (Primary, Secondary)<\/p>\n
# yum install gcc gcc-c++ make glibc flex kernel-devel kernel-headers<\/p>\n
Install rpm -Uvh http:\/\/www.elrepo.org\/elrepo-release-6-6.el6.elrepo.noarch.rpm<\/p>\n
Install DRBD on both nodes cluster1 and cluster2<\/p>\n
yum -y install drbd83-utils kmod-drbd83<\/p>\n
Insert drbd module manually on both machines or reboot:
\n\/sbin\/modprobe drbd<\/p>\n
add the config<\/p>\n
Create the Distributed Replicated Block Device resource file<\/p>\n
\/etc\/drbd.d\/clusterdb.res<\/p>\n
resource clusterdb
\n{
\nstartup {
\nwfc-timeout 30;
\noutdated-wfc-timeout 20;
\ndegr-wfc-timeout 30;
\n}<\/p>\n
net {
\ncram-hmac-alg sha1;
\nshared-secret sync_disk;
\n}<\/p>\n
syncer {
\nrate 10M;
\nal-extents 257;
\non-no-data-accessible io-error;
\n}
\non cluster1.rmohan.com {
\ndevice \/dev\/drbd0;
\ndisk \/dev\/sdb1;
\naddress 192.168.1.60:7788;
\nflexible-meta-disk internal;
\n}
\non cluster2.rmohan.com {
\ndevice \/dev\/drbd0;
\ndisk \/dev\/sdb1;
\naddress 192.168.1.62:7788;
\nmeta-disk internal;
\n}
\n}<\/p>\n
Initialize the DRBD meta data storage on both machines:<\/p>\n
drbdadm create-md clusterdb<\/p>\n
start drbd on both node
\nservice drbd start<\/p>\n
On the PRIMARY node run drbdadm command:
\ndrbdsetup \/dev\/drbd0 primary –overwrite-data-of-peer<\/p>\n
Every 2.0s: service drbd status Thu Nov 26 22:13:34 2015
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb SyncSource Primary\/Secondary UpToDate\/Inconsistent C
\n… sync’ed: 1.2% (50624\/51196)M
\nwatch service drbd status<\/p>\n
[root@cluster1 ~]# service drbd status
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb Connected Primary\/Secondary UpToDate\/UpToDate C
\n[root@cluster1 ~]#
\n\/sbin\/mkfs.ext4 \/dev\/drbd0
\nmke2fs 1.41.12 (17-May-2010)
\nFilesystem label=
\nOS type: Linux
\nBlock size=4096 (log=2)
\nFragment size=4096 (log=2)
\nStride=0 blocks, Stripe width=0 blocks
\n3276800 inodes, 13106615 blocks
\n655330 blocks (5.00%) reserved for the super user
\nFirst data block=0
\nMaximum filesystem blocks=4294967296
\n400 block groups
\n32768 blocks per group, 32768 fragments per group
\n8192 inodes per group
\nSuperblock backups stored on blocks:
\n32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
\n4096000, 7962624, 11239424<\/p>\n
Writing inode tables: done
\nCreating journal (32768 blocks): done
\nWriting superblocks and filesystem accounting information: done<\/p>\n
This filesystem will be automatically checked every 37 mounts or
\n180 days, whichever comes first. Use tune2fs -c or -i to override.<\/p>\n
root@cluster1 ~]# mount -t ext4 \/dev\/drbd0 \/mysql\/
\n[root@cluster1 ~]# df -TH
\nFilesystem Type Size Used Avail Use% Mounted on
\n\/dev\/mapper\/vg_cluster1-lv_root
\next4 53G 1.8G 49G 4% \/
\ntmpfs tmpfs 3.0G 0 3.0G 0% \/dev\/shm
\n\/dev\/sda1 ext4 500M 57M 417M 13% \/boot
\n\/dev\/mapper\/vg_cluster1-lv_home
\next4 47G 55M 44G 1% \/home
\n\/dev\/drbd0 ext4 53G 55M 50G 1% \/mysql
\n[root@cluster1 ~]#
\n[root@cluster1 mysql]# date ; drbdadm role clusterdb
\nFri Nov 27 13:28:57 SGT 2015
\nPrimary\/Secondary<\/p>\n
[root@cluster2 ~]# date ; drbdadm role clusterdb
\nFri Nov 27 13:29:02 SGT 2015
\nSecondary\/Primary
\nverify replication is up to date<\/p>\n
[root@cluster1 ~]# cat \/proc\/drbd
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\n0: cs:Connected ro:Primary\/Secondary ds:UpToDate\/UpToDate C r—–
\nns:56 nr:0 dw:0 dr:1052 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<\/p>\n
Admin Commands:<\/p>\n
[root@cluster1 ~]# drbd-overview
\n0:clusterdb Connected Primary\/Secondary UpToDate\/UpToDate C r—–<\/p>\n
[root@cluster1 ~]# \/etc\/init.d\/drbd status
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb Connected Primary\/Secondary UpToDate\/UpToDate C
\nBASIC MANUAL FAIL-OVER
\n# umount \/dev\/drbd0
\n# drbdadm secondary <resource>
\nNow on the node we wish to make primary promote the resource and mount the device.
\n# drbdadm primary <resource>
\n# mount \/dev\/drbd0\/by-res\/<resource> <mountpoint><\/p>\n
disconnecting disk:<\/p>\n
drbdam disconnect clusterdb
\ndrbdadm disconnect clusterdb
\ndf -TH
\numount \/dev\/drbd0
\ndrbdadm secondary clusterdb<\/p>\n
drbdadm connect clusterdb
\ndrbdadm primary clusterdb
\nmount -t ext4 \/dev\/drbd0 \/mysql\/
\ndf -TH
\ncd \/mysql\/
\nFix DRBD recovery from split brain<\/p>\n
root@cluster1 ~]# \/etc\/init.d\/drbd status
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb StandAlone Primary\/Unknown UpToDate\/DUnknown r—–
\n[root@cluster2 ~]# \/etc\/init.d\/drbd status
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb StandAlone Secondary\/Unknown UpToDate\/DUnknown r—–
\nSolution:<\/p>\n
Step 1: Start drbd manually on both nodes<\/p>\n
Step 2: Define one node as secondary and discard data on this<\/p>\n
drbdadm secondary all
\ndrbdadm disconnect all
\ndrbdadm — –discard-my-data connect all<\/p>\n
Step 3: Define anoher node as primary and connect
\ndrbdadm primary all
\ndrbdadm disconnect all
\ndrbdadm connect all<\/p>\n
[root@cluster1 ~]# \/etc\/init.d\/drbd status
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb Connected Primary\/Secondary UpToDate\/UpToDate C
\n[root@cluster2 ~]# \/etc\/init.d\/drbd status
\ndrbd driver loaded OK; device status:
\nversion: 8.3.16 (api:88\/proto:86-97)
\nGIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
\nm:res cs ro ds p mounted fstype
\n0:clusterdb Connected Secondary\/Primary UpToDate\/UpToDate C<\/p>\n","protected":false},"excerpt":{"rendered":"
CentOS 6.7 DRBD installation and configuration notes<\/p>\n
DRDB or Distributed Replicated Block Device is nice for HA situations of distributed block devices. This can be used for NAS, SAN, and a number of other use cases.<\/p>\n
** NOTE THIS SETUP IS FOR ACTIVE\/PASSIVE, NOT ACTIVE<\/p>\n
Primary: 192.168.1.60 (cluster1.rmohan.com)<\/p>\n
Secondary: 192.168.1.61 (cluster2.rmohan.com) Requirements<\/p>\n
\u2013 Two disks […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,57],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/5395"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5395"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/5395\/revisions"}],"predecessor-version":[{"id":5396,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/5395\/revisions\/5396"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5395"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5395"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5395"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}