CentOS 6.7 DRBD installation and configuration notes
DRDB or Distributed Replicated Block Device is nice for HA situations of distributed block devices. This can be used for NAS, SAN, and a number of other use cases.
** NOTE THIS SETUP IS FOR ACTIVE/PASSIVE, NOT ACTIVE
Primary: 192.168.1.60 (cluster1.rmohan.com)
Secondary: 192.168.1.61 (cluster2.rmohan.com)
Requirements
– Two disks (preferably same size)
– Networking between machines (node1 & node2)
– Working DNS resolution (/etc/hosts file)
– NTP synchronized times on both nodes
– Selinux Permissive
– Iptables ports (7788) allowed
(Primary) configured as a master server only
(Secondary) is configured as a server-side only
(Primary, Secondary) based server configuration from the server together
I. Prepare the environment: (Primary, Secondary)
Partition DRBD on both machines
[root@cluster1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe8577511.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won’t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-6527, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527):
Using default value 6527
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
root@cluster1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all — anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp — anywhere anywhere
ACCEPT all — anywhere anywhere
ACCEPT tcp — anywhere anywhere state NEW tcp dpt:ssh
REJECT all — anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all — anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@cluster1 ~]# /etc/init.d/iptables stop
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]
[root@cluster1 ~]# /etc/init.d/iptables stop
[root@cluster1 ~]# iptables -F
[root@cluster1 ~]# iptables-save > /etc/sysconfig/iptables
iptables-save
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [14:1480]
-A INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 22 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 7788:7799 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-host-prohibited
-A FORWARD -j REJECT –reject-with icmp-host-prohibited
-A OUTPUT -p tcp -m tcp –dport 7788:7799 -j ACCEPT
COMMIT
[root@cluster2 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe1b1fb03.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won’t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-6527, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527):
Using default value 6527
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@cluster2 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all — anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp — anywhere anywhere
ACCEPT all — anywhere anywhere
ACCEPT tcp — anywhere anywhere state NEW tcp dpt:ssh
REJECT all — anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all — anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@cluster2 ~]# /etc/init.d/iptables stop
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]
[root@cluster2 ~]# iptables -F
[root@cluster2 ~]# iptables-save > /etc/sysconfig/iptables
[root@cluster2 ~]#
Create mysql dir
[root@cluster1 ~]# mkdir /mysql
[root@cluster2 ~]# mkdir /mysql
Ntp update
[root@cluster1 ~]# chkconfig –level 345 ntpd on
[root@cluster1 ~]# /etc/init.d/ntpd restart
Shutting down ntpd: [FAILED]
Starting ntpd: [ OK ]
[root@cluster1 ~]# ntpdate -u 0.centos.pool.ntp.org
[root@cluster2 ~]# chkconfig –level 345 ntpd on
[root@cluster2 ~]# /etc/init.d/ntpd restart
Shutting down ntpd: [FAILED]
Starting ntpd: [ OK ]
[root@cluster2 ~]# ntpdate -u 0.centos.pool.ntp.org
1. Install dependencies: (Primary, Secondary)
# yum install gcc gcc-c++ make glibc flex kernel-devel kernel-headers
Install rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
Install DRBD on both nodes cluster1 and cluster2
yum -y install drbd83-utils kmod-drbd83
Insert drbd module manually on both machines or reboot:
/sbin/modprobe drbd
add the config
Create the Distributed Replicated Block Device resource file
/etc/drbd.d/clusterdb.res
resource clusterdb
{
startup {
wfc-timeout 30;
outdated-wfc-timeout 20;
degr-wfc-timeout 30;
}
net {
cram-hmac-alg sha1;
shared-secret sync_disk;
}
syncer {
rate 10M;
al-extents 257;
on-no-data-accessible io-error;
}
on cluster1.rmohan.com {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.60:7788;
flexible-meta-disk internal;
}
on cluster2.rmohan.com {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.1.62:7788;
meta-disk internal;
}
}
Initialize the DRBD meta data storage on both machines:
drbdadm create-md clusterdb
start drbd on both node
service drbd start
On the PRIMARY node run drbdadm command:
drbdsetup /dev/drbd0 primary –overwrite-data-of-peer
Every 2.0s: service drbd status Thu Nov 26 22:13:34 2015
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb SyncSource Primary/Secondary UpToDate/Inconsistent C
… sync’ed: 1.2% (50624/51196)M
watch service drbd status
[root@cluster1 ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb Connected Primary/Secondary UpToDate/UpToDate C
[root@cluster1 ~]#
/sbin/mkfs.ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3276800 inodes, 13106615 blocks
655330 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
root@cluster1 ~]# mount -t ext4 /dev/drbd0 /mysql/
[root@cluster1 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_cluster1-lv_root
ext4 53G 1.8G 49G 4% /
tmpfs tmpfs 3.0G 0 3.0G 0% /dev/shm
/dev/sda1 ext4 500M 57M 417M 13% /boot
/dev/mapper/vg_cluster1-lv_home
ext4 47G 55M 44G 1% /home
/dev/drbd0 ext4 53G 55M 50G 1% /mysql
[root@cluster1 ~]#
[root@cluster1 mysql]# date ; drbdadm role clusterdb
Fri Nov 27 13:28:57 SGT 2015
Primary/Secondary
[root@cluster2 ~]# date ; drbdadm role clusterdb
Fri Nov 27 13:29:02 SGT 2015
Secondary/Primary
verify replication is up to date
[root@cluster1 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r—–
ns:56 nr:0 dw:0 dr:1052 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Admin Commands:
[root@cluster1 ~]# drbd-overview
0:clusterdb Connected Primary/Secondary UpToDate/UpToDate C r—–
[root@cluster1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb Connected Primary/Secondary UpToDate/UpToDate C
BASIC MANUAL FAIL-OVER
# umount /dev/drbd0
# drbdadm secondary <resource>
Now on the node we wish to make primary promote the resource and mount the device.
# drbdadm primary <resource>
# mount /dev/drbd0/by-res/<resource> <mountpoint>
disconnecting disk:
drbdam disconnect clusterdb
drbdadm disconnect clusterdb
df -TH
umount /dev/drbd0
drbdadm secondary clusterdb
drbdadm connect clusterdb
drbdadm primary clusterdb
mount -t ext4 /dev/drbd0 /mysql/
df -TH
cd /mysql/
Fix DRBD recovery from split brain
root@cluster1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb StandAlone Primary/Unknown UpToDate/DUnknown r—–
[root@cluster2 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb StandAlone Secondary/Unknown UpToDate/DUnknown r—–
Solution:
Step 1: Start drbd manually on both nodes
Step 2: Define one node as secondary and discard data on this
drbdadm secondary all
drbdadm disconnect all
drbdadm — –discard-my-data connect all
Step 3: Define anoher node as primary and connect
drbdadm primary all
drbdadm disconnect all
drbdadm connect all
[root@cluster1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb Connected Primary/Secondary UpToDate/UpToDate C
[root@cluster2 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:clusterdb Connected Secondary/Primary UpToDate/UpToDate C
Recent Comments