July 2025
M T W T F S S
 123456
78910111213
14151617181920
21222324252627
28293031  

Categories

July 2025
M T W T F S S
 123456
78910111213
14151617181920
21222324252627
28293031  

Logical Volume Snapshots in CentOS 6

If it hasn’t happened to you yet, you’ll eventually experience a package update or system change that renders your server useless. You’ll then be suffer through multiple stages of fright, ranging from “Do we have a recent backup?” to “Where’s the documentation to rebuild this thing!?”

With the advent of logical volumes in Linux, we now have the ability to create snapshots of any our volumes before we make any changes to the system or the data. Snapshots allow us to rollback changes to a point in time before anything was done.

In this tutorial, I’ll show you how protect your system from bad patches or package installations. The method shown isn’t limited to system patching, it can be used for any use case where changes are made. For example, you could create snapshot of a volume hosting your users’ profiles before running scripts that modify them. If scripts causes a major problem, you can roll back.

Another great example for snapshots is backups. Although snapshots should never be considered a form of backup, they do allow you to create a consistent state of a volume which can be targeted by your backup software.

It’s important to remember that logical volume snapshots create crash consistent states of your data; lvm cannot create application consistent snapshots, therefore, If you are running database services, remember to stop them first. Failing to do so may result in corrupted tables.

Prerequisets for Logical Volume Snapshots

The following are required before you can use snapshots on a volume.

  • It must be a logical volume. Disk partitions cannot be snapshotted.
  • The volume group hosting the logical volume must have unused space available.
  • The amount of unused space must be enough to hold changes to the volumes data. If your change rate is 100MB a day and the snapshot is expected to be around for half a day, you’ll need at least 50MB.

System Configuration

To make this tutorial easier to follow along, our lab server had the following disk configuration.

Volume Group Device Size Unused Space
vg01 SDA2 120 GB 20 GB

The following logical volumes were created.

Log. Volume Volume Group Size Mount Point
swap vg01 2 GB
lv_root vg01 12 GB /
lv_var vg01 60 GB /var
lv_home vg01 26 GB /home

Creating a Snapshot

  1. Verify the amount of unused storage in the volume group. Replace vg01 with the name of your volume group.
    vgdisplay vg01
  2. The output should look similar to the one below. I’ve highlighted the area which details the unused space.
  3. Create a snapshot for the root volume, lv_root.
    lvcreate -s -n lv_root_snap -L 10G vg01/lv_root

    -s Signals to lvm that a snapshot is being created.
    -n Snapshot name
    -L Snapshot size, in kilobytes, megabytes, gigabytes, etc.
    vg01/lv_root Volume to be snapshotted.
  4. If successful, you will see the following output.

Modify Data on Logical Volume

With the snapshot created, we can now proceed with any planned modification to the volume. In this scenario, we’re going to install system and application patches.

  1. Install the latest system patches.
    yum update

Rollback Snapshot

Our applications stopped working after the update. Instead of spending time trying to isolate the problem and attempt to correct it, causing longer down-time for our applications, we’re simple going to rollback the snapshot to before the update.

  1. If possible, unmount the snapshotted volume. We created a snapshot of the root volume, so we won’t be able to.
  2. Rollback the snapshot to bring volume to a state before the updates were installed.
    lvconvert --merge /dev/vg01/lv_root_snap
  3. If the volume was still mounted, like ours was, you will get the following notification. Reboot the server to complete the rollback. Otherwise, if you can just mount the volume to see the rollback.
    Can't merge over open origin volume
    Merging of snapshot lv_root_snap will start next activation.

Mounting a Snapshot

Volume snapshots can also be mounted alongside their parent volumes. This allows us to do many things, such as browse the volume’s state before changes were made; modify the volume, with an ability to safely rollback our changes; or create a point-in-time copy of a volume for our backup software, so that our data remains consistent while users are logged in.

  1. Create a directory to mount our snapshot into.
    mkdir /snapshots
  2. Mount the snapshot.
    mount /dev/vg01/lv_root_snap /snapshots
  3. When done, unmount the snapshot.
    umount /snapshots

Deleting Snapshots

Finally, snapshot changes can be applied to their parent volume after changes are made, if the results of whatever modification you made are satisfactory. To do this, we simply delete the snapshot.

  1. Unmount the snapshot, if you have it mounted.
  2. Delete the snapshot.
    lvremove /dev/vg01/lv_root_snap
  3. When prompted, type ‘y’ and press Enter to confirm the deletion.
  4. You should see the following output if deletion was successfully completed.
    Do you really want to remove active logical volume lv_root_snap? [y/n]: y
    Logical volume "lv_root_snap" successfully removed
  5. Verify the snapshot was deleted.
    lvdisplay vg01/lv_root_snap
  6. You should get the following message if the snapshot was deleted.
    One or more specified logical volume(s) not found.

Kickstart CentOS 7: Making ISO for automated installation

Kickstart” is powerful tool for automated OS installation and configuration available in CentOS or RHEL. You can install CentOS without touching keyboard or mouse and even can automate post installation configuration process.

I’m recently building a Hadoop cluster for R&D, which makes me repeatedly install and re-install CentOS on several virtual machines. This is boring task takes hours. So I googled for automation and the answer was the Kickstart.

Basically a Kickstart is configuration file. CentOS load it at the beginning of installation process and apply it.

There are some ways to load the configuration file, through network, file system, or include installation ISO. I took the ISO way.

There is already good detail article about making kickstart-able ISO: “Building a custom CentOS 7 kickstart disc, part 1”, I followed this article with some modified steps.

Prepare directory and copy files from CentOS installation disk

$ mkdir ks_build
$ mkdir ks_build/isolinux
$ mkdir ks_build/isolinux/images
$ mkdir ks_build/isolinux/ks
$ mkdir ks_build/isolinux/LiveOS
$ mkdir ks_build/isolinux/Packages
$ mkdir ks_build/utils
$ cp /media/cdrom/CentOS7x86_64/isolinux/* ks_build/isolinux/
$ cp -R /media/cdrom/CentOS7x86_64/images/* ks_build/isolinux/images/
$ cp /media/cdrom/CentOS7x86_64/LiveOS/* ks_build/isolinux/LiveOS/
$ cp -p /media/cdrom/CentOS7x86_64/Packages/* ks_build/isolinux/Packages/
$ cp /media/cdrom/CentOS7x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz ks_build/comps.xml.gz
$ gunzip ks_build/comps.xml.gz

Create kickstart configuration file

I created two files. One for Hadoop master nodes and the other for slave nodes. The only difference is disk partition settings. I will only paste configuration for master nodes.

Meaning of each options are described in 23.3. Kickstart Syntax Reference, a Red Hat document. There is also an GUI tool named “system-config-kickstart” for editing a configuration file.

$ vi ks_master.cfg
#platform=x86, AMD64, or Intel EM64T
#version=DEVEL

# Install OS instead of upgrade
install
# Use CDROM installation media
cdrom
# System authorization information
auth  --enableshadow  --passalgo=sha512
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda
# Keyboard layouts
keyboard --vckeymap=jp106 --xlayouts='jp','us'
# System language
lang en_US.UTF-8
# Reboot after installation
reboot
# Root password
rootpw  --iscrypted $6$P3w8ID5.Zn3KObRb$eYGfYprCnPzwDF6WVPCLgx.tDFFo4/8prhovv.hEfdyfGRQ12MexcUwBESU3Hl2jS22KjA8si6.VdPChvnB4q1
# System language
timezone Asia/Tokyo --isUtc --nontp
# Firewall configuration
firewall --disabled
# Network information
network  --device=lo --hostname=localhost.localdomain
network --onboot=no --device=eth0 --bootproto=dhcp --noipv6
# Use graphical install
graphical
# SELinux configuration
selinux --disabled
# Do not configure the X Window System
skipx
# skip eula
eula --agreed

# System bootloader configuration
bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information
part /boot --asprimary --fstype="ext4" --size=500
part swap --asprimary --fstype="swap" --size=6144
part / --asprimary --fstype="ext4" --grow --size=1

# Repository
repo --name="CentOS"  --baseurl=cdrom:sr0 --cost=100
%packages
@core
@base
%end
$ cp ks_master.cfg ks_build/isolinux/ks/

Note, the “–onboot” option is set to “no”. Because I haven’t setup a DHCP server at this time. Without DHCP server, the instllation will stop at configuration screen.

network --onboot=no --device=eth0 --bootproto=dhcp --noipv6

Create repository

$ cd ~/ks_build/isolinux
$ createrepo -g ~/ks_build/comps.xml . 

Build ISO

$ cd ~/ks_build
$ chmod 664 isolinux/isolinux.bin
$ mkisofs -o custom.iso -b isolinux.bin -c boot.cat -no-emul-boot -V 'CentOS 7 x86_64' -boot-load-size 4 -boot-info-table -R -J -v -T isolinux/

Run kickstart

Boot from the ISO created by above command.

Press “ESC” at boot menu.

 

 

Type below command at boot prompt.

linux inst.ks=cdrom:/dev/cdrom:/ks/ks_master.cfg

 

 

IPPROCS or OPPROCS greater than 0 in a queue prevents normal termination of queue manager by “endmqm QmgrName” (Need to use: endmqm -i QmgrName)

Problem(Abstract)

You notice that when the WebSphere MQ (WMQ) queue attributes of IPPROCS or OPPROCS are greater than 0, and try to end the queue manager by issuing “endmqm QmgrName”, then the queue manager goes into “quiescing” but it does not fully terminate.

Is there a way to speed up the shutdown of the queue manager?

Resolving the problem

If “endmqm QMgrName” does not effectively end the queue manager, you can try to speed up the shutdown process by adding the flag -i such as:
endmqm -i QMgrName

Background:

The following page has more information about the IPPROCS and OPPROCS attributes for a queue.

Queue status attributes:

Attribute: Open input count
Meaning: This is the number of applications that are currently connected to the queue to get messages from the queue.
MQSC parameter: IPPROCS

Attribute: Open output count
Meaning: This is the number of applications that are currently connected to the queue to put messages on the queue.
MQSC parameter: OPPROCS

If “endmqm QMgrName” does not effectively end the queue manager, you can try to speed up the shutdown process by adding the flag -i such as:
endmqm -i QMgrName

For more details, see the product documentation:
endmqm

The default (endmqm QMgrName) is to use “-c” as in: endmqm -c QMgrName

-c Controlled (or quiesced) shutdown :
This is the default. The queue manager stops, but only after all applications have disconnected. Any MQI calls currently being processed are completed. Control is returned to you immediately and you are not notified of when the queue manager has stopped. The effect on any client applications connected through a server-connection channel is equivalent to a STOP CHANNEL command issued in QUIESCE mode.
>> Note: If an application has opened a queue for put or for get, but still is connected, then this will prevent the queue manager from ending. It will remain in “quiescing”.

-i Immediate shutdown:
The queue manager stops after it has completed all the MQI calls currently being processed. Any MQI requests issued after the command has been issued fail. Any incomplete units of work are rolled back when the queue manager is next started.
Control is returned after the queue manager has ended.
The effect on any client applications connected through a server-connection channel is equivalent to a STOP CHANNEL command issued in FORCE mode.
>> Note: If a client application has opened a queue and still is connected, such as the MQ Explorer, then this option will terminate the server-client connection, allowing the queue manager to terminate.

-p Preemptive shutdown:
WARNING: !!!! Use this type of shutdown only in exceptional circumstances. For example, when a queue manager does not stop as a result of a normal endmqm command. !!!

The queue manager might stop without waiting for applications to disconnect or for MQI calls to complete. This can give unpredictable results for WebSphere MQ applications.
The shutdown mode is set to immediate shutdown. If the queue manager has not stopped after a few seconds, the shutdown mode is escalated, and all remaining queue manager processes are stopped.
The effect on any client applications connected through a server-connection channel is equivalent to a STOP CHANNEL command issued in TERMINATE mode.

+++ Scenario

The impact that these attributes have whenever they have a value greater than Zero, when trying to terminate the queue manager is best described by a simple scenario, which is described in the rest of this technote.

A queue Q1 in the queue manager QM_VER has not been opened by any application. Thus, IPPROCS and OPPROCS are 0:
In one window enter:

$ runmqsc QM_VER
display qstatus(Q1)
     2 : display qstatus(Q1)
AMQ8450: Display queue status details.
   QUEUE(Q1)                               TYPE(QUEUE)
   CURDEPTH(0)                             IPPROCS(0)
   MSGAGE( )                               OPPROCS(0)

In another window, the MQ sample command “amqsput” is issued to put a message:

$ amqsput Q1 QM_VER
Sample AMQSPUT0 start
target queue is Q1

Notice that the text of the message has not being entered yet. The application is just waiting for the user to enter the text. The application has already opened the queue and thus, the counter for OPPROCS has been incremented from 0 to 1:

$ runmqsc QM_VER
display qstatus(Q1)
     3 : display qstatus(Q1)
AMQ8450: Display queue status details.
   QUEUE(Q1)                               TYPE(QUEUE)
   CURDEPTH(0)                             IPPROCS(0)
   MSGAGE( )                               OPPROCS(1)

In yet another window, the MQ sample command “amqsget” is issued to get a message:

$ amqsget Q1 QM_VER
Sample AMQSGET0 start

Notice that now the counter for IPPROCS has been incremented from 0 to 1, and that OPPROCS is still at 1 because amqsput is still connected:

$ runmqsc QM_VER
display qstatus(Q1)
     4 : display qstatus(Q1)
AMQ8450: Display queue status details.
   QUEUE(Q1)                               TYPE(QUEUE)
   CURDEPTH(0)                             IPPROCS(1)
   MSGAGE( )                               OPPROCS(1)

But 30 seconds later, amqsget terminates because no messages arrived within 30 seconds, and now the IPPROCS counter decreased from 1 to 0. But amqsput is still connected, waiting for the user to enter a message and thus OPPROCS is still 1.

$ runmqsc QM_VER
display qstatus(Q1)
     3 : display qstatus(Q1)
AMQ8450: Display queue status details.
   QUEUE(Q1)                               TYPE(QUEUE)
   CURDEPTH(0)                             IPPROCS(0)
   MSGAGE( )                               OPPROCS(1)

Let’s assume that we wanted to terminate the queue manager, keeping in mind that amqsput is still running and connected to the queue (OPPROCS is still 1):

$ endmqm QM_VER
Quiesce request accepted. The queue manager will stop when all outstanding work
is complete.

Notice that we cannot make new connections, not even running runmqsc:

$ runmqsc QM_VER
AMQ8156: WebSphere MQ queue manager quiescing.

$ amqsget Q1 QM_VER
Sample AMQSGET0 start
MQCONN ended with reason code 2161

$ mqrc 2161
      2161  0x00000871  MQRC_Q_MGR_QUIESCING

The monitoring of the MQ processes for this queue manager do not show any changes and the processes are still running.

$ ps -ef | grep -i QM_VER
mqm       1869     1  0 13:28 ?        00:00:00 amqzxma0 -m QM_VER
mqm       1874  1869  0 13:28 ?        00:00:00 /opt/mqm/bin/amqzfuma -m QM_VER
mqm       1878  1869  0 13:28 ?        00:00:00 amqzmuc0 -m QM_VER
mqm       1884  1869  0 13:28 ?        00:00:00 amqzmur0 -m QM_VER
mqm       1885  1869  0 13:28 ?        00:00:00 amqzmuf0 -m QM_VER
mqm       1888  1869  0 13:28 ?        00:00:00 /opt/mqm/bin/amqrrmfa -m QM_VER -t2332800 -s2592000 -p2592000 -g5184000 -c3600
mqm       1889  1869  0 13:28 ?        00:00:00 /opt/mqm/bin/amqzdmaa -m QM_VER
mqm       1914  1869  0 13:28 ?        00:00:00 /opt/mqm/bin/amqzmgr0 -m QM_VER
mqm       1928  1869  0 13:28 ?        00:00:00 amqzlaa0 -mQM_VER -fip0
rivera    2484 19416  0 13:28 pts/3    00:00:00 amqsput Q1 QM_VER
rivera    3605 19796  0 13:30 pts/4    00:00:00 grep -i QM_VER

Because an amqsput connection is preventing the queue manager from terminating, only until amqsput terminates, the queue manager will end.

One easy solution is to manually terminate amqsput, which will disconnect from the queue (OPPROCS will be 0) and the queue manager will terminate.

However, sometimes, it is not easy to identify which is the particular application that is preventing the shutdown, then it is OK to escalate the shutdown of the queue manager by issuing “-i” (for immediate).
Thus, in the same window or in another window, issue the following command, even though an “endmqm QMgrName” was issued already (but it is quiescing).

$ endmqm -i QM_VER
WebSphere MQ queue manager 'QM_VER' ending.
WebSphere MQ queue manager 'QM_VER' ended.

The “-i” flag will wait for in transit activities to terminate, but because there are none at this time, then it will terminate the connection of amqsput and then continue successfully to terminate the queue manager.

Notice that the pending amqsput is terminated with rc 2009:

$ amqsput Q1 QM_VER
Sample AMQSPUT0 start
target queue is Q1
MQCLOSE ended with reason code 2009
Sample AMQSPUT0 end

Notice that the rc 2009 means:
$ mqrc 2009
     2009  0x000007d9  MQRC_CONNECTION_BROKEN

Product Alias/Synonym

WebSphere MQ WMQ

KVM: How to resize (grow) an ext4 LVM VM

We use LVM both on the guest and the CentOS KVM host. Let’s upgrade from 5 GB to 15 GB on the guest.

1. Shutdown the VM.

2. On the KVM host, take a backup of the VM LV in case something goes wrong.

[root@host ~]# dd if=/dev/vg_host01/lv_guest01-vm of=./lv_guest01-vm-5gb

3. On the KVM host, resize the VM LV.

Add 1 GB:

[root@host ~]# lvresize -L+1G /dev/vg_host01/lv_guest01-vm

Or add all available space from the VG:

[root@host ~]# lvresize -l+100%FREE /dev/vg_host01/lv_guest01-vm

4. Boot up the VM again.

5. Delete and recreate the LVM partition.

root@guest:~$ parted /dev/vda
(parted) unit s
(parted) print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 30720000s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 2048s 1026047s 1024000s primary ext4 boot
2 1026048s 10239999s 9213952s primary lvm

(parted) rm 2
Warning: WARNING: the kernel failed to re-read the partition table on /dev/vda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.

Exit parted. You can also use fdisk to delete the partition, but when recreating the partition, fdisk doesn’t know about the right first and last cylinders. Maybe exiting and running fdisk again solves that (forgot to check that, we’ll try that next time) – UPDATE: on another try with fdisk it worked, no need for parted. You can do everything with fdisk in one run, just don’t exit it and use the start cylinder from the deleted partition. It will know the end cylinder automatically. Ok, so we deleted the partition using parted and now we start fdisk again to recreate the partition. Also, for some reason, the first partition (1) is now just half in size, 512 MB instead of 1024 MB. Weird. parted messed something up there?

root@guest:~$ fdisk /dev/vda

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).

Command (m for help): p

Disk /dev/vda: 15.7 GB, 15728640000 bytes
222 heads, 30 sectors/track, 4612 cylinders
Units = cylinders of 6660 * 512 = 3409920 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000831eb

Device Boot Start End Blocks Id System
/dev/vda1 * 1 155 512000 83 Linux

Command (m for help): n
Command action
e extended
p primary partition (1-4) p
Partition number (1-4): 2
First cylinder (155-4612, default 155):
Using default value 155
Last cylinder, +cylinders or +size{K,M,G} (155-4612, default 4612):
Using default value 4612

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/vda: 15.7 GB, 15728640000 bytes
222 heads, 30 sectors/track, 4612 cylinders
Units = cylinders of 6660 * 512 = 3409920 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000831eb

Device Boot Start End Blocks Id System
/dev/vda1 * 1 155 512000 83 Linux
/dev/vda2 155 4612 14844936 8e Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

6. Reboot the VM.

7. Resize the PV.

Let’s look at the PV first:

root@guest:~$ pvdisplay
— Physical volume —
PV Name /dev/vda2
VG Name vg_guest01
PV Size 4.39 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 1124
Free PE 0
Allocated PE 1124
PV UUID IzkvJd-G9JW-Ju7r-1bDK-FLzC-I9y1-Ll3N4R

Still 5 GB. Let’s resize it:

root@guest:~$ pvresize /dev/vda2
Failed to read physical volume “/dev/vda2”
0 physical volume(s) resized / 0 physical volume(s) not resized

Don’t worry about the “not resized”. It worked:

root@guest:~$ pvdisplay
— Physical volume —
PV Name /dev/vda2
VG Name vg_guest01
PV Size 14.16 GiB / not usable 1016.00 KiB
Allocatable yes
PE Size 4.00 MiB
Total PE 3624
Free PE 2500
Allocated PE 1124
PV UUID IzkvJd-G9JW-Ju7r-1bDK-FLzC-I9y1-Ll3N4R

8. Resize the LV.

root@guest:~$ lvresize -l+100%FREE /dev/vg_guest01/lv_root

Check it:

root@guest:~$ lvdisplay /dev/vg_guest01/lv_root
— Logical volume —
LV Path /dev/vg_guest01/lv_root
LV Name lv_root
VG Name vg_guest01
LV UUID 3Qfe2E-5j5H-H8el-iwxt-oN81-DvYq-AzENqH
LV Write Access read/write
LV Creation host, time guest01, 2013-03-02 00:41:16 +0100
LV Status available
# open 1
LV Size 14.03 GiB
Current LE 3592
Segments 2
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:0

9. Resize the ext4 FS.

root@guest:~$ resize2fs /dev/vg_guest01/lv_root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg_guest01/lv_root is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/vg_guest01/lv_root to 3678208 (4k) blocks.
The filesystem on /dev/vg_guest01/lv_root is now 3678208 blocks long.

10. Reboot the VM.

11. Done.

root@guest:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_guest01-lv_root
14G 1.2G 12G 9% /
tmpfs 939M 0 939M 0% /dev/shm
/dev/vda1 485M 53M 408M 12% /boot

Quick copy & paste how-to: Install KVM on CentOS 6

CentOS 6.7 64bit
ISO: ftp://ftp.hosteurope.de/mirror/centos.org/6.7/isos/x86_64/CentOS-6.7-x86_64-netinstall.iso

URL for netinstall: ftp://ftp.hosteurope.de/mirror/centos.org/6.7/os/x86_64

LVM vg: 100%

Host: lv_root / 200 GB

vi /etc/sysconfig/selinux
SELINUX=disabled

yum install wget screen lynx vim kvm libvirt python-virtinst qemu-kvm ncftp tigervnc tigervnc-server system-config-firewall-tui xterm twm bridge-utils virt-manager openssh-askpass openssh-clients dbus-python dbus

yum groupinstall Desktop

yum update

/etc/init.d/messagebus start

system-config-firewall-tui
-> disable firewall

vi /etc/sysconfig/vncservers
VNCSERVERS=”10:root”
VNCSERVERARGS[10]=”-geometry 1024×768″

vi /etc/ssh/sshd_config
Port 450
PermitRootLogin without-password

adduser admin
passwd admin

chkconfig rpcidmapd off
chkconfig rpcgssd off
chkconfig nfslock off
chkconfig netfs off
chkconfig rpcbind off
chkconfig dnsmasq off
chkconfig NetworkManager off
chkconfig messagebus off

vncpasswd

vi /root/.vnc/xstartup

#!/bin/sh
[ -r /etc/sysconfig/i18n ] && . /etc/sysconfig/i18n
export LANG
export SYSFONT
vncconfig -iconic &
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
OS=`uname -s`
if [ $OS = ‘Linux’ ]; then
case “$WINDOWMANAGER” in
*gnome*)
if [ -e /etc/SuSE-release ]; then
PATH=$PATH:/opt/gnome/bin
export PATH
fi
;;
esac
fi
if [ -x /etc/X11/xinit/xinitrc ]; then
exec /etc/X11/xinit/xinitrc
fi
if [ -f /etc/X11/xinit/xinitrc ]; then
exec sh /etc/X11/xinit/xinitrc
fi
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
xterm -geometry 80×24+10+10 -ls -title “$VNCDESKTOP Desktop” &
twm &

chmod 755 /root/.vnc/xstartup

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
# BROADCAST=x.x.x.x
# DNS1=x.x.x.x
# GATEWAY=x.x.x.x
HWADDR=00:25:90:38:66:c6
# IPADDR=x.x.x.x
# NETMASK=x.x.x.x
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
# DNS2=x.x.x.x
IPV6INIT=no
USERCTL=no
BRIDGE=br0

vi /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no
BROADCAST=x.x.x.x
DNS1=x.x.x.x
GATEWAY=x.x.x.x
IPADDR=x.x.x.x
NETMASK=x.x.x.x
DNS2=x.x.x.x
IPV6INIT=no
USERCTL=no

vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

In /etc/postfix/main.cf:
inet_protocols = ipv4

reboot

Add hostname to /etc/hosts

/etc/init.d/vncserver start

VNC connect to host:10

virt-manager:

localhost ->

1. Virtual Networks -> default -> Basic details: device: (or virbr0), Autostart: unchecked -> Apply

2. Storage -> + (add) -> logical, Name “vm-lvm” -> Build Pool: unchecked -> Source Name: name of existing LVM volume group -> Finish

Done!

CentOS & yum-cron: Automatic reboot after kernel update

If you’re using yum-cron and want to automatically reboot your CentOS box whenever the kernel gets updated you can add this code to /etc/cron.daily/0yum.cron right before exit 0 at the end:

entry=`cat /boot/grub/grub.conf | grep '^default' | cut -d '=' -f2`
entry=`expr $entry + 1`
if [ "`cat /boot/grub/grub.conf | grep '^title' | tail -n +$entry | head -1 | sed -e 's/.*(\(.*\)).*/\1/'`" != "`uname -r`" ]; then
sleep 10 ; reboot
fi

This was taken from here with a tiny correction displayed in bold above.

Depending on your version of CentOS you may want to adjust /etc/anacrontab or /etc/crontab to set the times when cron.daily and therefore yum-cron will be run to avoid reboots due to a kernel updates in the middle of the day.

Backup storage: CentOS & QNAP & iSCSI

I’m using a CentOS 6 box as backup server running BackupPC. Until a couple days ago I had a Thecus N7700PRO NAS with 4x 3 TB discs configured as RAID5 which was accessed via iSCSI from the CentOS box. Then, 2 harddrives died at the same time (or not, at least that’s what the Thecus reported at that point) and Thecus support said the system cannot see drive 1 (one of the apparently failed ones) anymore, although the Thecus still showed it, and I should try to install a new harddrive in place of drive 1 and try to duplicate all data from drive 2 to the new drive 1 using “dd”. When I rebooted the Thecus hoping that it might just magically work again, the whole RAID5 was gone. Like it never existed. I said screw it and bought a QNAP 19? 1U TS-412U off of eBay (new) for about 580 EUR. Along that four 4 TB enterprise discs from different vendors according to the supported harddrive compatibility list from QNAP. Here are the required steps to get the backup server back in business:

1. Insert harddrives, power on, initial setup, QNAP will get a IP via DHCP instead of the 169.254.100.100 that’s mentioned in the quick start guide.

2. Download latest firmware from here and upload when prompted to.

3. Set up RAID5 or whatever you prefer. You can only choose ext3 or ext4 but don’t get confused by it, it’s just the lower level that QNAP uses and on top we will later build our own XFS filesystem and use LVM.

4. Configured iSCSI target & LUN according to this QNAP link (sorry, in German only, but the pictures should be sufficient to figure it out), but I chose Instant Allocation. You may have to wait for the RAID5 to get built before you can choose the LUN location. Also, after you created the target it may take a while before the target becomes available (you can check the progress under “iSCSI Target List” – “Alias” – “id:0 …” – “Status”).

5. The QNAP is directly connected to eth1 on the CentOS box without a switch. eth1 has IP 192.168.1.1, the QNAP has 192.168.1.100.

6. On the CentOS box, delete the old Thecus from the iSCSI initiator database:
iscsiadm -m node -o delete

7. Make sure node startup is set to automatic in /etc/iscsi/iscsid.conf:
node.startup = automatic

8. Discover the new QNAP:
iscsiadm -m discovery -t sendtargets -p 192.168.1.100:3260

9. Make sure it’s there:
iscsiadm -m node
This should output something like:
192.168.1.100:3260,1 iqn.2004-04.com.qnap:ts-412u:iscsi.raid5.c8af3e

10. If you want, reboot the box and confirm that it’s still there when you execute iscsiadm -m node after the reboot.

11. In dmesg something like this should have popped up now:
scsi5 : iSCSI Initiator over TCP/IP
scsi 5:0:0:0: Direct-Access QNAP iSCSI Storage 3.1 PQ: 0 ANSI: 5
sd 5:0:0:0: Attached scsi generic sg1 type 0
sd 5:0:0:0: [sdb] 23017373696 512-byte logical blocks: (11.7 TB/10.7 TiB)
sd 5:0:0:0: [sdb] Write Protect is off
sd 5:0:0:0: [sdb] Mode Sense: 2f 00 00 00
sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
sdb: unknown partition table
sd 5:0:0:0: [sdb] Attached SCSI disk

12. I like LVM, it’s not necessary, but maybe it will be of use later on. Check with pvdisplay that /dev/sdb is there (maybe reboot or run pvscan if it’s not):
— Physical volume —
PV Name /dev/sdb

13. I created PV, VG and LV and assigned 100% of the available space to the LV:
pvcreate /dev/sdb
vgcreate data /dev/sdb
lvcreate --name rz-nas01 -l100%FREE data

14. Now, you should have /dev/mapper/data-rz--nas01 or /dev/data/rz-nas01 which are just links to a /dev/dm-x device. If you don’t, you can try restarting /etc/init.d/lvm2-monitor or just reboot. Run lvdisplay to check the LV is there and “LV Status” is “available”.

15. Create a filesystem on the LV, I chose XFS:
mkfs.xfs /dev/mapper/data-rz--nas01
This could take a few moments.

16. If you want the storage to get mounted automatically on boot, use something like this in /etc/fstab:
/dev/mapper/data-rz--nas01 /var/lib/BackupPC xfs defaults,_netdev 0 0

17. Mount it with mount /dev/mapper/data-rz--nas01 /mnt if you want to test first, otherwise you can just do mount -a.

18. Done!

CentOS: Fix broken yum repo metadata

Happened because I added EPEL, I believe. Very strange, on another, identical machine (steps 1:1) it worked fine, but on this box suddenly this appeared when trying to yum install something:

[...]
--> Processing Dependency: libnuma.so.1()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64
--> Processing Dependency: libnl.so.1()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64
--> Processing Dependency: libnetcf.so.1()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64
--> Processing Dependency: libgnutls.so.26()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64
--> Processing Dependency: libavahi-common.so.3()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64
--> Processing Dependency: libavahi-client.so.3()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64
---> Package qemu-kvm.x86_64 2:0.12.1.2-2.415.el6_5.10 will be installed
http://mirror2.hs-esslingen.de/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://centos.mirror.sharkservers.co.uk/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://ftp.plusline.de/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://ftp.hosteurope.de/mirror/centos.org/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://mirror.netcologne.de/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://centos.intergenia.de/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://centos.bio.lmu.de/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://centos.psw.net/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://mirror.maeh.org/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
ftp://mirror.fraunhofer.de/centos.org/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 19 - "Given file does not exist"
Trying other mirror.
Error: failure: repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2 from updates: [Errno 256] No more mirrors to try.
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

Fixed by googling and running this:

yum clean metadata
yum clean dbcache
yum update

No MySQL root access when installing Galera from binaries on CentOS 7

EDIT: I AM the one who is stupid! Arrrrgh. Just installed Galera on yet another box and saw this passing by in yum:

A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.

Arrrrghhhh. Apologies to the RPM packagers. EDIT END.

Stupid RPM packagers screwed up. If you follow this how-to – http://galeracluster.com/documentation-webpages/gettingstarted.html – after a while you end up at a point where it says “In the database client, run the following query: SHOW STATUS LIKE 'wsrep_cluster_size'; …so you try “mysql -p” only to find that you don’t have access. WTF. There’s probably already a password set by the RPM packagers but we don’t know it. So, we try our usual –skip-grant-tables thing and then try SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass'); but this will result in: “ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables option so it cannot execute this statement” – WTF again. When you are used to resetting MySQL root pw’s you usually first run the PW change command (SET or ALTER or UPDATE or whatever) and then you enter FLUSH PRIVILEGES. The trick here is to do the opposite. First enter

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass');
Query OK, 0 rows affected (0.00 sec)

Found this here after hours of troubleshooting: http://galeracluster.com/community/?place=msg%2Fcodership-team%2Fw8NEekKipwY%2FgGlkSQNOedMJ
F*CKERS!

Disable IPv6 on Linux (CentOS)

Disable IPv6 on Linux (CentOS)

Just put this into /etc/sysctl.conf. No need to mess with the IPv6 modules.

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

And also inet_protocols = ipv4 in /etc/postfix/main.cf

Why would you want to do that? I have a rogue IPv6 router advertising going on in my network that I can’t block. That results in Linux boxes assigning themselves an IPv6 address from that “foreign” router. This can cause trouble when sending eMail. For example DirectAdmins exim tries to send out via IPv6 by default and if there’s no PTR set for that IP eMails might not get delivered: “550-inconsistent or no DNS PTR record for 2a01:…”. The true solution is to set a PTR, remove the rogue RA or properly enable IPv6 on own network equipment with PTRs, so this is just a dirty workaround.

PS: If postfix doesn’t start up after you disabled IPv6 and complains with the error message “fatal: parameter inet_interfaces: no local interface found for ::1” then you could remove ::1 from /etc/hosts and postfix will run again.