August 2025
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

Categories

August 2025
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

Resizing an Online Multipath Device

Resizing an Online Multipath Device

If you need to resize an online multipath device, use the following procedure:

1. Resize your physical device

2. Use the following command to find the paths to the LUN:

# multipath -l
3. Resize your paths. FOr SCSI devices, writing a 1 to the rescan file for the device causes the SCSI driver to rescan:

# echo 1 > /sys/block/device_name/device/rescan
4. Resize your multipath device by running the multipathd resize command:

# multipath -k’reisze map mpathX’
5. Resize the file system (assuming no LVM or DOS partition are used):

# resize2fs /dev/mapper/mpathX

How to share ssh command line with someone via screen

Use screen to share an ssh session
It’s quite easy to share an ssh session with someone remote so that they can view the ssh session. This process works well if everyone involved has root access. Read further below if this is not the case. To begin, screen must be installed to create the screen. Once installed, create the screen as follows:

screen -L -S

Provide the session name to the person you wish to view the ssh session, and they can join this screen as follows:

screen -x

After the session is over, a .screenlog log is written, containing all the commands that were issued during the session.

For those limited to non-root, multiuser screen may be a solution. It is well documented here:
http://aperiodic.net/screen/multiuser

Other tips I’ve found include using the script command. I personally have not tried this one myself, but it looks like it should work depending on your use case:

$ script -t 1 /tmp/screencast

For those who wish to look on:

$ tail -f /tmp/screencast

The file /boot/grub/stage1 not read correctly

The file /boot/grub/stage1 not read correctly
After pxe booting a server and installing linux from a cpio archive, grub failed to load using the grub-install command. The error presented was as follows:

root@pxe:~# mkdir /target
root@pxe:~# mount /dev/sda1 /target
root@pxe:~# chroot /target
root@pxe:~# grub-install /dev/sda
The file /boot/grub/stage1 not read correctly.
Since command line didn’t work, I tried installing grub via the grub shell:

root@pxe:~# grub

GNU GRUB version 0.97 (640K lower / 3072K upper memory)

[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename.]
grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
Checking if “/boot/grub/stage1” exists… no
Checking if “/grub/stage1” exists… no

Error 2: Bad file or directory type
grub>
This too failed, with a different error message. You may have noticed the first error message given while attempting to install grub with the grub-install command yielded:

The file /boot/grub/stage1 not read correctly

While attempting to install grub via the grub shell yielded:

Error 2: Bad file or directory type

After some research, grub only works on an inode size of 128, and the pxe system formatted the file system with an inode size of 256.

root@pxe:~# tune2fs -l /dev/sda1 | grep -i ‘inode size’
Inode size: 256
The system being reimaged was an older legacy system, and the pxe system had been upgraded months prior. Having remembered this, I compared versions of mke2fs and found that the upgraded version creates 256-byte inodes by default, where the older version was 128. To correct this, I updated the pxe imaging scripts responsible for formatting the disks with the -I option as follows:

root@pxe:~# mke2fs -L / -I 128 -F -j -O dir_index /dev/sda1
Once recreating the file system with a defined inode size of 128, the grub-install command worked successfully.

root@pxe:~# tune2fs -l /dev/sda1 | grep -i ‘inode size’
Inode size: 128

How To Resize an iscsi LUN in Linux

Article ID: 83
by: Reese K.
Posted: 07 Jan, 2013
Last updated: 13 Mar, 2014
Views: 4849

How To Resize an iscsi LUN in Linux
NetApp storage systems also allow you to resize a LUN dynamically; however, the iSCSI layer in Linux is not capable of detecting the change in the LUN size. To display the new size of the LUN you must restart the iSCSI service. But a restart of the iSCSI service fails if any iSCSI devices are mounted at that time. Because of this, even if the LUN on which the file system is located is resized, you must unmount the file system and then restart iSCSI and the multipathing service before Linux can detect the new size.

Though this procedure below has been tested, it is always a good idea to back up your data before attempting resizing.

Collect some information on the partition to be resized:

For the purposes of this example, /dev/sdc1 is the iscsi lun partition to be resized. Run through the following commands and set the information aside:

~# df /dev/sdc1
~# df -h /dev/sdc1
~# fdisk -l /dev/sdc
~# fdisk -s /dev/sdc1

The first two commands are to provide a “snapshot” look at the size of the partition before making any changes. The third command contains useful information needed to successfully increase the partition in a later step. Make note of the starting cylinder from the output. For example:
PRODUCTION linux01 # fdisk -l /dev/sdc

Disk /dev/sdc: 32.2 GB, 32212254720 bytes
64 heads, 32 sectors/track, 30720 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 30720 31457264 83 Linux
PRODUCTION linux01 #
Note the amount of cylinders and what the starting cylinder is. In the example above, it is highlighted in yellow and the starting cylinder is 1.

If the partition to be increased contains system/OS files, you may need to boot off a rescue disk to perform the resize operation. Otherwise, it the parition is a data partition, then it simply needs to be unmounted.

Remember to increase the size of the LUN on the NetApp first before proceeding
netapp> lun resize /vol/mylun +5g

PRODUCTION linux01 # umount /dev/sdc1
PRODUCTION linux01 # fsck -n /dev/sdc1
PRODUCTION linux01 # /etc/init.d/iscsid restart
* Unmounting iscsi partition… …
* Logging out of iSCSI targets
* Logging out of session [sid: 1, target: iqn.1992-08.com.netapp:sn.151753609, portal: 10.10.2.2,3260]
* Logout of [sid: 1, target: iqn.1992-08.com.netapp:sn.151753609, portal: 10.10.2.2,3260]: successful
* Stopping iSCSI initiator service … [ ok ]
* Removing iSCSI modules … [ ok ]
* Loading iSCSI modules … [ ok ]
* Starting iSCSI initiator service … [ ok ]
* Logging into iSCSI targets
* Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.151753609, portal: 10.10.2.2,3260]
* Login to [iface: default, target: iqn.1992-08.com.netapp:sn.151753609, portal: 10.10.2.2,3260]: successful
* Waiting for udev to process new devices
* Mounting iscsi partition… …

Note: From the ‘fdisk -l’ output below, the device changed from /dev/sdc to /dev/sdb. This may happen due to udev during iscsi restart and is normal. You will also notice the amount of cylinders has changed (highlighted in blue) due to the lun size increase command performed on the netapp but the cylidner start/end is the same as before.

PRODUCTION linux01 # fdisk -l

Disk /dev/sdb: 107.3 GB, 107374182400 bytes
64 heads, 32 sectors/track, 102400 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 30720 31457264 83 Linux
PRODUCTION linux01 #

Run a file system check to ensure the file system is clean before attempting to change the layout of the partition.

PRODUCTION linux01 # fsck -n /dev/sdb1
fsck 1.38 (30-Jun-2005)
e2fsck 1.38 (30-Jun-2005)
/dev/sdb1 has gone 701 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found. Create? no

Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb1: ********** WARNING: Filesystem still has errors **********

/dev/sdb1: 16940/3932160 files (0.3% non-contiguous), 7425413/7864316 blocks
It still has errors because lost+found directory is missing in MY example, but I opted not to create it in this step to avoid further changes to the file system. Next, convert the file system into an ext2 file system if needed. The reason is because depending on your kernel, resize2fs may only work on ext2 file systems. Refer to your man page In my cases, it doesn’t support ext3 so remove the journal, delete the current partition table, and recreate it with the new cylinder boundaries.

PRODUCTION linux01 # tune2fs -O ^has_journal /dev/sdb1
tune2fs 1.38 (30-Jun-2005)
PRODUCTION linux01 # fdisk /dev/sdb

The number of cylinders for this disk is set to 102400.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 107.3 GB, 107374182400 bytes
64 heads, 32 sectors/track, 102400 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 30720 31457264 83 Linux

Command (m for help): d
Selected partition 1

Command (m for help): p

Disk /dev/sdb: 107.3 GB, 107374182400 bytes
64 heads, 32 sectors/track, 102400 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-102400, default 1): 1 <-taken from fdisk -l output in preliminary steps. make sure start cylinder is same as it was before!!! If you get an error like "Value out of range", you may need to use parted instead of fdisk. See footnotes below Last cylinder or +size or +sizeM or +sizeK (1-102400, default 102400): Using default value 102400 Command (m for help): p Disk /dev/sdb: 107.3 GB, 107374182400 bytes 64 heads, 32 sectors/track, 102400 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 102400 104857584 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. Run another file system check. Note this time we use e2fsck because we're still an ext2 file system. This time, I created the lost+found, resize the partition, and run another check to ensure it is clean. PRODUCTION linux01 # e2fsck -f /dev/sdb1 e2fsck 1.38 (30-Jun-2005) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity /lost+found not found. Create? yes

Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 16941/3932160 files (0.3% non-contiguous), 7392612/7864316 blocks
PRODUCTION linux01 #
PRODUCTION linux01 # resize2fs /dev/sdb1
resize2fs 1.38 (30-Jun-2005)
Resizing the filesystem on /dev/sdb1 to 26214396 (4k) blocks.
The filesystem on /dev/sdb1 is now 26214396 blocks long.

PRODUCTION linux01 # fsck -n /dev/sdb1
fsck 1.38 (30-Jun-2005)
e2fsck 1.38 (30-Jun-2005)
/dev/sdb1: clean, 16941/13107200 files, 7680539/26214396 blocks

The resize operation is now complete. Add an ext3 journal to the filesystem and remount the lun. Enjoy the new size!

PRODUCTION linux01 # tune2fs -j /dev/sdb1
tune2fs 1.38 (30-Jun-2005)
Creating journal inode: done
This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
PRODUCTION linux01 #
PRODUCTION linux01 # mount /dev/sdb1 /mnt
PRODUCTION linux01 # df -h /dev/sdb1
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 99G 28G 67G 30% /var/services/db/mysql
PRODUCTION linux01 #
PRODUCTION linux01 # ls -l /mnt
total 28853216
drwx—— 2 mysql mysql 4096 Jul 30 22:47 database22_100
drwx—— 2 mysql mysql 4096 Jul 30 22:47 database22_101
drwx—— 2 mysql users 4096 Jul 30 22:47 database22_102
drwx—— 2 mysql users 4096 Jul 30 22:47 database22_103
drwx—— 2 mysql users 4096 Jul 30 22:47 database22_104
drwx—— 2 mysql users 4096 Jul 30 22:47 database22_105
drwx—— 2 mysql users 4096 Jul 30 22:47 database22_106
drwx—— 2 mysql users 4096 Jul 30 22:47 database22_107


Footnotes

In this example, fdisk supplied an error “Value out of range” when trying to set a starting cyclinger to a value less than 2048. I had an incident where a LUNs starting cyclinder was 63, and since I did not want to lose the data held between data 63 and 2048, I used parted to repartition the disk. Having a starting cyclinder of 63 more than likely means not be aligned with the underlying storage, but that’s another topic. There may be better ways to do this, but this is how I’ve worked it out and am documenting it until I have the opportunity to run through the process again. If improvements can be made, I’ll update the doc or will review anyone who cares to comment.

PRODUCTION linux101 ~ # parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) print
Model: NETAPP LUN (scsi)
Disk /dev/sdb: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 107GB 107GB primary
(parted) unit cyl
(parted) print
Model: NETAPP LUN (scsi)
Disk /dev/sdb: 13055cyl
Sector size (logical/physical): 512B/512B
BIOS cylinder,head,sector geometry: 13055,255,63. Each cylinder is 8225kB.
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 0cyl 13054cyl 13054cyl primary
(parted) unit s (you might have to type ‘unit s’)
(parted) print
Model: NETAPP LUN (scsi)
Disk /dev/sdb: 209743872s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 2048s 209717247s 209715200s primary
(parted) rm 1
(parted) print
Model: NETAPP LUN (scsi)
Disk /dev/sdb: 209743872s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
(parted) mkpart
Partition type? primary/extended? primary
File system type? [ext2]? ext3
Start? 63s
End? 209717247s <<< use value from output in print above in matching highlight (see footnote below) Warning: The resulting partition is not properly aligned for best performance. Ignore/Cancel? Ignore (parted) print Model: NETAPP LUN (scsi) Disk /dev/sdb: 209743872s Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 63s 209717247s 209717185s primary ext2 (parted) quit Information: You may need to update /etc/fstab. PRODUCTION atl4mysqlv101 ~ # fdisk -l Disk /dev/sda: 292.3 GB, 292326211584 bytes 255 heads, 63 sectors/track, 35539 cylinders, total 570949632 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x77e3ed41 Device Boot Start End Blocks Id System /dev/sda1 63 8016434 4008186 83 Linux /dev/sda2 8016435 16032869 4008217+ 83 Linux /dev/sda3 16032870 20049119 2008125 82 Linux swap / Solaris /dev/sda4 20049120 570934034 275442457+ 5 Extended /dev/sda5 20049183 36065924 8008371 83 Linux /dev/sda6 36065988 52082729 8008371 83 Linux /dev/sda7 52082793 570934034 259425621 83 Linux Disk /dev/sdb: 107.4 GB, 107388862464 bytes 255 heads, 63 sectors/track, 13055 cylinders, total 209743872 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0005ed03 Device Boot Start End Blocks Id System /dev/sdb1 63 209717247 104858592+ 83 Linux PRODUCTION mysqlv101 ~ # Footnotes: If the end sector matches what it was previous, I've had to use fdisk just to see what the End value is of the extended LUN, then use parted to ensure the use of the correct starting cyl.

Clear Memory Cache on Linux Server

Our Virtuozzo VPS hwnodes would hang onto large amounts of cached memory after a period of uptime and as it grew, load average of the hwnode would correspondingly increase during activities such as container restarts, create, stop, or start. After troubleshooting and applying virtuozzo tool and kernel updates, the problem persisted (though not as bad). What I’ve noticed however, is that load average spikes were not occuring after a fresh reboot, where free -m showed a lot of free RAM as opposed to cached RAM. After a couple of days of uptime, there was only a couple gig of free RAM and 58GB + of cached RAM (we run 150 containers per hwnode). These conditions do not appear favorable and the system starves for resources during container restarts, etc. Noticablely, CPU sy would skyrocket from vztop output. After clearing cache mem via:

sync; echo 3 > /proc/sys/vm/drop_caches

load average spikes during a container restart have stopped.

Howto Setup yum repositories to install from ISO

Mount the iso to /mnt
In the case of redhat, copy the media.repo file from the root dir of the mount point to /etc/yum.repos.d. Edit the file and add the baseurl at the end as depicted below.

[root@mkoffer2 ~]# cat /etc/yum.repos.d/media.repo
[InstallMedia]
name=Red Hat Enterprise Linux 6.3
mediaid=1339640147.274118
metadata_expire=-1
gpgcheck=0
cost=500
baseurl=file:///mnt

* mediaid comes from the .diskinfo file located in the root of the DVD

This will allow installation of base packages. If you need non-base packages, you’ll need to include the correct stanza in the media.repo config file.

Enabling automatic logoff

Let’s face it. Sometimes admins leave themselves logged into a system and walk away. Systems can be set to automatically logoff a user after a period of activity by setting some parameters in /etc/profile (Linux/Solaris), /etc/security/.profile (AIX).

Append the following to the system profile:

#Enable automatic logoff
TMOUT=600
TIMEOUT=600
readonly TMOUT TIMEOUT
export TMOUT TIMEOUT
Set readonly to prevent the variables from being overwritten.

automount in linux

[root@mobileweb2:~]$ cat /etc/auto.home
* netapp1b:/vol/vol1/home/&

[root@mobileweb2:~]$ cat /etc/auto.master
/home /etc/auto.home
/misc /etc/auto.misc
/net -hosts
+auto.master

[root@mobileweb2:~]$ /etc/init.d/autofs restart

Resizing an Online Multipath Device

esizing an Online Multipath Device

If you need to resize an online multipath device, use the following procedure:

1. Resize your physical device

2. Use the following command to find the paths to the LUN:

# multipath -l
3. Resize your paths. FOr SCSI devices, writing a 1 to the rescan file for the device causes the SCSI driver to rescan:

# echo 1 > /sys/block/device_name/device/rescan
4. Resize your multipath device by running the multipathd resize command:

# multipath -k’reisze map mpathX’
5. Resize the file system (assuming no LVM or DOS partition are used):

# resize2fs /dev/mapper/mpathX

How To Add Disk to Linux System without Rebooting

It is possible to add a new hard disk without a reboot. Whether or not it is a physical server where a physical disk was inserted into an available slot, or a new virtual disk was added to a VMware VM, the command structure to discover the new disk is the same.

First, take a look at the existing scsi devices using the lsscsi command:

~# lsscsi
[0:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
[0:0:1:0] disk VMware Virtual disk 1.0 /dev/sdb
After adding a new vmdk or a new physical disk, /dev/sdc should pop in after running through the following sequence of commands in order to make the operating system aware of the new storage device.

The recommended command to do this, as per redhat documentation, is:

echo “c t l” > /sys/class/scsi_host/hosth/scan

From the lsscsi output above, this would translate into:

echo “0 2 0” > /sys/class/scsi_host/host0/scan

where host 0, channel 0, SCSI target ID 2, and LUN 0 are the next in-line / appropriate values:

The deprecated method was to submit a scsi command to /proc/scsi/scsi as follows, but does still work:

echo “scsi add-single-device 0 0 2 0” > /proc/scsi/scsi

If that’s not a comfortable option, then a rescan of the entire scsi bus for additional disks can be performed. Some systems have more than one scsi_host. With one command, they can all be rescanned as follows:

for host in $(ls -1d /sys/class/scsi_host/*); do echo “- – -” > ${host}/scan ; done

To rescan them individually, perform an ls in /sys/class/scsi_host and scan each one via a single command where # is the host number:

echo “- – -” > /sys/class/scsi_host/host#/scan

Note that in some distros of Linux, the kernel may recognize the new device once it has been added to the system, and the bus scan may not need to be performed. This would be validated by observing the output of lssci or by looking in syslog (/var/log/messages) for any indication of a newly added device. Something like the following would show up in the logs:

sd 0:0:2:0: Attached scsi disk sdc

This matches up nicely with lsscsi output:

~# lsscsi
[0:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
[0:0:1:0] disk VMware Virtual disk 1.0 /dev/sdb
[0:0:2:0] disk VMware Virtual disk 1.0 /dev/sdc
Next, the new device can be partitioned, labeled, create a file system, and mounted.

echo -e -n “o\nn\np\n1\n\n\nw\n” | fdisk /dev/sdc > /dev/null 2>&1
mkfs.ext3 /dev/sdc1
mkdir /data
e2label /dev/sdc1 /data
vi /etc/fstab
mount -a

On the flip side of the coin, a device may be deleted using the following command:

# echo 1 > /sys/block/devName/device/delete (where devName is the device name to be removed, for example sdc in the following line)
# echo 1 > /sys/block/sdc/device/delete

Another variation of this operation is echo 1 > /sys/class/scsi_device/h:c:t:l/device/delete
A deprecated method of the above command is:
# echo “scsi remove-single-device 0 0 2 0” > /proc/scsi/scsi