Amazon EC2 instance running on CentOS 7. Apache logs keep saying that it can’t write to file due to permission where file permissions are properly setup, only to realize it was SELinux in action.

Problem 1: Can’t serve files on a custom directory

The first problem I have encountered is that I tried to setup the application inside /data/www/html/sites/mysite. When viewed on the browser, it says 403 Forbidden and error logs says:

1
13)Permission denied: [client 121.54.44.93:23180] AH00529: /data/www/html/sites/mysite/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that ‘/data/www/html/sites/mysite/’ is executable
The directory structure has proper ownership and permissions, ex: directory is owned by apache:apache, file permission is 0644 and directory permission is 0755. It doesn’t make sense at all. I noticed though that the default document root has no problem serving the php file so I decided to serve it off the /var/www/html/mysite directory, which is the default document root.

Problem 2: Can’t write to file

Moving to the default document root directory did the trick and I was able to run the application but with errors. The error says it can’t write to file although again, proper permissions are already set to the directory. Below is the error (it is a custom error log, but if writing to log file doesn’t work, imagine how your upload functionality would work):

1
PHP Warning: fopen(/var/www/html/mysite/application/config/../../logs/web/20150708.ALL.log): failed to open stream: Permission denied in /var/www/html/mysite/application/core/App_Exceptions.php
Surprise! SELinux is here!

After realizing that it was SELinux whose messing with me for the past 2 hours, I was thinking of ditching CentOS and go with the recommended Ubuntu instead. But then my instinct tells me that if SELinux is blocking the read/write operations, it must did it for a good reason, and that was for security. I realize that you need to specify which files/directories Apache can serve files and which files/directories it can write into.

SELinux seems to have some rules/policies that applies to files/directories on top of the unix file permissions structure. When I run the command below on the default document root, I saw more information on the file/directory permissions.

1
ls -Z /var/www/html/mysite
Below is the output (some information removed):

1
2
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 application
-rw-r–r–. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 index.php
And below is what I got for other normal directories:

1
drwxr-xr-x. apache apache unconfined_u:object_r:default_t:s0 www
Therefore, we can conclude that we need to specify the proper SELinux permissions on directories in order to serve files on a custom directory and set another SELinux permissions to allow writing to file. Therefore, we can solve the original problem then.

Fixing the original problem

So we want to serve our files at /data/www/html/sites/mysite and enable writing to log files and file uploads as well? Let’s play nice with SELinux.

First, copy the files as usual to /data/www/html/sites/mysite, then set the proper ownership and permissions.

# Ownership
sudo chown apache:apache -R /data/www/html/sites/mysite
cd /data/www/html/sites/mysite

# File permissions, recursive
find . -type f -exec chmod 0644 {} \;

# Dir permissions, recursive
find . -type d -exec chmod 0755 {} \;

# SELinux serve files off Apache, resursive
sudo chcon -t httpd_sys_content_t /data/www/html/sites/mysite -R

# Allow write only to specific dirs
sudo chcon -t httpd_sys_rw_content_t /data/www/html/sites/mysite/logs -R
sudo chcon -t httpd_sys_rw_content_t /data/www/html/sites/mysite/uploads -R
httpd_sys_content_t – for allowing Apache to serve these contents and httpd_sys_rw_content_t – for allowing Apache to write to those path.

KernelCare for CentOS & RHEL 7

KernelCare is now available for CentOS & RHEL 7 kernels.
Latest CentOS / RHEL kernels can be patched against privilege escalation vulnerability CVE-2014-4943. Other supported kernels were patched against it last week

CVEs: CVE-2014-4943

Systems with AUTO_UPDATE=True (DEFAULT) in /etc/sysconfig/kcare/kcare.conf will automatically update, and no action is needed for them.

You can manually update the server by running:
# /usr/bin/kcarectl –update

Details:
CVE-2014-4943 kernel: net: pppol2tp: level handling in pppol2tp_[s,g]etsockopt()
A flaw in the Linux kernel allowing an unprivileged user to escalate to kernel privilege when CONFIG_PPPOL2TP is enabled.

Let’s face it. NFS is a magical thing. It allows you to centralize your storage, share volumes across systems, and all while maintaining sane permissions and ownership. Unfortunately, it can also be a bit of a fickle beast. Let’s say you just had your volume configured and you set up the mounts. You go and run this command:
mount -t nfs 10.10.10.1:/vol1/fs1 /data

Works like a champ, you now have your data partition mounted over NFS. So you add this line to your /etc/fstab and make it mount automagically.
10.10.10.1:/vol1/fs1 /data nfs defaults 0 0

A few weeks go by and you apply a kernel update. No big deal, you apply the updates and during your next maintenance window reboot to apply the new kernel. Then you start to see applications failing and notice the volume isn’t actually mounted. This is an unfortunate result of the automounter subsystem.

It’s like this. At boot time the root partition gets mounted, automounter reads the /etc/fstab file, and boots any filesystem that doesn’t have noauto as a mount option. We’re still very early in the boot process so the network isn’t up yet, so naturally any network filesystems fail. The real problem here is that at no point does automounter go back and attempt to remount those systems. So your NFS mount points fail because there is no network, and done is done.

The developers were nice enough to provide a fix for this. There exists a mount option called _netdev. If we quote directly from the man page (sourced from RHEL 6.4):

_netdev
The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

This is awesome, and exactly what we want. So you modify your entry in fstab to look like this:
10.10.10.1:/vol1/fs1 /data nfs defaults,_netdev 0 0

You’ve been bitten by NFS mounting in the past so you throw this in your test environment and reboot immediately. After the system comes up you notice a problem. Your NFS volumes are still unmounted. You see, there’s a bit of a hitch. Automounter followed the same procedure that it did before, except this time it didn’t even attempt to mount /data. The _netdev option doesn’t tell the system to mount the filesystem when network comes up, it says don’t attempt to mount it at all if the network isn’t up. There is still a missing piece to the puzzle. If you look at your init scripts there is a service called netfs. If you read the script you can see in the chkconfig header this description:

# description: Mounts and unmounts all Network File System (NFS),
# CIFS (Lan Manager/Windows), and NCP (NetWare) mount points.

This is exactly what you need. It is a service whose sole purpose is to read your /etc/fstab and mount network filesystems. All you have to do is enable it
chkconfig netfs on

and watch the magic happen. Now your mount boot process should look something like this:
1.Automounter reads /etc/fstab
2.Ignores /data since it has _netdev option set
3.Mounts all other filesystems
4.Finishes mount jobs and allows system to continue booting
5.Network comes up
6.Service netfs started
7.netfs reads /etc/fstab and finds an nfs filesystem
8.netfs mounts /data

What’s funny is that while I was researching this problem I never stumbled across netfs as a service. I had even gone so far as to start planning out my own custom init script that would do exactly this, except specifically for my mount points instead of generalizing. It’s nice to see that I was on the right track, but even better that the tools already existed.

In the situation where you have exhausted all of the disk space in a volume group, you can add additional disks to the volume group in order to remediate the situation.

Locate the additional disk using the fdisk -l command.

[root@slave ~]# fdisk -l
Disk /dev/sdd: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
The next step is to create a LVM partition using the above disk (/dev/sdd).

[root@slave ~]# fdisk /dev/sdd
Type in n and press enter three times.
Type in +2G and press enter.
Type in t and press enter.
Type in 8e and press enter.
Type in w and press enter.
Type in q and press enter.
Run partprobe to make the kernel aware of the disk changes.

[root@slave ~]# partprobe
Check the partition path by running fdisk -l.

[root@slave ~]# fdisk -l
Disk /dev/sdd: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xeef01ba1
Device Boot Start End Blocks Id System
/dev/sdd1 2048 4196351 2097152 8e Linux LVM
Create the physical volume using pvcreate.

[root@slave ~]# pvcreate /dev/sdd1
Physical volume “/dev/sdd1” successfully created
We’re now going to add 2GB of space from the new /dev/sdd1 partition to the lvtestvolume volume group.

[root@slave ~]# vgextend lvtestvolume /dev/sdd1
Volume group “lvtestvolume” successfully extended
Verify the size of the volume group by running vgdisplay lvtestvolume.

[root@slave ~]# vgdisplay lvtestvolume
— Volume group —
VG Name lvtestvolume
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 4.99 GiB
PE Size 4.00 MiB
Total PE 1278
Alloc PE / Size 512 / 2.00 GiB
Free PE / Size 766 / 2.99 GiB
VG UUID wxeQ0N-ZboT-lN2s-CCeQ-zkbb-B24Q-Khh6NB
The disk space has now been made available to the volume group, however the logical volume needs to be extended in order to make use of the additional space.

[root@slave ~]# lvdisplay lvtestvolume
— Logical volume —
LV Path /dev/lvtestvolume/data
LV Name data
VG Name lvtestvolume
LV UUID fwCnof-OoOu-8PNR-wPC2-LqBL-TQK6-DCZbiR
LV Write Access read/write
LV Creation host, time slave, 2014-10-13 19:23:50 -0400
LV Status available
# open 1
LV Size 2.00 GiB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:4
We can now use lvextend to add an additional 2GB of space to the lvtestvolume.

[root@slave ~]# lvextend -L +2G /dev/lvtestvolume/data
Extending logical volume data to 4.00 GiB
Logical volume data successfully resized
We can finally verify the disk space addition from using lvdisplay /dev/lvtestvolume/data or through using df -hk | grep /data.

[root@slave ~]# lvdisplay /dev/lvtestvolume/data
— Logical volume —
LV Path /dev/lvtestvolume/data
LV Name data
VG Name lvtestvolume
LV UUID fwCnof-OoOu-8PNR-wPC2-LqBL-TQK6-DCZbiR
LV Write Access read/write
LV Creation host, time slave, 2014-10-13 19:23:50 -0400
LV Status available
# open 1
LV Size 4.00 GiB
Current LE 1024
Segments 2
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:4
[root@slave ~]# df -hk | grep /data
/dev/mapper/lvtestvolume-data 1998672 6144 1871288 1% /data