my last example of Bash if statements we created a backup script which would run “rsync” as per the time set in a cron.
But wouldn’t it be nice if we could run our “rsync” script only if there was a change to the files in our directory instead of running a cron every hour despite no change? With inotify-tools you can do just that.
What is inotify-tools?
Inotify tools are a set of command line programs based on inotify a Linux kernel (2.6.13 or later) feature which can be used to monitor filesystem events.
Installing inotify-tools
Software versions :
inotify-tools.x86_64 3.14-1.el6
CentOS 6.5
Linux kernel 2.6.32-042stab085.20
#Add the epel repo
[db1@rmohan.com ~]$ rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
#Install inotify-tools
[db1@rmohan.com ~]$ yum install inotify-tools
Inotify-tools has two commands
Inotifywait
This command simply blocks for inotify events, making it appropriate for use in shell scripts. It can watch any set of files and directories, and can recursively watch entire directory trees.
inotifywatch
inotifywatch collects filesystem usage statistics and outputs counts of each inotify event.
We will use inotifywait, since we do not need statistics.
inotifywait syntax
inotifywait filename
inotifywait example
[db1@rmohan.com ~]$ inotifywait /tmp
Setting up watches.
Watches established.
/tmp/ MODIFY test
As you can see in the above example an event (in this case a “modify action” was performed on a file named “test” inside /tmp) triggered the output.
Now “inotifywait” by default checks for all events including if a file was opened but not written to, Since we only want “rsync” to trigger on change events like when a file is modified, we will need to specify the “-e” flag along with the list of events we want to be notified about.
[db1@rmohan.com ~]$ inotifywait -m -r -e \ modify,attrib,close_write,move,create,delete /tmp
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/tmp/ MODIFY a
/tmp/ CLOSE_WRITE,CLOSE a
The -m flag is for continuous monitoring as by default inotifywait will exit on the first event and -r means recursively or check through sub-directories as well.
List of inotifywait events
access
A watched file or a file within a watched directory was read from.
modify
A watched file or a file within a watched directory was written to.
attrib
The metadata of a watched file or a file within a watched directory was modified. This includes timestamps, file permissions, extended attributes etc.
close_write
A watched file or a file within a watched directory was closed, after being opened in writeable mode. This does not necessarily imply the file was written to.
close_nowrite
A watched file or a file within a watched directory was closed, after being opened in read-only mode.
close
A watched file or a file within a watched directory was closed, regardless of how it was opened. Note that this is actually implemented simply by listening for both close_write and close_nowrite, hence all close events received will be output as one of these, not CLOSE.
open
A watched file or a file within a watched directory was opened.
moved_to
A file or directory was moved into a watched directory. This event occurs even if the file is simply moved from and to the same directory.
moved_from
A file or directory was moved from a watched directory. This event occurs even if the file is simply moved from and to the same directory.
move
A file or directory was moved from or to a watched directory. Note that this is actually implemented simply by listening for both moved_to and moved_from, hence all close events received will be output as one or both of these, not MOVE.
move_self
A watched file or directory was moved. After this event, the file or directory is no longer being watched.
create
A file or directory was created within a watched directory.
delete
A file or directory within a watched directory was deleted.
delete_self
A watched file or directory was deleted. After this event the file or directory is no longer being watched. Note that this event can occur even if it is not explicitly being listened for.
unmount
The filesystem on which a watched file or directory resides was unmounted. After this event the file or directory is no longer being watched. Note that this event can occur even if it is not explicitly being listened to.
Now let’s use inotifywait with our script.
[db1@rmohan.com~]$ vim inotify-example
while true #run indefinitely
do
inotifywait -r -e modify,attrib,close_write,move,create,delete /dir && /bin/bash backup-script
done
Since we want to continuously monitor changes, we use an infinite while loop and the Logic “&&” operator will ensure that our backup script is only triggered on a successful completion of the inotifywait event
[db2@rmohan.com]$bash inotify-example
+ true
+ inotifywait -r -e modify,attrib,close_write,move,create,delete /
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/dir DELETE a
+ /bin/bash backup-script
+ rsync -avz -e “ssh ” /path/to/yourfile user@backupserver.com:/backup/ 2> \ /tmp/error.txt
+ mail -s “backup complete” user@youremail.com
+ echo “backup for $(date) “
+ true
+ inotifywait -r -e modify,attrib,close_write,move,create,delete /dir
Setting up watches. Beware: since -r was given, this may take a while!
About rsync + inotify-tools real-time synchronization mode
Introduce the synchronization mode
On the left is the original, the general rsync cs architecture (client & server) synchronization mode, the data source server to install rsync server, unified control by the server can transfer the contents of the data, such as permissions, directories, the number of files, Is rsync client, that is, it seems that the data from the source server to the backup server.
On the right is coupled with the inotify-tools synchronization mode, in the data source server to install rsync client, the backup source server to install rsync server, but also by the unified control of the transmission of data content, but here is the data source server as client side , Because the initiation of the transmission is rsync client, so it seems to be the data from the source server to the backup server.
From the logical cs architecture c and s into the opposite position, but the transmission mode is still from s to c.
The data source server ip is tentatively scheduled as 192.168.1.112 and the data backup server ip is tentatively designated as 192.168.1.111
I need to do the synchronization directory is /app/www/uploads, both the source server and the backup server, and the same path,
the same permissions (permissions consistent is very important, because the synchronization data related to permissions,
and because I am not using root To synchronize, so to ensure that the transfer of both sides of the authority is writable or readable Caixing)
The operating system is centos 6.8 x64
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package inotify-tools.i686 0:3.14-1.el6 will be installed
—> Package rsync.i686 0:3.0.6-9.el6_4.1 will be updated
—> Package rsync.i686 0:3.0.6-12.el6 will be an update
–> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================================================================================================
Package Arch Version RepositorySize
=============================================================================================================================================================================================================================================
Installing:
inotify-tools i686 3.14-1.el6epel 45 k
Updating:
rsync i686 3.0.6-12.el6 base 329 k
Transaction Summary
=============================================================================================================================================================================================================================================
Install 1 Package(s)
Upgrade 1 Package(s)
Total download size: 375 k
Downloading Packages:
(1/2): inotify-tools-3.14-1.el6.i686.rpm | 45 kB 00:00
(2/2): rsync-3.0.6-12.el6.i686.rpm| 329 kB 00:00
———————————————————————————————————————————————————————————————————————————————
Total509 kB/s | 375 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : rsync-3.0.6-12.el6.i686 1/3
Installing : inotify-tools-3.14-1.el6.i686 2/3
Cleanup: rsync-3.0.6-9.el6_4.1.i6863/3
Verifying : inotify-tools-3.14-1.el6.i686 1/3
Verifying : rsync-3.0.6-12.el6.i686 2/3
Verifying : rsync-3.0.6-9.el6_4.1.i6863/3
Installed:
inotify-tools.i686 0:3.14-1.el6
Updated:
rsync.i686 0:3.0.6-12.el6
Complete!
mkdir /app/rsync
vi /app/rsync/rsync_do.sh
#!/bin/bash
src=’/app/www/Uploads/’
passwordfile=’/app/rsync/rsync.passwd’
user=’www’
host=’192.168.1.111′
rsync_module=’uploads’
/usr/bin/inotifywait -mrq –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w %f’ -e modify,delete,create,attrib ${src} | while read DATE TIME DIR file
do
/usr/bin/rsync -vzrtopg –delete –progress ${src} ${user}@${host}::${rsync_module} –password-file=${passwordfile}
echo “${file} was rsynced at ${DATE}_${TIME} in ${DIR}” >> /var/log/rsync.log 2>&1
done
1.inotifywait is the initify command tool
2.rsync src directory need to pay attention, the source directory is the same as rsync, for the folder is to distinguish between / and no /
3. Need to pay attention to rsync synchronization module name to match
4.inotifywait command can listen to the operation, the output information, and is always output, so with the read together, the output of the inotifywait through the capture of the variables forced to record the synchronization log output to /var/log/rsync.log
5. – delete parameters need attention, this will delete the source directory does not have anything, so in the absence of a good test and test before the script can not temporarily add this parameter
6. rsync in the script parsed out is /usr/bin/rsync -vzrtopg –delete –progress /app/www/uploads/ www@$192.168.1.111 :: uploads –password-file = /app/rsync/rsync.passwd, synchronous directory is /app/www/uploads/, use www this user to synchronize, the target server is 192.168.1.111, rsync module is uploads, so that the source server / app / www / uploads / All the data (uploads directory itself)
to the backup server /app/www/uploads/, there are delete parameters, keep the source and backup consistent.
/app/rsync/rsync.passwd
echo “123456” > /app/rsync/rsync.passwd
nohup /app/rsync/rsync_do.sh &
yum install rsync
mkdir /etc/rsyncd
mkdir /app/rsync
vi /app/rsync/rsync.conf
uid = root
gid = root
use chroot = no
max connections = 3
timeout = 800
pid file = /var/run/rsyncd.pid
lockfile = /var/run/rsyncd.lock
log file = /var/log/rsyncd.log
[uploads]
path = /app/www/Uploads
ignore errors = yes
hosts allow = 10.111.111.112
hosts deny = *
read only = no
write only = no
list = yes
uid = www
gid = www
auth users = www
secrets file = /app/rsync/rsync.passwd
echo “www:123456” > /app/rsync/rsync.passwd
chmod 600 /app/rsync/rsync.passwd
rsync –daemon –config=/app/rsync/rsync.conf
1.rsync
Compared with the traditional cp, tar backup method, rsync has the advantages of high security, fast backup, support incremental backup, etc., through rsync can solve the real-time requirements of the data backup requirements, such as regular backup file server data to Remote server, the local disk to do regular data mirroring, with the application of the scale of the system continues to expand, the data security and reliability also made better requirements, rsync in the high-end business system has gradually exposed a lot In addition, rsync synchronization data, you need to scan all the files after the comparison, the difference transmission. If the number of files reaches the level of millions or even tens of millions, scanning all the files will be very time-consuming. And is changing is often a very small part of it, which is very inefficient way. Second, rsync can not be real-time to monitor, synchronize data, although it can trigger the process through the linux daemon synchronization, but the trigger action will have two time difference, which led to the server and client data may be inconsistent, Completely recover data when a failure is applied. Based on the above reasons, rsync + inotify combination appeared!
2.inotify
Inotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism, the Linux kernel from 2.6.13, joined the Inotify support, through Inotify can monitor the file system to add, delete, modify, move and other subtle Event, the use of the kernel interface, third-party software can monitor the file system under the various changes in the file, and inotify-tools is such a third-party software. In the above section, we talked about, rsync can achieve the trigger file synchronization, but through the crontab daemon way to trigger the synchronization of data and actual data will be different, and inotify can monitor the file system changes, when the file There is any change, it triggers rsync synchronization, so just to solve the real-time synchronization of data problems.
3.rsync.conf configuration explanation
uid = root # rsync server
gid = root # rsync server
use chroot = no #chroot
max connections = 3 #rsync
timeout = 800 #
pid file = /var/run/rsyncd.pid #
lockfile = /var/run/rsyncd.lock
log file = /var/log/rsyncd.log #
[uploads] #rsync
path = /app/www/Uploads #
ignore errors = yes #
hosts allow = 192.168.1.112 #
hosts deny = * #
read only = no #
write only = no #
list = yes #
uid = www #
gid = www #
auth users = www
secrets file = /app/rsync/rsync.passwd #
a) rsync -av –progress <src-dir>/ <dst-dir> *** (/) ***
b) rsync -av –progress <src-dir> <dst-dir>
c) rsync -avu –progress –delete <src-dir>/ <dst-dir>
d) rsync -av –progress –temp-dir=/tmp <src-dir>/ <dst-dir>
Synchronize all files in the src-dir directory to the dst-dir directory
B) Synchronize all files in the src-dir directory to the dst-dir / src-dir directory
C) to src-dir directory content to the dst-dir directory under the difference update, add / update to add the replacement, there is a reduction of its deletion
D) than a) more – temp-dir = / tmp, that is, specify / tmp for the temporary swap area, so as to avoid the target directory space is not enough to cause the file can not sync errors.
Corresponding to the above six command formats, rsync has six different modes of operation:
1) copy the local file. This mode of operation is started when neither SRC nor DES path information contains a single colon “:” delimiter. Such as: rsync-a / data / backup
2) use a remote shell program (such as rsh, ssh) to achieve the contents of the local machine copy to the remote machine. This mode is started when the DST path address contains a single colon “:” delimiter. Such as: rsync -avz * .c foo: src
3) use a remote shell program (such as rsh, ssh) to achieve the contents of the remote machine to copy to the local machine. This mode is started when the SRC address path contains a single colon “:” delimiter. Such as: rsync -avz foo: src / bar / data
4) Copy files from the remote rsync server to the local machine. This mode is started when the SRC path information contains a “::” delimiter. Such as: rsync-av root@172.16.78.192 :: www / databack
5) Copy files from the local machine to the remote rsync server. This mode is started when the DST path information contains a “::” delimiter. Such as: rsync -av / databack root@172.16.78.192 :: www
6) List of remote machine files. This is similar to rsync transmission, but as long as the command can be omitted in the local machine information. Such as: rsync -v rsync: //172.16.78.192/www
-m, –monitor
-r, –recursive
-e <event>, –event <event>
Listen for specific event(s) only. The events which can be listened for are listed in the EVENTS section. This option can be specified more than once. If omitted, all events are listened for.
-q, –quiet
If specified once, the program will be less verbose. Specifically, it will not state when it has completed establishing all inotify watches.
–timefmt <fmt>
Set a time format string as accepted by strftime(3) for use with the ‘%T’ conversion in the –format option.
–format <fmt>
Output in a user-specified format, using printf-like syntax. The event strings output are limited to around 4000 characters and will be truncated to this length. The following conversions are supported:
%w
This will be replaced with the name of the Watched file on which an event occurred.
%f
When an event occurs within a directory, this will be replaced with the name of the File which caused the event to occur. Otherwise, this will be replaced with an empty string.
%e
Replaced with the Event(s) which occurred, comma-separated.
%Xe
Replaced with the Event(s) which occurred, separated by whichever character is in the place of ‘X’.
%T
Replaced with the current Time in the format specified by the –timefmt option, which should be a format string suitable for passing to strftime(3).
Recent Comments