{"id":7674,"date":"2018-07-15T05:31:32","date_gmt":"2018-07-14T21:31:32","guid":{"rendered":"http:\/\/rmohan.com\/?p=7674"},"modified":"2018-07-15T06:45:28","modified_gmt":"2018-07-14T22:45:28","slug":"how-to-improve-rsync-performance","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=7674","title":{"rendered":"How to Improve rsync Performance"},"content":{"rendered":"

I need to transfer 10TB of data from one machine to another machine. Those 10TB of files are living in a large RAID which span across 7 different disks. The target machine has another large RAID which span across 12 different disks. It is not easy to copying those files locally. Therefore, I decide to copy the files over the LAN.<\/p>\n

There are four options popping up in my head:\u00a0scp<\/strong>,\u00a0rsync<\/strong>,\u00a0rsyncd (rsync as daemon)<\/strong>\u00a0and\u00a0netcat<\/strong>.<\/p>\n

<\/div>\n

scp<\/h2>\n

scp is handy, easy to use but comes with two disadvantages: slow and not fault-tolerant. Since scp comes with the highest security, all data are encrypted before the transfer. It will slow down the overall performance because of the extra encryption stuffs (which makes the data larger), and extra computational resource (which uses more CPU). If the transfer is interrupted, there is no easy way to resume the process other than transferring everything again. Here are some example commands:<\/p>\n

#Source machine\r\n#Typical speed is about 20 to 30MB\/s\r\nscp -r \/data target_machine:\/data\r\n\r\n#Or you can enable the compression on the fly\r\n#Depending on the type of your data, if your data is already compressed, you may see no or negative speed improvement\r\nscp -rC \/data target_machine:\/data\r\n<\/pre>\n
rsync<\/h2>\n
rsync is similar to scp. It comes with the encryption (via SSH) such that the data is safe. It also allows you to transfer the newer files only. This will reduce the amount of data being transferred. However, it comes with few disadvantages: long decision time, encryption (which increase the size of overhead) and extra computational resource(e.g., data comparison, encryption and decryption etc). For example, if I use rsync to transfer 10TB of files from one machine to another machine (where the directory on the target machine is blank), it can easily take 5 hours to determine which files will need to be transferred before the actual data transfer is initialized.<\/p>\n
#Run on the target machine\r\nrsync -avzr -e ssh --delete-after source_machine:\/data\/ \/data\/\r\n\r\n#Use a less secure encryption algorithm to speed up the process\r\nrsync -avzr --rsh=\"ssh -c blowfish\" --delete-after source_machine:\/data\/ \/data\/\r\n\r\n#Use an even less secure algorithm to get the top speed\r\nrsync -avzr --rsh=\"ssh -c arcfour\" --delete-after source_machine:\/data\/ \/data\/\r\n\r\n#By default, rsync compares the files using checksum, file size and modification date.\r\n#Reduce the decision process by skipping the hash check\r\nrsync -avzr --rsh=\"ssh -c arcfour\" --delete-after --whole-file source_machine:\/data\/ \/data\/\r\n<\/pre>\n
Anyway, no matter what you do, the top speed of rsync in a consumer-grade gigabit network is around 45MB\/s. On average, the speed is around 25-35MB\/s. Keep in mind that this number does not include the decision time, which can be few hours.<\/p>\n
rsyncd (rsync as a daemon)<\/h2>\n
Thanks for the comment of our reader. I got a chance to investigate the rsync as a daemon. Basically, the idea of running rsync as a daemon is similar to rsync. On the server, we run rsync as a service\/daemon. We specify which directory we want to \u201cexport\u201d to the clients (e.g., \/usr\/ports). When the files get changed on the server, it records the changes so that the when the clients talk to the server, the decision time will be faster. Here is how to set up rsync server on FreeBSD<\/p>\n
sudo nano \/usr\/local\/etc\/rsyncd.conf\r\n<\/pre>\n
And this is my configuration file:<\/p>\n
pid file = \/var\/run\/rsyncd.pid\r\n\r\n#Notice that I use derrick here instead of other systems users, such as nobody\r\n#That's because nobody does not have permission to access the path, i.e., \/data\/\r\n#Either you make the source directory available to \"nobody\", or you change the daemon user.\r\nuid = derrick\r\ngid = derrick\r\nuse chroot = no\r\nmax connections = 4\r\nsyslog facility = local5\r\npid file = \/var\/run\/rsyncd.pid\r\n\r\n[mydata]\r\n   path = \/data\/\r\n   comment = data\r\n<\/pre>\n
Don't forget to include the following in \/etc\/rc.conf, so that the service will be started automatically.\r\n\r\nrsyncd_enable=\"YES\"\r\n<\/pre>\n
#Let's start the rsync service:\r\n\r\nsudo \/usr\/local\/etc\/rc.d\/rsyncd start\r\n\r\n<\/pre>\n
To pull the files from the server to the clients, run the following:<\/p>\n
rsync -av myserver::mydata \/data\/\r\n\r\n#Or you can enable compression\r\nrsync -avz myserver::mydata \/data\/\r\n<\/pre>\n
To my surprise, it works much better than running rsync alone. Here are some data I collected during transferring 10TB files from ZFS to ZFS:<\/p>\n
Bandwidth measured on the client machine: 70MB\/s<\/p>\n
zpool IO speed on the client side: 75MB\/s<\/p>\n
P.S. Initially, the speed was about 45-60MB\/s, after I tweak my Zpool, I can get the top speed to 75-80MB\/s. Please check out\u00a0here<\/a>\u00a0for references.<\/p>\n
I notice that the decision time is much faster than running rsync alone. Also the process is much more stable, with zero interruption, i.e.,<\/p>\n
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(521) [receiver=3.1.0]\r\nrsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [generator=3.1.0]\r\nrsync: [receiver] write error: Broken pipe (32)\r\n<\/pre>\n
NetCat<\/h2>\n
NetCat is similar to cat, except that it works at the network level. I decide to use netcat for the initial transfer. If it is interrupted, I will let rsync to kick in the process. Netcat does not encrypt the data, so the overhead is very small. If you transfer the file within a local network and you don\u2019t care about the security, netcat is a perfect choice.<\/p>\n
There is only one disadvantage of using netcat. It can only handle one file at a time. It doesn\u2019t mean you need to run netcat for every single file. Instead, we can tar the file before feeding to netcat, and untar the file at the receiving end. As long as we do not compress the files, we can keep the CPU usage small.<\/p>\n
#Open two terminals, one for the source and another one for the target machine.\r\n\r\n#On the target machine:\r\n#Go to the directory, e.g., \r\ncd \/data\r\n\r\n#Run the following:\r\nnc -l 9999| tar xvfp -\r\n\r\n#On the source machine:\r\n#Go to the directory, e.g.,\r\ncd \/data\r\n\r\n#Pick a port number that is not being used, e.g., 9999\r\ntar -cf - . | nc target_machine 9999\r\n<\/pre>\n
Unlike rsync, the process will start right the way, and the maximum speed is around 45 to 60MB\/s in a gigabit network.<\/p>\n
Conclusion<\/h2>\n\n\n\n\n\n\n\n
Candidates<\/td>\nTop Speed (w\/o compression)<\/td>\nTop Speed (w\/ compression)<\/td>\nResume<\/td>\nStability<\/td>\nInstant Start?<\/td>\n<\/tr>\n
scp<\/td>\n40MB\/s<\/td>\n25MB\/s<\/td>\nNo<\/td>\nLow<\/td>\nInstant<\/td>\n<\/tr>\n
rsync<\/td>\n25MB\/s<\/td>\n50MB\/s<\/td>\nYes<\/td>\nMedium<\/td>\nLong Preparation<\/td>\n<\/tr>\n
rsyncd<\/td>\n30MB\/s<\/td>\n70MB\/s<\/td>\nYes<\/td>\nHigh<\/td>\nShort Preparation<\/td>\n<\/tr>\n
netcat<\/td>\n60MB\/s (tar w\/o -z)<\/td>\n40MB\/s (tar w\/ -z)<\/td>\nNo<\/td>\nVery High<\/td>\nInstant<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
 <\/p>\n
\n
\n\n\n\n\n\n\n
\n
Choice<\/b><\/div>\n<\/td>\n
\n
Command<\/b><\/div>\n<\/td>\n
\n
Pros<\/b><\/div>\n<\/td>\n
\n
Cons<\/b><\/div>\n<\/td>\n<\/tr>\n
\n
#1<\/div>\n<\/td>\n
\n
scp<\/div>\n<\/td>\n
\n
    \n
  • \n
    can be speed up choosing simple encryption<\/div>\n<\/li>\n
  • \n
    can recursively copy directories<\/div>\n<\/li>\n<\/ul>\n<\/td>\n
\n
<\/div>\n<\/td>\n<\/tr>\n
\n
#2<\/div>\n<\/td>\n
\n
rsync<\/div>\n<\/td>\n
\n
flexible and convenient for directory synchronization<\/div>\n<\/td>\n
\n
possible, but not easy to configure to NOT to use encryption<\/div>\n<\/td>\n<\/tr>\n
\n
#3<\/div>\n<\/td>\n
\n
sftp<\/div>\n<\/td>\n
\n
can be speed up choosing simple encryption<\/div>\n<\/td>\n
\n
can’t recursively copy directories<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
<\/div>\n
<\/div>\n
Notice<\/i>: when running any tool it consumes about 5-10% of CPU at both sender and receiver machines, apparently, doing encryption\/decryption.<\/p>\n
TEST DETAILS<\/div>\n
These are the actual commands and generated output by these tools.<\/p>\n
\n
\n
rsync<\/h4>\n<\/div>\n\n\n\n<\/colgroup>\n\n\n\n
\n
default “-a” –archive mode<\/div>\n<\/td>\n
\n
“-z” compress<\/div>\n<\/td>\n<\/tr>\n
\n
rsync\u00a0-a\u00a0–progress DIR2REPLICATE root@10.10.10.2:\/tmp<\/div>\n
\u00a0\u00a0\u00a0411533312\u00a0\u00a0\u00a00%\u00a0\u00a0\u00a045.38MB\/s\u00a0\u00a0\u00a0\u00a00:26:04<\/div>\n
\u00a0\u00a01407877120\u00a0\u00a0\u00a01%\u00a0\u00a0\u00a044.41MB\/s\u00a0\u00a0\u00a0\u00a00:26:16<\/div>\n
\u00a0\u00a01716748288\u00a0\u00a0\u00a02%\u00a0\u00a0\u00a042.09MB\/s\u00a0\u00a0\u00a0\u00a00:27:36<\/div>\n
\u00a0\u00a02002550784\u00a0\u00a0\u00a02%\u00a0\u00a0\u00a046.47MB\/s\u00a0\u00a0\u00a0\u00a00:24:541<\/div>\n
\u00a0\u00a02382397440\u00a0\u00a0\u00a03%\u00a0\u00a0\u00a045.31MB\/s\u00a0\u00a0\u00a0\u00a00:25:24<\/div>\n
\u00a0\u00a02762407936\u00a0\u00a0\u00a03%\u00a0\u00a0\u00a045.34MB\/s\u00a0\u00a0\u00a0\u00a00:25:15<\/div>\n<\/td>\n
\n
\u00a0rsync -az\u00a0–progress DIR2REPLICATE root@10.10.10.2:\/tmp<\/div>\n
991383915 100%\u00a0\u00a0\u00a013.67MB\/s\u00a0\u00a0\u00a0\u00a00:01:09<\/div>\n
\u00a0\u00a0\u00a0990955265 100%\u00a0\u00a0\u00a014.02MB\/s\u00a0\u00a0\u00a0\u00a00:01:07<\/div>\n
\u00a0\u00a0\u00a0202624740 100%\u00a0\u00a0\u00a015.42MB\/s\u00a0\u00a0\u00a0\u00a00:00:12<\/div>\n
\u00a0\u00a0\u00a0202771784 100%\u00a0\u00a0\u00a015.87MB\/s\u00a0\u00a0\u00a0\u00a00:00:12<\/div>\n
\u00a0\u00a0\u00a0\u00a091676674 100%\u00a0\u00a0\u00a012.86MB\/s\u00a0\u00a0\u00a0\u00a00:00:06<\/div>\n
\u00a0\u00a0\u00a0\u00a091628045 100%\u00a0\u00a0\u00a011.76MB\/s\u00a0\u00a0\u00a0\u00a00:00:07<\/div>\n
\u00a0\u00a01082301721 100%\u00a0\u00a0\u00a016.86MB\/s\u00a0\u00a0\u00a0\u00a00:01:01<\/div>\n
\u00a0\u00a01081744094 100%\u00a0\u00a0\u00a017.14MB\/s\u00a0\u00a0\u00a0\u00a00:01:00<\/div>\n
\u00a0\u00a0\u00a0444531263 100%\u00a0\u00a0\u00a013.06MB\/s\u00a0\u00a0\u00a0\u00a00:00:32<\/div>\n
\u00a0\u00a0\u00a0444311917 100%\u00a0\u00a0\u00a012.97MB\/s\u00a0\u00a0\u00a0\u00a00:00:32<\/div>\n
\u00a0\u00a0\u00a0\u00a025956199 100%\u00a0\u00a0\u00a011.99MB\/s\u00a0\u00a0\u00a0\u00a00:00:02<\/div>\n
\u00a0\u00a0\u00a0\u00a025387962 100%\u00a0\u00a0\u00a016.94MB\/s\u00a0\u00a0\u00a0\u00a00:00:01<\/div>\n
\u00a0\u00a0\u00a0\u00a094059363 100%\u00a0\u00a0\u00a015.51MB\/s\u00a0\u00a0\u00a0\u00a00:00:05<\/div>\n
\u00a0\u00a0\u00a0\u00a094189273 100%\u00a0\u00a0\u00a014.61MB\/s\u00a0\u00a0\u00a0\u00a00:00:06<\/div>\n
\u00a0\u00a0\u00a0369550738 100%\u00a0\u00a0\u00a016.31MB\/s\u00a0\u00a0\u00a0\u00a00:00:21<\/div>\n
\u00a0\u00a0\u00a0370924791 100%\u00a0\u00a0\u00a015.96MB\/s\u00a0\u00a0\u00a0\u00a00:00:22<\/div>\n
\u00a0\u00a0\u00a0143659839 100%\u00a0\u00a0\u00a014.75MB\/s\u00a0\u00a0\u00a0\u00a00:00:09<\/div>\n
\u00a0\u00a0\u00a0141681760 100%\u00a0\u00a0\u00a014.58MB\/s\u00a0\u00a0\u00a0\u00a00:00:09<\/div>\n
\u00a0\u00a0\u00a0\u00a074662680 100%\u00a0\u00a0\u00a014.45MB\/s\u00a0\u00a0\u00a0\u00a00:00:04<\/div>\n
\u00a0\u00a0\u00a0\u00a073882769 100%\u00a0\u00a0\u00a012.73MB\/s\u00a0\u00a0\u00a0\u00a00:00:05<\/div>\n
\u00a0\u00a0\u00a0\u00a0\u00a01809543 100%\u00a0\u00a0\u00a013.59MB\/s\u00a0\u00a0\u00a0\u00a00:00:00<\/div>\n
<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\n
###
\n### “-c arcfour” cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key
\n###
\nrsync -a -P -e “ssh -T -c arcfour -o Compression=no -x”\u00a0<\/b>DIR2REPLICATE root@10.10.10.2:\/tmp<\/b><\/div>\n
\u00a0 1081744094 100%\u00a0\u00a0 65.35MB\/s\u00a0\u00a0\u00a0 0:00:15
\n444531263 100%\u00a0\u00a0 56.34MB\/s\u00a0\u00a0\u00a0 0:00:07
\n444311917 100%\u00a0\u00a0 61.61MB\/s\u00a0\u00a0\u00a0 0:00:06
\n369550738 100%\u00a0\u00a0 53.94MB\/s\u00a0\u00a0\u00a0 0:00:06
\n370924791 100%\u00a0\u00a0 60.03MB\/s\u00a0\u00a0\u00a0 0:00:05
\n23319017231 100%\u00a0\u00a0 65.89MB\/s\u00a0\u00a0\u00a0 0:05:37
\n23308793162 100%\u00a0\u00a0 64.88MB\/s\u00a0\u00a0\u00a0 0:05:42
\n11951287020 100%\u00a0\u00a0 65.68MB\/s\u00a0\u00a0\u00a0 0:02:53
\n3453648896\u00a0 28%\u00a0\u00a0 68.11MB\/s\u00a0\u00a0\u00a0 0:02:0<\/div>\n
<\/h4>\n
scp<\/h4>\n<\/div>\n<\/div>\n
\n\n\n\n<\/colgroup>\n\n\n\n\n\n
\n
default “-r” recursive<\/div>\n<\/td>\n
\n
“-C” compress “-r” recursive<\/div>\n<\/td>\n<\/tr>\n
\n
scp -r DIR2REPLICATE root@10.10.10.2:\/tmp<\/div>\n
100%\u00a0\u00a0193MB\u00a0\u00a064.4MB\/s\u00a0\u00a0\u00a000:03<\/div>\n
100%\u00a0\u00a0424MB\u00a0\u00a060.6MB\/s\u00a0\u00a0\u00a000:07<\/div>\n
100%\u00a0\u00a0945MB\u00a0\u00a063.0MB\/s\u00a0\u00a0\u00a000:15<\/div>\n
100%\u00a0\u00a0945MB\u00a0\u00a059.1MB\/s\u00a0\u00a0\u00a000:16<\/div>\n
100% 1032MB\u00a0\u00a064.5MB\/s\u00a0\u00a0\u00a000:16<\/div>\n
100% 1032MB\u00a0\u00a060.7MB\/s\u00a0\u00a0\u00a000:17<\/div>\n
100%\u00a0\u00a0749MB\u00a0\u00a053.5MB\/s\u00a0\u00a0\u00a000:14<\/div>\n
100% 1253MB\u00a0\u00a062.6MB\/s\u00a0\u00a0\u00a000:20<\/div>\n
18% 4615MB\u00a0\u00a062.6MB\/s\u00a0\u00a0\u00a005:18<\/div>\n<\/td>\n
\n
scp -Cr DIR2REPLICATE root@10.10.10.2:\/tmp<\/div>\n
100%\u00a0\u00a0193MB\u00a0\u00a016.1MB\/s\u00a0\u00a0\u00a000:12<\/div>\n
100%\u00a0\u00a0424MB\u00a0\u00a014.6MB\/s\u00a0\u00a0\u00a000:29<\/div>\n
100%\u00a0\u00a0945MB\u00a0\u00a015.0MB\/s\u00a0\u00a0\u00a001:03<\/div>\n
100%\u00a0\u00a0945MB\u00a0\u00a014.8MB\/s\u00a0\u00a0\u00a001:04<\/div>\n
100%\u00a0\u00a0424MB\u00a0\u00a014.1MB\/s\u00a0\u00a0\u00a000:30<\/div>\n
100%\u00a0\u00a0352MB\u00a0\u00a017.6MB\/s\u00a0\u00a0\u00a000:20<\/div>\n
100%\u00a0\u00a0193MB\u00a0\u00a017.6MB\/s\u00a0\u00a0\u00a000:11<\/div>\n
100%\u00a0\u00a0135MB\u00a0\u00a016.9MB\/s\u00a0\u00a0\u00a000:08<\/div>\n
100% 1032MB\u00a0\u00a017.8MB\/s\u00a0\u00a0\u00a000:58<\/div>\n
100% 1032MB\u00a0\u00a017.8MB\/s\u00a0\u00a0\u00a000:58<\/div>\n
100%\u00a0\u00a0354MB\u00a0\u00a017.7MB\/s\u00a0\u00a0\u00a000:20<\/div>\n
100%\u00a0\u00a0749MB\u00a0\u00a018.3MB\/s\u00a0\u00a0\u00a000:41<\/div>\n
100% 1253MB\u00a0\u00a018.4MB\/s\u00a0\u00a0\u00a001:08<\/div>\n
\u00a0\u00a06% 1518MB\u00a0\u00a017.7MB\/s\u00a0\u00a0\u00a021:43<\/div>\n<\/td>\n<\/tr>\n
\n
“-c arcfour” cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key.<\/div>\n<\/td>\n
\n
<\/div>\n<\/td>\n<\/tr>\n
\n
scp\u00a0-c arcfour\u00a0-r DIR2REPLICATE root@10.10.10.2:\/tmp<\/div>\n
100%\u00a0\u00a0424MB 141.3MB\/s\u00a0\u00a0\u00a000:03<\/div>\n
100%\u00a0\u00a0945MB 135.0MB\/s\u00a0\u00a0\u00a000:07<\/div>\n
100%\u00a0\u00a0945MB 189.1MB\/s\u00a0\u00a0\u00a000:05<\/div>\n
100%\u00a0\u00a0424MB 141.2MB\/s\u00a0\u00a0\u00a000:03<\/div>\n
100%\u00a0\u00a0352MB 117.5MB\/s\u00a0\u00a0\u00a000:03<\/div>\n
100% 1032MB 147.4MB\/s\u00a0\u00a0\u00a000:07<\/div>\n
100% 1032MB 147.5MB\/s\u00a0\u00a0\u00a000:07<\/div>\n
100%\u00a0\u00a0749MB 149.8MB\/s\u00a0\u00a0\u00a000:05<\/div>\n
100% 1253MB 156.6MB\/s\u00a0\u00a0\u00a000:08<\/div>\n
100%\u00a0\u00a0\u00a024GB 142.0MB\/s\u00a0\u00a0\u00a002:53<\/div>\n
100%\u00a0\u00a0595MB 119.1MB\/s\u00a0\u00a0\u00a000:05<\/div>\n
100%\u00a0\u00a0\u00a082GB 138.3MB\/s\u00a0\u00a0\u00a010:09<\/div>\n
51% 9099MB 141.3MB\/s\u00a0\u00a0\u00a001:01<\/div>\n<\/td>\n
\n
<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
sftp<\/div>\n
\n\n\n\n<\/colgroup>\n\n\n\n\n\n
\n
default behavior<\/div>\n<\/td>\n
\n
“-R” to increase request queue length (default is 64)<\/div>\n
“-B” to increase read\/write request size (default is 32 KB)<\/div>\n<\/td>\n<\/tr>\n
\n
sftp\u00a0\u00a0root@10.10.10.2:\/tmp<\/b><\/div>\n
10% 2363MB\u00a0\u00a057.8MB\/s\u00a0\u00a0\u00a005:43 ETA<\/div>\n
15% 3349MB\u00a0\u00a058.1MB\/s\u00a0\u00a0\u00a005:25 ETA<\/div>\n
32% 7311MB\u00a0\u00a059.3MB\/s\u00a0\u00a0\u00a004:11 ETA<\/div>\n
35% 7803MB\u00a0\u00a060.6MB\/s\u00a0\u00a0\u00a003:58 ETA<\/div>\n
43% 9594MB\u00a0\u00a062.1MB\/s\u00a0\u00a0\u00a003:23 ETA<\/div>\n
69%\u00a0\u00a0\u00a015GB\u00a0\u00a058.6MB\/s\u00a0\u00a0\u00a001:55 ETA<\/div>\n
77%\u00a0\u00a0\u00a017GB\u00a0\u00a062.1MB\/s\u00a0\u00a0\u00a001:20 ETA<\/div>\n<\/td>\n
\n
sftp\u00a0\u00a0-R 128 -B 65536\u00a0root@10.10.10.2:\/tmp<\/b><\/div>\n
\u00a0\u00a02%\u00a0\u00a0551MB\u00a0\u00a058.9MB\/s\u00a0\u00a0\u00a006:08 ETA<\/div>\n
\u00a0\u00a08% 1806MB\u00a0\u00a062.3MB\/s\u00a0\u00a0\u00a005:28 ETA<\/div>\n
41% 9170MB\u00a0\u00a060.6MB\/s\u00a0\u00a0\u00a003:35 ETA<\/div>\n
56%\u00a0\u00a0\u00a012GB\u00a0\u00a062.6MB\/s\u00a0\u00a0\u00a002:32 ETA<\/div>\n
100%\u00a0\u00a0\u00a022GB\u00a0\u00a062.5MB\/s\u00a0\u00a0\u00a005:56<\/div>\n<\/td>\n<\/tr>\n
\n
“-c arcfour” cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key.<\/div>\n<\/td>\n
\n
<\/div>\n<\/td>\n<\/tr>\n
\n
sftp\u00a0-oCiphers=arcfour\u00a0root@10.10.10.2:\/tmp<\/b><\/div>\n
3%\u00a0\u00a0711MB 142.5MB\/s\u00a0\u00a0\u00a002:31 ETA<\/div>\n
18% 4115MB 146.0MB\/s\u00a0\u00a0\u00a002:04 ETA<\/div>\n
23% 5156MB 148.1MB\/s\u00a0\u00a0\u00a001:55 ETA<\/div>\n
28% 6379MB 144.6MB\/s\u00a0\u00a0\u00a001:49 ETA<\/div>\n
34% 7672MB 144.0MB\/s\u00a0\u00a0\u00a001:41 ETA<\/div>\n
37% 8389MB 143.7MB\/s\u00a0\u00a0\u00a001:36 ETA<\/div>\n
62%\u00a0\u00a0\u00a014GB 143.8MB\/s\u00a0\u00a0\u00a000:58 ETA<\/div>\n
85%\u00a0\u00a0\u00a019GB 142.4MB\/s\u00a0\u00a0\u00a000:22 ETA<\/div>\n
92%\u00a0\u00a0\u00a020GB 142.3MB\/s\u00a0\u00a0\u00a000:12 ETA<\/div>\n
100%\u00a0\u00a0\u00a022GB 144.4MB\/s\u00a0\u00a0\u00a002:34<\/div>\n<\/td>\n
\n
<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/div>\n<\/div>\n
\n
\n
TEST ENVIRONMENT<\/h4>\n<\/div>\n
The test was performed between two servers interconnected by private 10 Gbit link with 9000 MTU “jumbo frame<\/i><\/a>“. The files copies were large (100’s GB) binary files.<\/p>\n<\/div>\n\n\n\n<\/colgroup>\n\n\n\n
\n
iperf\u00a0network bandwidth test between 10.10.10.2 and 10.10.10.1<\/div>\n<\/td>\n
\n
Network interface configuration (10 Gbit, MTU 9000)<\/div>\n<\/td>\n<\/tr>\n
\n
Server listening on TCP port 5001<\/div>\n
TCP window size: 85.3 KByte (default)<\/div>\n
————————————————————<\/div>\n
[\u00a0\u00a04] local 10.10.10.2 port 5001 connected with 10.10.10.1 port 57279<\/div>\n
[ ID] Interval\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Transfer\u00a0\u00a0\u00a0\u00a0\u00a0Bandwidth<\/div>\n
[\u00a0\u00a04]\u00a0\u00a00.0-10.0 sec\u00a0\u00a011.5 GBytes\u00a0\u00a09.89 Gbits\/sec<\/div>\n
[\u00a0\u00a05]\u00a0\u00a00.0-30.0 sec\u00a0\u00a034.6 GBytes\u00a0\u00a09.89 Gbits\/sec<\/div>\n
[\u00a0\u00a04]\u00a0\u00a00.0- 0.9 sec\u00a0\u00a01000 MBytes\u00a0\u00a09.80 Gbits\/sec<\/div>\n
[\u00a0\u00a05]\u00a0\u00a00.0- 8.8 sec\u00a0\u00a09.77 GBytes\u00a0\u00a09.53 Gbits\/sec<\/div>\n
[\u00a0\u00a04]\u00a0\u00a00.0- 8.7 sec\u00a0\u00a010.0 GBytes\u00a0\u00a09.89 Gbits\/sec<\/div>\n
[\u00a0\u00a05]\u00a0\u00a00.0-86.8 sec\u00a0\u00a0\u00a0100 GBytes\u00a0\u00a09.89 Gbits\/sec<\/div>\n<\/td>\n
\n
<\/div>\n
[root@10.10.10.2]# cat \/etc\/sysconfig\/network-scripts\/ifcfg-bond0<\/div>\n
DEVICE=”bond0″<\/div>\n
BOOTPROTO=none<\/div>\n
IPADDR=10.10.10.2<\/div>\n
NETMASK=255.255.255.192<\/div>\n
ONBOOT=”yes”<\/div>\n
USERCTL=no<\/div>\n
TXQUEUELEN=100000<\/div>\n
MTU=8912<\/div>\n
BONDING_OPTS=”mode=1 miimon=200 primary=eth0″<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
 <\/p>\n
ALTERNATIVE METHODS TO MOVE DATA FAST<\/h3>\n
Copy directories via\u00a0netcat<\/a>: tar | nc. Renders speed\u00a0 ~251 Mb\/s ( = ~1 TB\/hr).<\/p>\n
### On receiver ###
\nnc -v -l 5555\u00a0 | tar -xvf –<\/b><\/p>\n
### On Sender: test2del – large directory to move ###
\ntime tar -cvf – test2del | nc -v 10.100.100.2 5555<\/b><\/p>\n
### Output calculated ###
\n11GB in\u00a0 27.465 s = 293 MB\/s
\n42GB in 2m51.513s = 249 Mb\/s (~1 TB\/hr)
\n42GB in 2m50.630s = 251 Mb\/s (~1 TB\/hr)<\/p>\n","protected":false},"excerpt":{"rendered":"
I need to transfer 10TB of data from one machine to another machine. Those 10TB of files are living in a large RAID which span across 7 different disks. The target machine has another large RAID which span across 12 different disks. It is not easy to copying those files locally. Therefore, I decide to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7674"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7674"}],"version-history":[{"count":2,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7674\/revisions"}],"predecessor-version":[{"id":7676,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7674\/revisions\/7676"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7674"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7674"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7674"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}