{"id":6522,"date":"2017-03-02T14:47:06","date_gmt":"2017-03-02T06:47:06","guid":{"rendered":"http:\/\/rmohan.com\/?p=6522"},"modified":"2017-03-02T14:47:06","modified_gmt":"2017-03-02T06:47:06","slug":"use-tar-pigz-ssh-to-achieve-efficient-transmission-of-large-data","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=6522","title":{"rendered":"Use tar + pigz + ssh to achieve efficient transmission of large data"},"content":{"rendered":"

Use tar + pigz + ssh to achieve efficient transmission of large data<\/p>\n

<\/span>Before we copy large data when the host, such as to copy more than 100GB of mysql raw data, we usually practice as follows:<\/p>\n

<\/span>Package the tar.gz file at the source
\n<\/span>Using scp or rsync copy to the target host
\n<\/span>Unzip the file at the target host<\/p>\n

<\/span>These three processes are synchronized, that is, they can not be executed at the same time, resulting in inefficiency.<\/p>\n

<\/span>Now we will optimize the process to the data stream, while the implementation of (non-blocking mode), the efficiency can generally be increased to more than 3 times the original, the specific realization is as follows:<\/p>\n

<\/span>Disk read —-> packaging —-> compression ——> transmission —-> decompression -> unpacking —-> plate<\/p>\n

<\/span>-> tar | -> gzip | -> ssh | -> gzip | -> tar<\/p>\n

<\/span>For example, I want to copy the local test directory to the “target IP” data directory, the command is as follows:<\/p>\n

<\/span>Tar -c test \/ | pigz | ssh -c arcfour128 Target IP “gzip -d | tar -xC \/ data”<\/p>\n

<\/span>Of course, here the decompression process is still using the efficiency of the lower gzip, if the decompression tool replaced lz4 (but need to compile and install separately), then the efficiency can be improved a lot.<\/p>\n

<\/span>If you do not need to extract, the command changes to:<\/p>\n

<\/span>Tar -c test \/ | pigz | ssh -c arcfour128 target IP “cat> \/data\/test.tar.gz”<\/p>\n

<\/span>Note: Because of the use of streaming compression, decompression process must be added-i parameters, and tar -ixf \/ data\/test.tar.gz.<\/p>\n

<\/span>Description: pigz is an efficient compression tool that can be used for each sub-core CPU’s remaining performance for compression calculations. <\/span>The traditional gzip can only use single-core CPU. <\/span>For example, a 2 8core cpu server using pigz and gzip compression of the same data, the general performance gap of at least 7-8 times more than (generally do not reach the theory of 16 times, because limited by the disk read and write speed and memory resources <\/span>).<\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Use tar + pigz + ssh to achieve efficient transmission of large data<\/p>\n

Before we copy large data when the host, such as to copy more than 100GB of mysql raw data, we usually practice as follows:<\/p>\n

Package the tar.gz file at the source Using scp or rsync copy to the target host Unzip the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6522"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6522"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6522\/revisions"}],"predecessor-version":[{"id":6523,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6522\/revisions\/6523"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6522"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6522"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6522"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}