{"id":460,"date":"2020-10-21T02:54:52","date_gmt":"2020-10-21T02:54:52","guid":{"rendered":"https:\/\/dft.wiki\/?p=460"},"modified":"2022-05-31T22:45:13","modified_gmt":"2022-05-31T22:45:13","slug":"using-scp-and-rsync-for-copy-and-directory-sync","status":"publish","type":"post","link":"https:\/\/dft.wiki\/?p=460","title":{"rendered":"Using SCP and RSYNC for Copying and Syncing"},"content":{"rendered":"<p>Both SCP and RSYNC, use SSH Protocol as a way to transfer files and directories.<\/p>\n<p>While SCP just uploads or downloads files and folders, Rsync can be used to synchronize the content of both folders copying only what is new.<\/p>\n<p>The basic Syntax of both looks like this:<\/p>\n<pre>[scp\/rsync] [OPTION] \u2026 [<strong>SRC<\/strong>] \u2026 [USER@]HOST:<strong>DEST<\/strong>\r\n[scp\/rsync] [OPTION] \u2026 [USER@]HOST:<strong>SRC<\/strong> [<strong>DEST<\/strong>]<\/pre>\n<p><strong>SCP<\/strong><\/p>\n<pre><strong>scp \/home\/user\/file.zip user@200.100.50.3:\/home\/user\/file.zip<\/strong>\r\nscp file.zip user@200.100.50.3:\/home\/user\r\nscp file.zip 200.100.50.3:\/home\/user\r\nscp file.zip user@200.100.50.3:\r\nscp file.zip 200.100.50.3:\r\n\r\n<strong>scp user@200.100.50.3:\/home\/user\/file.zip \/home\/user\/file.zip<\/strong>\r\nscp user@200.100.50.3:\/home\/user\/file.zip \/home\/user\r\nscp user@200.100.50.3:\/home\/user\/file.zip .\r\nscp 200.100.50.3:file.zip \/home\/user\r\nscp 200.100.50.3:file.zip .<\/pre>\n<p>The examples above are multiple ways of doing basically upload (1 to 5) and download (6 to 10).<\/p>\n<p>The first of each block in bold show the complete syntax except if you need to apply any additional option.<\/p>\n<p>By <strong>omitting the name<\/strong> of the file at the destiny location it will copy with the <strong>same name<\/strong>.<\/p>\n<p>By <strong>omitting the full path on the origin<\/strong> means the <strong>current directory<\/strong>.<\/p>\n<p>By <strong>omitting the full path at the destiny<\/strong> means to save at the <strong>home directory<\/strong> of the user.<\/p>\n<p>By <strong>omitting the username<\/strong> means you will use the <strong>same user<\/strong> remotely as you are logged in locally.<\/p>\n<p>Use the option <strong>-r<\/strong> to copy recursively entire directories, <strong>-v<\/strong>\u00a0 (verbose) to print debugging messages, and <strong>-P<\/strong> (same as <strong>&#8211;progress<\/strong>) to see real-time progress.<\/p>\n<p><strong>RSYNC<\/strong><\/p>\n<p>The big feature of RSYNC is the ability to check the dates to define which is the most recent version of the file and compare the content of both directories to understand what is missing on both sides.<\/p>\n<pre>rsync <strong>-r<\/strong> \/directory user@200.100.50.3:\/directory\r\nrsync <strong>-r<\/strong> user@200.100.50.3:\/directory \/directory<\/pre>\n<p>The usage option <strong>-r<\/strong> (recursively) will copy all the content but option <strong>-a<\/strong> (archive) would only copy after comparing both directories (local and remote), and copy what is missing or update files with a new version.<\/p>\n<p>Option <strong>-b<\/strong> (backup) would not overwrite the old with the new version but would make a backup copy of the old version first (<strong>file.zip~<\/strong>), and <strong>\u2010\u2010delete<\/strong> would delete files at the destination that were deleted from the source.<\/p>\n<p><span data-preserver-spaces=\"true\">Using option <\/span><strong><span data-preserver-spaces=\"true\">-z<\/span><\/strong><span data-preserver-spaces=\"true\">\u00a0(compress) the traffic will be compressed, saving time in slow connections but increasing CPU usage on both ends. The option\u00a0<\/span><strong><span data-preserver-spaces=\"true\">\u2010\u2010dry-run<\/span><\/strong><span data-preserver-spaces=\"true\"> is going to show everything like if it was going to do but in fact will not perform anything because it is only informative.<\/span><\/p>\n<p>The attribute <strong>&#8211;exclude &#8216;dir1&#8217;<\/strong> will prevent the synchronization to compare and copy the content of a file or directory.<\/p>\n<p>You can also try the option <strong>\u2010\u2010backup-dir=$(date +%F)<\/strong> and see what it does.<\/p>\n<hr \/>\n<p><strong>BONUS<\/strong><\/p>\n<p>There is a tool called LSYNCD that runs as a service and periodically compare and sync files (using RSYNC) according to the provided configuration.<\/p>\n<pre>sudo apt install lsyncd -y\r\nsudo mkdir \/etc\/lsyncd\r\nsudo nano \/etc\/lsyncd\/lsyncd.conf.lua<\/pre>\n<p>Add the following content customizing the parameters in bold:<\/p>\n<pre>settings {\r\nlogfile = \"\/var\/log\/lsyncd\/lsyncd.log\",\r\nstatusFile = \"\/var\/log\/lsyncd\/lsyncd.status\",\r\nstatusInterval = <strong>20<\/strong>,\r\nnodaemon   = false\r\n}\r\n\r\nsync {\r\ndefault.rsync,\r\nsource = \"\/<strong>path<\/strong>\/\",\r\ntarget = \"\/<strong>path<\/strong>\"\r\n}<\/pre>\n<p>Note: the interval (20) is in seconds.<\/p>\n<pre class=\"command\">systemctl start lsyncd\r\nsystemctl enable lsyncd\r\nsystemctl status lsyncd<\/pre>\n<p>For troubleshooting:<\/p>\n<pre>tail -f \/var\/log\/lsyncd\/lsyncd.log\r\ntail -f \/var\/log\/lsyncd\/lsyncd.status<\/pre>\n<hr \/>\n<p>Backing up additional or newer files from source to destination.<\/p>\n<pre>rsync -<strong>rlptgo<\/strong> \/source \/destination<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Both SCP and RSYNC, use SSH Protocol as a way to transfer files and directories. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,6],"tags":[],"class_list":["post-460","post","type-post","status-publish","format-standard","hentry","category-linux","category-raspberry-pi"],"_links":{"self":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/460","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=460"}],"version-history":[{"count":13,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/460\/revisions"}],"predecessor-version":[{"id":2946,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/460\/revisions\/2946"}],"wp:attachment":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=460"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=460"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=460"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}