Metawerx Support - How to use RSync at Metawerx

Downloading the tools

For these instructions, you will need:

• plink on Windows, or ssh on Linux/Unix - to establish the tunnel - download here
• cwrsync on Windows, or rsync on Linux/Unix - the rsync client - download here

Download these tools before continuing.

Starting the tunnel

We use rsync over an SSH tunnel for security. You will need to start the tunnel before using rsync.

On Windows, the tunnel is established using plink. On Linux, the ssh tool is used. These are both command line tools.

You will have been provided with:

• a metawerx username, which will be referred to as metausername below
• a metawerx password, which will be referred to as metapassword below
• an rsync server name and port (eg: rsync-meta1.metawerx.net:873, or rsync-himiko.metawerx.net:40873), these are referred to as servername:serverport below.

Your username for the SSH connection will usually be your metawerx username, and you will use your Private Key instead of a password.

To start the tunnel, execute the following command on Windows:

plink -C -v -i c:\path\to\keyfile -L 873:servername:serverport -P 22 metausername@ssh-meta1.metawerx.net

or on Linux:

ssh -Cv -i /path/to/keyfile -L 873:servername:serverport -p 22 metausername@ssh-meta1.metawerx.net

Example, if your username is cassie, key file is in c:\path\to\keyfile, server rsync-himiko.metawerx.net:40873, and you are using Windows, use the following command:

plink -C -v -i c:\path\to\keyfile -L 873:rsync-himiko.metawerx.net:40873 -P 22 cassie@ssh-meta1.metawerx.net

This will open a tunnel between your localhost port 873, and our RSync server.

• Leave this window open
• When you close this window, the tunnel will be closed
• To use rsync, open a new Command Prompt window

In the new window, set your password in an environment variable. There is no command line argument for passwords, so you will be prompted each time if this is not done.

set RSYNC_PASSWORD=metapassword

Now execute rsync commands to synchronize the folders.

Before you begin

Note 1: In the examples below it is very important to remember the trailing slash at the ends of all paths. A trailing slash means copy the contents of the folder. Leaving it off means copy the folder. This can cause a lot of confusion and incorrect results, so for now always use a trailing slash on all paths.

Note 2: Any paths you specify, must always exist. So for the example below, create a folder called c:\mysite, or other folders as appropriate.

Note 3: When using a Windows drive such as C:\, it becomes /cygdrive/c/ in the rsync commands. For example, G:\ becomes /cygdrive/g/

General Examples:

Dry run test only (-n option), see what would happen if you sync'd your entire site from the server, to your local c:\mysite folder, recursively (-r option). It is not recommended to sync your entire site, as it may cause problems with the permissions we have set on your account. Instead, sync specific folders only, such as Java applications.

rsync -vvnrzut rsync://metausername@127.0.0.1/metausername/ /cygdrive/c/mysite/

Dry run test only (-n option), see what would happen if you sync'd your entire local c:\mysite folder to the server

rsync -vvnrzut /cygdrive/c/mysite/ rsync://metausername@127.0.0.1/metausername/

Examples - Tomcat ROOT application

Sync the ROOT application from the server's /private-cgi-bin/tomcat/ROOT folder, to your local c:\mysite\ROOT folder, recursively

rsync -vvrzut rsync://metausername@127.0.0.1/metausername/private-cgi-bin/tomcat/ROOT/ /cygdrive/c/mysite/ROOT/

Sync the ROOT application from your local c:\mysite\ROOT folder, to the server's /private-cgi-bin/tomcat/ROOT folder, recursively

rsync -vvrzut /cygdrive/c/mysite/ROOT/ rsync://metausername@127.0.0.1/metausername/private-cgi-bin/tomcat/ROOT/

Examples - JBoss application

Sync the jmx-console application from the server's /server/default/deploy/jmx-console.war folder, to your local c:\mysite\jmx-console.war folder, recursively

rsync -vvrzut rsync://metausername@127.0.0.1/metausername/server/default/deploy/jmx-console.war/ /cygdrive/c/mysite/jmx-console.war/

Sync the jmx-console application from the server's /server/default/deploy/jmx-console.war folder, to your local c:\mysite\jmx-console.war folder, recursively

rsync -vvrzut /cygdrive/c/mysite/jmx-console.war/ rsync://metausername@127.0.0.1/metausername/server/default/deploy/jmx-console.war/

Examples - Download logs or upload specific files

Sync *.log files from the server's /private-cgi-bin/tomcat/ROOT/log4j folder to your local c:\mylogs folder, non-recursive (this folder only, so without -r option)

rsync -vvzut rsync://metausername@127.0.0.1/metausername/private-cgi-bin/tomcat/ROOT/log4j/*.log /cygdrive/c/mylogs/

Sync *.html files from your local c:\mysite\ROOT folder to the server into /private-cgi-bin/tomcat/ROOT, recursively

rsync -vvrzut /cygdrive/c/mysite/*.html rsync://metausername@127.0.0.1/metausername/private-cgi-bin/tomcat/ROOT

RSync options explained

The following options are used in the examples.

• -n dry run, display what would be copied only, don't transfer any files
• -v verbose
• -vv very verbose
• -r recursive
• -z compression enabled (zip), always use this as it improves transfer speed
• -u update only, don't overwrite newer files
• -t preserve times, important if you are going to modify some files and sync them back
• --delete delete files on the receiving side, that don't exist on the sending side. This option should be used with care, as it is quite easy to delete your entire site using this option.

Tips and Tricks

I generally prefer to have batch files which perform the syncs above. There are some examples in the http://www.metawerx.net/admin/resources.htm area which I will gradually move to here.

Specifically, I use %SERVER% and %LOCAL% environment variables to make the commands clearer. For example:

set SERVER=rsync://metausername@127.0.0.1/metausername/private-cgi-bin/tomcat
set LOCAL=/cygdrive/d/sync/server4
set RSYNC_PASSWORD=metapassword

Then I can use much shorter commands, to sync the folders both ways, such as:

REM --- Sync ROOT app (server to local, then send up any local changes)
rsync -vvrzut %SERVER%/ROOT/ %LOCAL%/ROOT/
rsync -vvrzut %LOCAL%/ROOT/ %SERVER%/ROOT/

REM --- Sync Wiki app (server to local, then send up any local changes)
rsync -vvrzut %SERVER%/Wiki/ %LOCAL%/Wiki/
rsync -vvrzut %LOCAL%/Wiki/ %SERVER%/Wiki/

I also use a few other tricks, such as using the touch command to update the timestamp on my web.xml files before sending them up to the server, when jars or class files are changed. Tomcat automatically reloads applications when web.xml changes, so this means I can edit and deploy changes easily from my text editor.

Recently, I have also started using the following options, to make exact backup copies in Linux from one folder to another: -vrzutpoglD --delete

• -v verbose
• -r recursive
• -z compression enabled (zip), always use this as it improves transfer speed
• -u update only, don't overwrite newer files
• -t preserve times, important if you are going to modify some files and sync them back
• -p preserve permissions
• -o preserve owner (note this is done by ID, so owner IDs may be different on different servers, but syncing back will restore the correct owner if you need to restore)
• -g preserve group (this is by ID as well, see note about -o above)
• -l copy symlinks as symlinks
• -D copy devices and specials (same as --devices --specials)
• --delete delete files on the receiving side, that don't exist on the sending side

--Neale Rudd