Archive for the ‘Admin stuff’ Category

Resume rsync transfer after SSH connection crash

May 15th, 2009 Comments off

Are you using SSH for your secure maintenance of your servers? – Sure.

Do you copy files with SCP between hosts? – Most likely.

Is there a need to transfer big files over slow and unreliable network connections (*)? – Could be.

Can SCP resume a download after the connection crashed? – No.

So why not simply use rsync over SSH for your file transfer. A minor drawback is, that unless you set up a rsync daemon (not appropriate for my case) you have to call rsync manually. Sadly rsync doesn’t offer something like “automatic retry in case of a connection failure”. (**)

Good for us, because now it’s tool time again; a single bash script does the trick:

# reliable file transfer

# try rsync for x times
while [ $I -le $MAX_RESTARTS ]
  I=$(( $I + 1 ))
  echo $I. start of rsync
  rsync -av --partial --progress -e "ssh" .
  if [ $LAST_EXIT_CODE -eq 0 ]; then

# check if successful
if [ $LAST_EXIT_CODE -ne 0 ]; then
  echo rsync failed for $I times. giving up.
  echo rsync successful after $I times.

Ah, just a sidenote as I always forget the syntax: If you need to remote execute a command via SSH with variables from your local shell, take this:

CMD="test -e M_BIG_FILE || cp MY_BIG_FILE `hostname -s`-MY_BIG_FILE"
ssh $CMD

(*) If you only have an unstable satelitte link, even 150 MB are way too big.
(**) Make sure that you actually test over the network; using rsync with source and destination files on the same system deactivates the delta-calculation algorithm.

Categories: Admin stuff