Home > Admin stuff > Resume rsync transfer after SSH connection crash

Resume rsync transfer after SSH connection crash

May 15th, 2009

Are you using SSH for your secure maintenance of your servers? – Sure.

Do you copy files with SCP between hosts? – Most likely.

Is there a need to transfer big files over slow and unreliable network connections (*)? – Could be.

Can SCP resume a download after the connection crashed? – No.

So why not simply use rsync over SSH for your file transfer. A minor drawback is, that unless you set up a rsync daemon (not appropriate for my case) you have to call rsync manually. Sadly rsync doesn’t offer something like “automatic retry in case of a connection failure”. (**)

Good for us, because now it’s tool time again; a single bash script does the trick:

#!/bin/sh
# reliable file transfer

# try rsync for x times
I=0
MAX_RESTARTS=5
LAST_EXIT_CODE=1
while [ $I -le $MAX_RESTARTS ]
do
  I=$(( $I + 1 ))
  echo $I. start of rsync
  rsync -av --partial --progress -e "ssh" x-ian@x-ian.net:~/MY_BIG_FILE .
  LAST_EXIT_CODE=$?
  if [ $LAST_EXIT_CODE -eq 0 ]; then
    break
  fi
done

# check if successful
if [ $LAST_EXIT_CODE -ne 0 ]; then
  echo rsync failed for $I times. giving up.
else
  echo rsync successful after $I times.
fi

Ah, just a sidenote as I always forget the syntax: If you need to remote execute a command via SSH with variables from your local shell, take this:

CMD="test -e M_BIG_FILE || cp MY_BIG_FILE `hostname -s`-MY_BIG_FILE"
ssh x-ian@x-ian.net $CMD

(*) If you only have an unstable satelitte link, even 150 MB are way too big.
(**) Make sure that you actually test over the network; using rsync with source and destination files on the same system deactivates the delta-calculation algorithm.

Categories: Admin stuff
Comments are closed.