Home > admin-stuff > Resume rsync transfer after SSH connection crash

Resume rsync transfer after SSH connection crash

Fri May 15 2009 02:00:00 GMT+0200 (Central European Summer Time)

Are you using SSH for your secure maintenance of your servers? - Sure.

Do you copy files with SCP between hosts? - Most likely.

Is there a need to transfer big files over slow and unreliable network connections (*)? - Could be.

Can SCP resume a download after the connection crashed? - No.

So why not simply use rsync over SSH for your file transfer. A minor drawback is, that unless you set up a rsync daemon (not appropriate for my case) you have to call rsync manually. Sadly rsync doesn't offer something like "automatic retry in case of a connection failure". (**)

Good for us, because now it's tool time again; a single bash script does the trick:

#!/bin/sh 
# reliable file transfer

# try rsync for x times 
I=0 
MAX_RESTARTS=5 
LAST_EXIT_CODE=1 
while [ $I -le $MAX_RESTARTS ] 
do 
	I=$(( $I + 1 )) 
	echo $I. start of rsync 
	rsync -av --partial --progress -e "ssh" x-ian@x-ian.net:~/MY_BIG_FILE . 
	LAST_EXIT_CODE=$? 
	if [ $LAST_EXIT_CODE -eq 0 ]; then 
		break 
	fi 
done

# check if successful 
if [ $LAST_EXIT_CODE -ne 0 ]; then 
	echo rsync failed for $I times. giving up. 
else 
	echo rsync successful after $I times. 
fi 

Ah, just a sidenote as I always forget the syntax: If you need to remote execute a command via SSH with variables from your local shell, take this:

CMD="test -e M\_BIG\_FILE || cp MY\_BIG\_FILE \`hostname -s\`-MY\_BIG\_FILE" 
ssh x-ian@x-ian.net $CMD 

--

(*) If you only have an unstable satellite link, even 150 MB are way too big. (**) Make sure that you actually test over the network; using rsync with source and destination files on the same system deactivates the delta-calculation algorithm.

Categories: admin-stuff
Comments are closed.