If you try to rsync a subset of many files from a single directory, you might get the error “Argument list too long”. Actually, you can get this error with many bash commands. This article explains how to work around it.
rsync -avz /images/* cdn.example.com:/images/ -bash: /usr/bin/rsync: Argument list too long
The solution is to either remove the “*” from the source as in the following example:
rsync -avz /images/ cdn.example.com:/images/
Or do things a little smarter. Here’s the two step process. Here’s how to do it.
Create a list of the files (in this case we want pictures) you want to copy:
find /images/ -name *.png > /tmp/my_image_list.txt
And now we feed that into rsync:
rsync -avz --files-from=/tmp/my_image_list.txt / cdn.example.com:/images/
The last command needs an explanation. The “rsync -avz” says to use archive mode, be verbose and compress the session. The second part “–files-from=/tmp/my_image_list.txt” is how we feed the source list into rsync. Then we have the “/” by its self. This is the starting point (or reference point) and can be changed if you wanted to change where to get the files from. Finally we have the target of “cdn.example.com:/images/” on the remote machine.
very helpful and clear – thanks!
The thing about –files-from is that it automatically implies -R (–relative), unlike giving files on the command line. So unless that’s really what you want (and it probably isn’t, unless you change the find command a bit), you probably want to put –no-R in the rsync command.
Thanks John. I appreciate the note.
Many many thanks.
I was unable of copy 945 GB image files from source to destination. I was getting “rsync: Argument list too long” error. “rsync -avz /source/ /destination/” command solve my problem.
FYI this method does not append a newline to the last line, so the last file will be ignored.
May want to add the printf switch with “%p\n”
this is really great I was wondering why this wasnt on the top of the google recommended pages