Why does my rsync slow down over time?

distro_dave

I'm trying to back up some files from my PC to an external hard drive using rsync. The copy works fine but takes a lot of time even for the standard USB 2.0. For an USB 2.0 the real transfer speed should be round 30MB/s, in my test the speed reaches sometimes 2MB/s. The pen-drive I'm using is a good 32GB USB3.0 with FAT32 fs. Note: before each test I erased the whole content of the pen-drive and I also tried on a different USB port. Part of the...

Originally asked by lcit on Stack Exchange (unix) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

rsync is not the best solution for copy. It is the best solution to synchronize files. Speed for most data write/read depends on a buffer. In the case of writing files, buffers get filled and you see a higher speed for that. As soon the buffer will be written to the device (buffer flush) the speed is going down. Write speed to the device depends not only on specification (USB 2.0 or 3.0 ...), also on protocol (implementation), driver, block sizes. Sometimes it is better to copy the whole file instead of letting sync programs first scan for changes. This brings overhead and is lowering the speed in case of local attached devices. If you doing this via network the story may be different.

Full answer by 0x0C4: https://unix.stackexchange.com/a/303942 (CC BY-SA 4.0).

Need it done for your business? UphillTech.