Rsync: The Algorithm That Only Moves What Changed in a File

⏱️ Lectura: 15 min

Copying a 10 GB file all over again just to update three lines is a waste that most copy tools still commit. The rsync algorithm has solved that problem since 1996: it compares the source and destination files block by block, detects what changed, and transmits only that difference.

📑 En este artículo

1. TL;DR
2. What Is the Rsync Algorithm and Why It Matters
3. How the Delta Sync Algorithm Works
4. Practical Examples of Syncing with Rsync
5. Getting Started with Rsync
6. Real-World Use Cases for the Rsync Algorithm
7. Common Mistakes and Best Practices with Rsync
8. Comparison: Rsync vs. Other Sync Tools
9. Deep Dive: The Math Behind the Rolling Checksum
10. Frequently Asked Questions
    1. Does rsync need the full file to exist on both sides before syncing?
    2. What does the trailing slash in an rsync path mean?
    3. Does rsync encrypt the data it transmits?
    4. How do I know how much data actually traveled over the network during a sync?
    5. Is rsync good for syncing a database that’s actively receiving writes?
    6. How does rsync differ from Restic or Borg if both avoid retransmitting repeated data?
11. References

Behind the command almost every developer has used at some point to sync a folder lies an elegant piece of engineering. A checksum slides byte by byte over the file without being recalculated from scratch, and that idea, published by Andrew Tridgell in his doctoral thesis, ended up becoming the foundation of modern backup tools like Borg and Restic.

TL;DR

• The rsync algorithm splits the destination file into fixed-size blocks and calculates two checksums per block: a weak one (Adler-32 style) and a strong one (MD5).
• A rolling checksum lets the comparison window slide byte by byte without recalculating everything from scratch.
• rsync -avz --progress source/ destination/ transfers only the blocks that changed between two file trees.
• The destination file’s signature travels over the network first; the sender compares it against its own copy and builds a delta of instructions.
• Borg and Restic apply the same block-level deduplication idea for incremental backups with full history.
• A 1 GB file with just one byte modified can be synced by transmitting only a few kilobytes of delta.
• The --checksum flag forces comparison by hash instead of by size and modification date: slower, but exact.

What Is the Rsync Algorithm and Why It Matters

rsync is both a command-line program and the name of the synchronization algorithm that makes it possible. It has existed since 1996 and comes preinstalled on nearly every Linux distribution, on macOS, and, via WSL or Cygwin, on Windows. Its job can be summed up in one sentence: keep two copies of the same file tree identical while moving as little data as possible over the network.

The rsync algorithm matters because it solves something cp, scp, or an FTP client don’t. All those commands retransmit the entire file even if only a single byte changed. For a 2 KB config file that’s not noticeable, but for a 40 GB database, a log directory, or a virtual machine image, retransmitting everything on every sync is simply not viable over most connections.

Tridgell’s solution splits the problem into two roles. Whoever holds the destination file, which already has a version even if outdated, calculates a block-by-block signature of that file. Whoever holds the source file receives that signature, compares it against their own copy byte by byte, and builds a list of instructions: copy block 4 as is, or transmit these 37 bytes because they’re literal and don’t exist at the destination. That list of instructions, much lighter than the full file, is the only thing that travels over the network.

How the Delta Sync Algorithm Works

The full process has four steps and happens every time you run rsync between two trees that already share a previous version.

First, the receiver (the side holding the destination file) splits that file into fixed-size blocks, typically between 700 bytes and a few kilobytes depending on the file’s total size. For each block it calculates two values: a weak checksum, fast to compute and update, and a strong MD5-style checksum, slow but practically collision-free. That list of pairs is the file’s signature, and it’s the first thing that travels over the network, from the receiver to the sender.

Second, the sender receives the signature and loads it into a hash table indexed by the weak checksum. Then it opens its own copy of the file and starts sliding a window of the same block size, byte by byte, calculating the weak checksum of that window at every position.

Third, at each window position the sender looks up the weak checksum in the hash table. If there’s no match, that byte is marked as literal and the window advances one position. If the weak checksum matches, the sender calculates the strong checksum of that same window and compares it against the one stored in the signature: the weak one can match by chance between different data, but the strong one practically never does. If both match, the entire block is replaced with a copy instruction and the window jumps the full block size instead of advancing byte by byte.

Fourth, the sender assembles a final sequence of instructions and sends it to the receiver, who applies it to their local copy to reconstruct a file identical to the source. At no point does the receiver send the full file, nor does the sender retransmit blocks the receiver already had.

sequenceDiagram
    participant D as Destination (receiver)
    participant O as Source (sender)
    D->>D: splits the file into blocks
    D->>O: sends signature (weak + strong checksum per block)
    O->>O: slides window byte by byte and calculates weak checksum
    O->>O: if there's a weak match, verifies with strong checksum
    O-->>D: instructions (copy block N or literal bytes)
    Note over D,O: only the signature and instructions travel, never the full file

flowchart TD
    A["New position of the sliding window"] --> B{"Does the weak checksum match?"}
    B -->|"No"| C["Mark byte as literal and advance 1 position"]
    B -->|"Yes"| D{"Does the strong MD5 checksum match?"}
    D -->|"No"| C
    D -->|"Yes"| E["Emit COPY instruction for block N and skip the entire block"]
    C --> A
    E --> A

Practical Examples of Syncing with Rsync

The simplest possible command copies one local folder to another while preserving permissions, symlinks, and timestamps:

rsync -av local_folder/ backup_folder/

The -a flag (archive mode) bundles recursion, and preservation of permissions, owners, timestamps, and symlinks into a single option. The -v flag adds verbose output so you can see which files were transferred. Notice both paths end in a slash: that tells rsync to sync the contents of local_folder into backup_folder, not the folder itself as a new subdirectory.

A more realistic case is deploying an application’s build to a remote server over SSH, excluding heavy folders and deleting at the destination anything that no longer exists at the source:

rsync -avz --delete --exclude='.git' --exclude='node_modules' -e ssh ./dist/ [email protected]:/var/www/app/current/

The -z flag compresses data in transit, useful on slow connections. -e ssh forces transport over an encrypted SSH tunnel, and --delete makes the destination end up as an exact mirror of the source: if you deleted a local file, it also disappears from the server.

Getting Started with Rsync

On Debian, Ubuntu, or any derivative, install it with:

sudo apt install rsync

On macOS, with Homebrew (the version that ships with the system tends to be old):

brew install rsync

Confirm it’s installed and check which protocol version it negotiates with:

rsync --version

The first line of that output shows the version number, and further down, the protocol it uses to talk to another rsync instance; both sides of a sync need compatible protocols.

To expose a folder as a service without needing a full SSH account, rsync can run as a daemon with its own configuration file. These are the exact keys a minimal module needs in /etc/rsyncd.conf:

[backups]
    path = /srv/backups
    comment = Backups module
    read only = false
    uid = rsyncuser
    gid = rsyncuser
    hosts allow = 10.0.0.0/24

With that saved, start the daemon and connect from another machine using the rsync:// syntax:

sudo rsync --daemon
rsync -av local_file.tar.gz rsync://10.0.0.5/backups/

💡 Tip: always run a new sync with --dry-run --stats first. The “Literal data” versus “Matched data” section of the final report tells you, in bytes, how much of the transfer was real delta and how much had to be sent in full.

Real-World Use Cases for the Rsync Algorithm

The most common use is still incremental backup: running rsync -a --link-dest=../previous-backup source/ new-backup/ in a nightly cron job creates a copy that only takes up disk space for what changed, while each full snapshot remains browsable as a normal folder thanks to hardlinks shared with the previous copy.

Linux package mirrors (Debian, Arch, CPAN, CTAN) have used rsync for decades to replicate their repositories across hundreds of mirror servers around the world: only new or updated packages generate real traffic between the master mirror and its replicas.

In the modern backup world, tools like Borg and Restic take the same idea a step further: instead of comparing only against the last copy, they split each file into variable-size chunks and index them in a repository with global deduplication. The result isn’t a two-way sync tool, but a backup system with full history where each unique block is stored only once, no matter how many snapshots it appears in.

A fourth common case is deploying static sites or frontend assets to a file server: instead of uploading the entire build via SFTP every time, a CI pipeline runs rsync against the production server and only transfers the files the bundler actually regenerated.

Common Mistakes and Best Practices with Rsync

The trailing slash in paths is the most repeated mistake. rsync -a source destination (no slash on source) copies the entire source folder into destination, creating destination/source/. rsync -a source/ destination (with slash) copies the contents of source directly into destination. Confusing the two is the number one cause of “why did my folder get duplicated” questions in support forums.

⚠️ Heads up: --delete removes any file at the destination that no longer exists at the source. If you swapped the paths by mistake, you can empty a folder that took years to accumulate data in seconds. Always run with --dry-run first and review the “deleting” list before dropping that flag.

File permissions and ownership are only fully preserved if the user running rsync has the privileges to assign them, typically root on both sides. Without that, -a preserves what it can and silently skips the rest unless you add -v to see the warnings.

Sparse files (with real holes on disk, common in disk images and databases) expand to their full logical size by default during transfer. The --sparse flag tells rsync to detect those holes and avoid writing extra zeros, something critical if you’re syncing VM images that are logically hundreds of gigabytes but actually take up a fraction of that on disk.

Finally, broken symlinks or ones pointing outside the synced tree are copied as-is with -a and not followed. If you need to copy the actual content they point to, the flag is -L, and it should be used carefully because it can create infinite loops with circular symlinks.

Comparison: Rsync vs. Other Sync Tools

Tool	When to use it	Advantage	Limitation
rsync	Syncing two file trees that already share a previous version	Real delta transfer, no dedicated server needed	Doesn’t keep a history of previous versions
scp	Copying a single file once	Simplicity, always available alongside SSH	Always retransmits the full file
rclone	Syncing against cloud storage (S3, GCS, Backblaze)	Dozens of supported cloud backends	Real delta only applies between local paths; not all backends support it equally
Borg Backup	Backups with full history and deduplication across snapshots	Each unique block is stored only once, forever	Requires initializing and maintaining your own repository
Unison	Bidirectional sync between two active machines	Resolves conflicts in both directions	More complex configuration than one-way rsync

Deep Dive: The Math Behind the Rolling Checksum

Rsync’s classic weak checksum is a variant of Adler-32 designed to update in constant time. For a window of bytes b1...bn, two sums are calculated: a = (b1 + b2 + ... + bn) mod M and b = (n*b1 + (n-1)*b2 + ... + bn) mod M, and the final checksum combines both as a + b*65536.

The key part is that, when moving the window forward one byte, both a and b are recalculated with one subtraction and one addition, without re-summing all n bytes from scratch:

a_new = a_old - b1 + b_n_plus_1
b_new = b_old - n * b1 + a_new

That O(1) update per position is what makes it feasible to slide the window byte by byte over gigabyte-sized files: without it, calculating the checksum at each of the millions of possible positions would be as costly as not having the algorithm at all.

graph LR
A["Window at bytes 0 to 511"] -. "slides 1 byte" .-> B["Window at bytes 1 to 512"]
B -. "subtracts outgoing byte, adds incoming byte in O(1)" .-> C["Checksum updated without recalculating everything"]

The weak checksum alone isn’t enough because a 32-bit Adler-32 eventually collides between different data windows, especially in files with repetitive patterns like binaries or database dumps. That’s why every weak match gets confirmed with a strong hash, MD5 in the classic protocol or more modern options like xxhash and BLAKE3 via --checksum-choice in recent versions, before accepting the block as a valid copy.

Block size isn’t arbitrary either: by default rsync calculates it as a function roughly approximating the square root of the file size, with a fixed floor and ceiling. Smaller blocks detect finer-grained changes but produce heavier signatures; larger blocks lower the signature overhead but any change within a block forces the whole block to be retransmitted.

This is different from binary diff tools like xdelta or bsdiff: those need both files, old and new, available on the same machine to generate the patch. The rsync algorithm is designed precisely for the case where that isn’t possible: when source and destination live on two different machines and only one side needs to know the other’s full content through its signature, not through the file itself.

Your next step: create two test folders, copy a large file between them, modify a few bytes in the middle with dd or a hex editor, and run rsync -av --stats source/ destination/ to see in the report how many bytes actually traveled over the network.

📖 Summary on Telegram: View summary

Frequently Asked Questions

Does rsync need the full file to exist on both sides before syncing?

The first time, yes. Without a prior signature to compare against, rsync transmits the full file just like any other copy tool. The savings from the delta algorithm kick in starting with the second sync against a destination that already has a previous version of the file.

What does the trailing slash in an rsync path mean?

It changes what gets synced. source/ (with a slash) syncs the contents of that folder into the destination. source (without a slash) syncs the entire folder as a new subdirectory inside the destination.

Does rsync encrypt the data it transmits?

It depends on the transport. With -e ssh, all traffic travels inside an encrypted SSH tunnel. Against a plain rsync daemon (rsync:// protocol without SSH), traffic goes in plain text unless you add your own tunnel with stunnel or a VPN.

How do I know how much data actually traveled over the network during a sync?

Add the --stats flag to the command. The final report separates “Literal data” (bytes sent in full because they didn’t match anything at the destination) from “Matched data” (bytes that were reconstructed from blocks that already existed).

Is rsync good for syncing a database that’s actively receiving writes?

It’s not the ideal tool for that. If the file changes while the signature is being calculated or the delta is being transmitted, the result can end up inconsistent. The usual approach is to stop the service, use a filesystem snapshot mechanism, or dump the database with a dedicated tool before running rsync on the resulting file.

How does rsync differ from Restic or Borg if both avoid retransmitting repeated data?

rsync compares a specific source and destination at the moment the command runs. Restic and Borg maintain a repository with deduplication across all historical snapshots, so a block that was already stored six months ago won’t be written again even if it shows up today in a completely different file.

References

• Rsync technical report: the original paper by Andrew Tridgell and Paul Mackerras describing the rolling checksum algorithm.
• Official rsync website: the project’s documentation, manual, and changelog.
• Rsync on Wikipedia: the project’s history and a comparison with related protocols.
• Borg Backup: documentation for a backup tool that applies deduplication using variable-size content blocks.
• Restic: backup with deduplication and encryption that builds on the same idea of unique blocks.

📱 Do you like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion