Also at Deasil Works · txn2 · Plexara
Profiles GitHub · X · LinkedIn
Theme Light · Auto · Dark
imti.co
Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
116 NOTES IN PRINT
FOLIO IX 2018-04-02 · 6 MIN · SHORT-FORM

rSync Files on Interval

Sync media to Raspberry Pi or any ARM SoC.

Diagram · folio ix
stateDiagram-v2
    [*] --> Waiting: cron schedule
    Waiting --> Running: scheduled tick
    Running --> Syncing: rsync over SSH
    Syncing --> Verifying: checksum compare
    Verifying --> Done: match
    Verifying --> Retrying: mismatch
    Retrying --> Syncing: backoff retry
    Done --> Waiting: next interval
    Running --> Failed: SSH or network error
    Failed --> Waiting: log + skip cycle

A recurring requirement for my IOT projects involves keeping a set of files synced with a central server. Many of these projects include media players, kiosk systems, or applications that need frequently updated configuration files, all while entirely unattended, and in most cases unreachable through firewalls. I have one project that alone has 2000+ devices pulling media continuously from an rsync server. Many of these devices are on dodgy wifi networks.

The rsync utility works excellently on Raspberry Pi as well as an assortment of Armbian installed devices. However, writing scripts to manage rsync when it fails, or restarting it on some interval when it finishes, can be a pain. I have a dozen rickety, cobbled-together bash hacks that have somewhat worked in the past. I needed something more stable, portable, and upgradeable.

Open Source: https://github.com/txn2/irsync

§2026 Update

This post is from 2018, and irsync still does what it says: it wraps rsync in a loop with a timeout and restarts it when a transfer stalls. I have not actively developed it since 2018, but it is a thin wrapper around rsync, so it has aged about as well as rsync has, which is to say fine. Two practical updates if you are setting this up today.

The commands moved from docker-compose to docker compose. Compose v1, the hyphenated docker-compose Python tool, reached end of life in 2024 and was removed from Docker in April 2025. The v2 plugin is invoked as docker compose, a space rather than a hyphen, and ships with Docker today. I have updated the demo commands below to match. While you are at it, the version: '3' line at the top of a Compose file is now obsolete and just prints a warning; modern files start straight at services:.

If you would rather not run a container at all, a systemd timer does the same job natively. Most Raspberry Pi OS and Armbian images run systemd now, so you can write a small oneshot service that runs your rsync command, plus a timer that fires it on an interval after the previous run finishes:

# /etc/systemd/system/mediasync.timer
[Unit]
Description=Sync media on an interval

[Timer]
OnBootSec=2min
OnUnitInactiveSec=2min
Persistent=true

[Install]
WantedBy=timers.target

Pair that with a mediasync.service of Type=oneshot that runs rsync .... The OnUnitInactiveSec key gives you the “wait N minutes after the last run finished” behavior that irsync’s interval flag provides, with systemd handling restarts and logging for free. Use whichever fits: irsync if you are already container-first on the device, a systemd timer if you want one less moving part.

Original article below. Everything from here down is the post as originally written. The 2026 Update above covers what’s changed since.

irsync: interval rsync

I built irsync to operate on any (amd64/x86-64 or armhf) system that has Docker running on it.

If we stick with the defaults, the interval duration is 30 seconds and the activity timeout is 2 hours.

An example Docker run command, local to local:

docker run --rm -v "$(pwd)"/data:/data cjimti/irsync \
    -pvrt --delete /data/source/ /data/dest/

An example Docker run, server to local:

docker run --rm -e RSYNC_PASSWORD=password \
    -v "$(pwd)"/data:/data cjimti/irsync \
    -pvrt --delete rsync://[email protected]:873/data/ /data/dest/

Compose example (docker-compose.yml):

# Use for testing Interval rSync
#
# This docker compose file creates server and
# client services, mounting ./data/dest and ./data/source
# respectively.
#
# source: https://hub.docker.com/r/cjimti/irsync/
# irsync docker image: https://hub.docker.com/r/cjimti/irsync/
# rsyncd docker image: https://hub.docker.com/r/cjimti/rsyncd/ (or use any rsync server)
#
# Note: the obsolete `version:` key has been removed for Compose v2.

# Setting up an internal network allows us to use the
# default port and not worry about exposing ports on our
# host.
#
networks:
  sync-net:

services:
  server:
    image: "txn2/rsyncd"
    container_name: rsyncd-server
    environment:
      - USERNAME=test
      - PASSWORD=password
      - VOLUME_PATH=/source_data
      - READ_ONLY=true
      - CHROOT=yes
      - VOLUME_NAME=source_data
      - HOSTS_ALLOW=0.0.0.0/0
    volumes:
      - ./data/source:/source_data
    networks:
      - sync-net
  client:
    image: "txn2/irsync"
    container_name: irsync-client
    environment:
      - RSYNC_PASSWORD=password

    # irsync and rsync options can be intermixed.
    #
    # irsync - has two configuration directives:
    #     --irsync-interval-seconds=SECONDS  number of seconds between intervals
    #     --irsync-timeout-seconds=SECONDS   number of seconds allowed for inactivity
    #
    # rsync has over one hundred options:
    #     see: https://download.samba.org/pub/rsync/rsync.html
    #
    command: [
      "--irsync-interval-seconds=30",
      "-pvrt",
      "--delete",
      "--modify-window=2",
      "rsync://test@rsyncd-server:873/source_data/",
      "/data/"
    ]
    volumes:
      - ./data/dest:/data
    depends_on:
      - server
    networks:
      - sync-net

Say you need to ensure your device (or another server) always has the latest files from the server. However, syncing hundreds or even thousands of files could take hours or days. First, rsync will only grab the data you don’t have, or may have an outdated version of. You can never assume the state of the data on your device. irsync uses its built-in rsync to do the heavy lifting of determining your state versus the server. But rsync is not perfect, and dealing with an unstable network can sometimes cause it to hang or fail. The good news is that, if restarted, rsync will pick up where it left off.

In the IOT device world you can’t sit and watch the transfer and restart it when needed; this is what irsync was built for.

irsync manages the output of its internal rsync and will restart a synchronization process if the timeout exceeds the specified directive. Most of the files I need synchronized are under 200 megabytes, so I set my timeout to 2 hours per file. If a file takes longer than 2 hours to sync, then I assume there is a network or connection failure and let irsync start the process over.

irsync allows me to start a file synchronization on an interval. In other words, I want my device to sync every 2 minutes, but I don’t want to start an rsync every 2 minutes. So if the sync takes 2 hours, then 2 hours and 2 minutes later another synchronization attempt will be made. When my device is up-to-date, these calls are relatively light on the device, and my client knows that only 2 minutes after they update their media it will likely be on its way to their devices.

§Demo

You can run a simple little demo on your local workstation using a docker-compose file I put together.

# create a source and dest directories (mounted from the docker-compose)
mkdir -p ./data/source
mkdir -p ./data/dest

# make a couple of sample files
touch ./data/source/test1.txt
touch ./data/source/test2.txt

# get the docker-compose.yml
curl https://raw.githubusercontent.com/txn2/irsync/master/docker-compose.yml >docker-compose.yml

# run compose in the background (-d flag)
docker compose up -d

# view logs
docker compose logs -f

# drop some more files in the ./data/source directory
# irsync is configured to check every 30 seconds in this demo.

#### Cleanup

# stop containers
# docker compose stop

# remove containers
# docker compose rm

I recorded a video performing the demo above:

§Custom Docker Container

Another useful implementation method involves creating a custom Docker image for each source and destination synchronizations you want to keep running. See the following example Dockerfile:

FROM txn2/irsync
LABEL vendor="txn2.com"
LABEL com.txn2.irsync="https://github.com/txn2/irsync"

# if the rsync server requires a password
ENV RSYNC_PASSWORD=password

# exampe: keep local synchronized with server
# interval default: --irsync-interval-seconds=120
# activity timout default: --irsync-timeout-seconds=7200
CMD ["-pvrt", "--modify-window=30", "--delete", "--exclude='fun'", "rsync://[email protected]:873/data/", "/media"]

Build:

docker build -t custom-sync .

Run:

docker run -d --name custom-sync --restart on-failure \
    -v "$(pwd)"/data:/data custom-sync

§Wear it

If you have read this far, you might be the kind of person to appreciate a high-end fashion statement:

§Resources

· 2018-04-02 ← back to all notes