Server to Server Backup: Working Methods and Restore Tests

You've got two servers, a nightly copy between them, and green job logs. Then you delete the wrong directory, the mirror deletes it too, and you learn you've built replication, not recovery. For most Linux hosts, fix that with a versioned, encrypted server to server backup, a push-only account on the target, and a second copy the source can't reach. Start with rsync with history for file trees, or restic or Proxmox Backup Server for deduplicated snapshots, then prove it with a restore.

Table of Contents

• Pick a Backup Method for Each Server
  • Set RPO and RTO Per Workload
• Build a Push-Only Identity on the Backup Target
  • Restrict the Key to One Directory
• Run Transfers with rsync, scp and rclone
  • Step 1 Dry Run the rsync Job
  • Step 2 Run Live with Hard-Linked History
  • Step 3 Use scp Only for Single Archives
  • Step 4 Copy Off Site with rclone
  • Failure Modes and Rollback
• Encrypted Repositories with restic and Proxmox Backup Server
  • Set Up restic over SFTP
  • Back Up to Proxmox Backup Server
  • Prune and Garbage Collection Are Two Events
  • Verify, Recover from Locks, and Test Safely
• Retention Tiers That Survive a Missed Job
  • What This Looks Like in Production
• Verify Backups and Test Restores Before You Need Them
  • Run Verification Manually First
  • Run a Quarterly Restore Drill
• Confirm the Setup With a First Restore
• Keep the Restore Drill on the Calendar

Pick a Backup Method for Each Server

Use rsync for file trees, restic or Proxmox Backup Server for encrypted deduplicated history, and a second copy in another facility. Pick the tool after you inventory the workload, not before. List web roots, database dumps, VM disks, configuration directories, and application state for every host.

Set RPO and RTO Per Workload

Your recovery point objective is how much data you can lose, and your recovery time objective is how long a restore may take. A database that can't lose more than minutes needs frequent dumps or log shipping. A static web root can often run nightly. Use this recovery point objective guide to set those numbers per workload, then write one line per server covering workload, RPO, RTO, method, and second copy location.

Treat completion and recovery as separate checks. The Computer Weekly survey report found 37% of backup jobs failing and 34% of restores failing, so a successful job doesn't prove a usable restore. Quiesce databases or use native dump and log tools instead of copying live data files and hoping they're consistent.

Backup methods compared by the criteria that change outcomes

Before any command, confirm the prerequisites. You need a Linux source and target, key-based SSH with no password prompt, sudo on both hosts, rsync 3.x on both ends, and rclone on the host that writes the off-site copy. Create the target layout under a backup-owned directory, for example one directory per source server.

Build a Push-Only Identity on the Backup Target

If the same root key can administer the web server and the backup host, whoever owns the web server owns the backups. Recent research reported in Blocks and Files found 96% of ransomware victims reported attackers targeted backup data. Don't ask whether server B can receive server A's files. Ask whether server B can still produce a clean restore after server A's credentials are compromised.

The target hardware below is where that separation has to hold, even when the source is already lost.

Restrict the Key to One Directory

Create a dedicated backup user on the target with no login shell, and install rrsync from the rsync package. Restrict each source key in that user's authorized_keys file to one directory, in write-only mode where your workflow allows it.

command="rrsync -wo /srv/backups/web01",restrict ssh-ed25519 AAAA... web01-backup

The restrict option disables port forwarding, agent forwarding, pty allocation, and user rc. That key can push to one path and do nothing else, so a stolen source key doesn't become an interactive shell on the backup node. Keep administrative keys on a separate account, reached through a jump host with MFA, and never reuse them for scheduled jobs.

Independence test: A copy that shares credentials, a management plane, or an administrator with the source isn't an independent recovery copy.

For repository tools, add append-only mode. Run the restic REST server with its append-only flag, or serve Borg in append-only mode, so the source can add snapshots but can't forget or prune them. Pruning then runs from the target side under a different identity, on your schedule.

This walkthrough shows the target-side account and directory layout before the first production run.

Apply the same test to facilities. The path should run from source server, through the push-only SSH key, to the backup host, then to an isolated off-site copy with separate credentials. Two servers in one Tampa rack protect you from a disk failure, not from a facility event, so the final copy needs different power, network, and administration.

Run Transfers with rsync, scp and rclone

Run every transfer dry first, then cap it, then keep history. Skipping the dry run is how a trailing slash or wrong destination becomes a production incident.

Step 1 Dry Run the rsync Job

rsync -aHAXn --numeric-ids --itemize-changes /var/www/ backup@backup01:/srv/backups/web01/current/

Read the itemized list before you remove the n flag. Confirm the trailing slash gives you the contents layout you expect, because a source path without it copies the directory itself into the destination. If you need the SSH basics first, this guide on how to copy a file via SSH covers the connection pattern these jobs use.

Step 2 Run Live with Hard-Linked History

rsync -aHAX --numeric-ids --bwlimit=40M --link-dest=/srv/backups/web01/prev /var/www/ backup@backup01:/srv/backups/web01/current/

Unchanged files become hard links to the previous run, so each dated directory looks complete but costs only the changed bytes. The flags preserve hard links, ACLs, extended attributes, and numeric ownership, which matters when you restore onto a host where user IDs don't match by name. Rotate only after success: rename current to a dated directory, then point prev at it.

Don't rotate on failure. A half-written current directory that becomes prev poisons the next run's link base.

Step 3 Use scp Only for Single Archives

Use scp for one compressed database dump or a single archive, because it copies the full file every time. It has no delta logic and no resume-friendly history model. For anything recurring or large, rsync or a repository tool will finish inside the window more reliably.

Step 4 Copy Off Site with rclone

Prefer rclone copy over rclone sync for the off-site tier, because sync deletes remote files that no longer exist locally. That behavior is exactly what you don't want after a local deletion or ransomware event. Verify the result instead of trusting the exit message.

rclone copy /srv/backups/ offsite:arphost-backups --bwlimit 10M
rclone check /srv/backups/ offsite:arphost-backups

Failure Modes and Rollback

• Exit code 23: Partial transfer, usually unreadable files or permission gaps. Find the paths in the log and fix ownership or exclusions.
• Exit code 24: Files vanished during the run, common on live upload and cache directories. Snapshot first, or log it and accept it for volatile paths.
• Silent cron failure: A job that never ran looks fine until restore day. Log every run and alert a person on any non-zero exit.

Wrap each cron job in flock so runs never overlap.

30 4 * * * flock -n /var/lock/offsite.lock rclone copy /srv/backups/ offsite:arphost-backups --bwlimit 10M && rclone check /srv/backups/ offsite:arphost-backups

For rollback, leave the previous dated directory untouched. A restore is then an rsync from a known point, and a bad new run never overwrites your last good copy.

Cron slots and bandwidth caps by workload

Encrypted Repositories with restic and Proxmox Backup Server

Choose a repository tool when you want encryption and deduplication enforced by default rather than bolted onto a file copy. Both restic and Proxmox Backup Server keep versioned snapshots, so a deletion today doesn't erase last week's recovery point.

Set Up restic over SFTP

1. Install a current restic release, set the repository password, and keep an offline copy outside the backup target.
2. Initialize the repository over SFTP.

restic -r sftp:backup@backup01:/srv/restic/web01 init

3. Back up the paths that define the service, with a tag so retention and reporting can filter by server.

restic -r sftp:backup@backup01:/srv/restic/web01 backup /var/www /etc --tag web01

4. Apply retention and prune together, from the target side or a trusted admin host if you've made the source append-only.

restic -r sftp:backup@backup01:/srv/restic/web01 forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune

A lost repository password means unrecoverable backups, no matter how healthy the datastore looks. Store it in your password manager and in sealed offline escrow, and test that the escrow copy opens the repository.

Back Up to Proxmox Backup Server

For Proxmox VE guests and file-level backups, send jobs to a Proxmox Backup Server 3.x or 4.x datastore.

proxmox-backup-client backup root.pxar:/ --repository backup@pbs@pbs01:store1 --keyfile /root/pbs-keyfile.json

Keep the keyfile separately from the datastore, with a copy in escrow. Without it, client-side encrypted backups are unreadable by design. Teams that don't want to run that target themselves can use ARPHost Proxmox Backup as a Service as the off-site PBS destination, while keeping the same key custody rules.

Prune and Garbage Collection Are Two Events

The Proxmox Backup Server documentation separates retention from physical deletion: prune removes snapshot metadata including manifests, indexes, blobs, logs, and notes, but chunks remain until garbage collection runs. A successful prune therefore doesn't prove space was reclaimed, which surprises admins watching a full datastore.

proxmox-backup-manager garbage-collection start store1

Run collection inside a controlled maintenance window, then check both the snapshot inventory and actual datastore capacity. Note that prune keeps the last failed backup when it's newer than the last successful one, so an incomplete snapshot lingering after prune isn't automatically a bug.

Verify, Recover from Locks, and Test Safely

Check what the repository thinks it holds before you trust it.

restic -r sftp:backup@backup01:/srv/restic/web01 snapshots
restic -r sftp:backup@backup01:/srv/restic/web01 stats --mode raw-data

If a killed job leaves a stale lock, confirm no restic process is running, then clear it. Running the lock-removal command against a live job can corrupt the state you're trying to protect.

restic -r sftp:backup@backup01:/srv/restic/web01 unlock

For drills and rollback, restore to a scratch path and inspect, never over production paths.

restic -r sftp:backup@backup01:/srv/restic/web01 restore latest --target /tmp/restore-test

Retention Tiers That Survive a Missed Job

Retention is a set of intervals, not a file count, and that distinction bites after a missed job. The Proxmox Backup Server maintenance documentation states that empty intervals don't count, so keep-daily 7 preserves the latest snapshot from the last seven days that contain backups, not seven consecutive calendar days. Weekly rules use ISO weeks beginning Monday and ending Sunday, and only the newest snapshot in each retained interval is kept.

Write the recovery window in days, separately from the snapshot count. If a job fails for three nights, your daily tier now reaches further back than the calendar suggests, which can be helpful or can mask the failure. Alerting on the missed run matters more than the extra coverage.

Retention tiers mapped to the failures they cover

A combined policy covers both fast mistakes and slow discoveries.

restic forget --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune

What This Looks Like in Production

On multi-tenant backup nodes, schedules and retention differ by datastore and backup group, so one tenant's missed job changes only that group's coverage. Keep monthly and yearly tiers on a separate datastore or repository so a pruning mistake on the daily job can't reach them. Make the yearly tier immutable or offline where the platform allows it.

Replication is not a retention tier. A mirror copies deletions and encrypted files almost immediately.

Keep one copy a compromised source can't reach, and one copy outside the primary facility. A hurricane or grid event can take a Florida primary site and an in-building copy together, which is why the off-site tier belongs in a different facility with its own credentials.

Verify Backups and Test Restores Before You Need Them

Verification proves a stored copy can be read, which a green job doesn't. The Proxmox Backup Server configuration documentation provides scheduled verification jobs and controls for skipping already verified snapshots and expiring stale verifications. Jobs can stay green while a later read exposes damaged or unavailable chunks, so verification is the control that catches silent datastore damage.

Run Verification Manually First

Run one verification by hand after the first server to server migration, before you schedule anything.

proxmox-backup-manager verify store1 --read-threads 1 --verify-threads 4 --ignore-verified false

The documented defaults are one reader thread and four verification threads, and both settings range from 1 to 32. Raising concurrency shortens the window but adds disk and CPU pressure on a busy backup node. Raise it on a quiet night, watch iowait, and keep failed-job notifications routed to a person rather than a log nobody reads.

For restic repositories, sample the data regularly and read it fully on a slower cadence.

restic -r sftp:backup@backup01:/srv/restic/web01 check --read-data-subset=5%

Run a Quarterly Restore Drill

1. Restore to an isolated clone: Build a clean VM or bare metal target with no route to production, then restore the newest snapshot into it.
2. Boot and inspect: Check services, DNS, file ownership, and permissions against the source.
3. Check the database natively: Run the database's own integrity check, then run a real application transaction.
4. Time the drill: Record elapsed time against the RTO you set for that workload, including the time spent finding credentials and commands.
5. Log the gaps: Write down anything missing, then fix the job that should have captured it before the next drill.

Testing that a tarball extracts isn't a restore test. The service has to boot, connect, and answer, because that's what you'll need at 3 a.m. after a real failure.

Measure restore success by workload and RTO, not by how many jobs reported success last night.

Restore failures, checks and fixes

Escalate when verification fails on multiple snapshots, when the drill misses its RTO twice, or when the off-site copy can't restore on its own. Those are architecture problems, not tuning problems, and they won't fix themselves next run.

Confirm the Setup With a First Restore

Finish with a one-week proof, not a green status light.

1. Restore in isolation: Run one full restore to a host with no route to production and record elapsed time against the RTO.
2. Confirm the schedules: Check that verification and prune jobs are scheduled, and that failures notify a person, not only a log file.
3. Alert on drift: Watch for missed jobs, sudden size changes, stale agents, and low destination capacity.
4. Prove the second copy: Confirm it uses separate credentials, sits outside the source facility, and restores on its own.
5. Write the runbook: Document where the repository password and PBS keyfile live, plus the exact restore command sequence, where on-call can find it fast.

Record the date of the first successful drill. That date starts your quarterly cadence and gives you a baseline restore time to compare against as data grows.

Keep the Restore Drill on the Calendar

If your hosts run Proxmox VE, use the Proxmox private cloud service for a backup target layout that matches this design, or hand the schedule, verification, and quarterly drills to managed services. Keep the cadence honest with this disaster recovery testing checklist.

Your backup target, off-site copy, and restore drills need hardware that stays reachable when production doesn't, and ARPHost, LLC provides Tampa-based VPS, bare metal, Proxmox private clouds, and managed infrastructure you can build that separation on. Talk to the team about placing your backup server under separate credentials and power, then schedule your first isolated restore before you need it.