> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudstic.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Automating Backups
> Schedule automatic backups with cron, systemd timers, and CI/CD pipelines
This guide shows you how to automate Cloudstic backups using cron jobs, systemd timers, and other scheduling tools.
## Why Automate Backups?
Manual backups are unreliable. Automation ensures:
* **Consistency**: Backups run on schedule, even when you forget
* **Versioning**: Multiple snapshots over time for point-in-time recovery
* **Disaster recovery**: Recent backups are always available
* **Peace of mind**: Set it and forget it
Test your automation thoroughly before relying on it. Verify backups are created and restorable.
## Prerequisites
Before automating:
1. **Initialize your repository**
```bash theme={null}
cloudstic init -password "your passphrase" -add-recovery-key
```
2. **Test manual backup**
```bash theme={null}
cloudstic backup -source local:~/Documents
```
3. **Test manual restore**
```bash theme={null}
cloudstic restore -output test-restore.zip
```
## Using Profiles for Automation (Recommended)
Profiles are the cleanest way to automate backups. Set up once, then your scripts only need the encryption password:
```bash theme={null}
cloudstic store new \
-name prod-s3 \
-uri s3:my-bucket/backups \
-s3-region eu-west-1 \
-password-env BACKUP_PASSWORD
```
```bash theme={null}
# Local documents
cloudstic profile new \
-name documents \
-source local:~/Documents \
-store-ref prod-s3
# Google Drive
cloudstic auth new -name google-main -provider google
cloudstic auth login -name google-main
cloudstic profile new \
-name gdrive \
-source gdrive-changes \
-store-ref prod-s3 \
-auth-ref google-main
```
```bash ~/bin/backup.sh theme={null}
#!/bin/bash
set -euo pipefail
export BACKUP_PASSWORD="your passphrase"
# Back up all profiles
cloudstic backup -all-profiles --no-prompt
# Apply retention
for profile in documents gdrive; do
cloudstic forget -profile "$profile" \
-keep-daily 30 -keep-weekly 8 -keep-monthly 12 \
-prune --no-prompt
done
```
Make it executable:
```bash theme={null}
chmod +x ~/bin/backup.sh
```
```cron theme={null}
0 2 * * * /home/user/bin/backup.sh >> /home/user/logs/backup.log 2>&1
```
Use `--no-prompt` in automation scripts to ensure commands never hang waiting for interactive input. Missing credentials will cause a clear error instead.
Add or remove profiles from `profiles.yaml` without touching the backup script. Use `cloudstic backup -all-profiles` and it picks up changes automatically.
## Setting Up Environment Variables
Store credentials in environment variables to avoid typing them repeatedly.
### Create a Configuration File
```bash ~/.cloudstic_env theme={null}
#!/bin/bash
# Cloudstic configuration
# Storage backend
export CLOUDSTIC_STORE=s3:my-backup-bucket/laptop/
# AWS credentials (for S3)
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export CLOUDSTIC_S3_REGION=us-east-1
# Encryption (choose one)
export CLOUDSTIC_PASSWORD="your passphrase"
# OR for automation:
# export CLOUDSTIC_ENCRYPTION_KEY="64-character-hex-key"
# Source configuration
export CLOUDSTIC_SOURCE=local:$HOME/Documents
```
Secure the file:
```bash theme={null}
chmod 600 ~/.cloudstic_env
```
Never commit this file to version control. Add it to `.gitignore` if your home directory is tracked.
### Load Configuration
In your backup scripts:
```bash theme={null}
#!/bin/bash
source ~/.cloudstic_env
cloudstic backup -source "$CLOUDSTIC_SOURCE"
```
## Automation with Cron (Linux/macOS)
Cron is the standard Unix job scheduler.
### Basic Cron Job
Create `~/bin/backup.sh`:
```bash theme={null}
#!/bin/bash
set -euo pipefail
# Load configuration
source ~/.cloudstic_env
# Log file
LOG_FILE="$HOME/logs/backup.log"
mkdir -p "$(dirname "$LOG_FILE")"
# Run backup
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup" >> "$LOG_FILE"
cloudstic backup -source local:~/Documents >> "$LOG_FILE" 2>&1
if [ $? -eq 0 ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup successful" >> "$LOG_FILE"
else
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup failed" >> "$LOG_FILE"
exit 1
fi
```
Make it executable:
```bash theme={null}
chmod +x ~/bin/backup.sh
```
Run manually to verify:
```bash theme={null}
~/bin/backup.sh
cat ~/logs/backup.log
```
Edit your crontab:
```bash theme={null}
crontab -e
```
Add a cron entry:
```cron theme={null}
# Run backup daily at 2 AM
0 2 * * * /home/user/bin/backup.sh
# Run backup every 6 hours
0 */6 * * * /home/user/bin/backup.sh
# Run backup Monday-Friday at 6 PM
0 18 * * 1-5 /home/user/bin/backup.sh
```
Use absolute paths in cron jobs. Avoid `~` or relative paths.
List your cron jobs:
```bash theme={null}
crontab -l
```
Wait for the scheduled time and check logs:
```bash theme={null}
tail -f ~/logs/backup.log
```
### Cron Schedule Examples
| Schedule | Cron Expression | Description |
| ---------------- | --------------- | ---------------- |
| Daily at 2 AM | `0 2 * * *` | Once per day |
| Every 6 hours | `0 */6 * * *` | Four times daily |
| Weekdays at 6 PM | `0 18 * * 1-5` | Monday-Friday |
| Sundays at noon | `0 12 * * 0` | Weekly |
| First of month | `0 3 1 * *` | Monthly |
Use [crontab.guru](https://crontab.guru) to test cron expressions.
## Automation with Systemd (Linux)
Systemd timers are a modern alternative to cron.
### Create a Systemd Service
Create `~/.config/systemd/user/cloudstic-backup.service`:
```ini theme={null}
[Unit]
Description=Cloudstic backup service
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
EnvironmentFile=%h/.cloudstic_env
ExecStart=/usr/local/bin/cloudstic backup -source local:%h/Documents
StandardOutput=append:%h/logs/backup.log
StandardError=append:%h/logs/backup.log
[Install]
WantedBy=default.target
```
Create `~/.config/systemd/user/cloudstic-backup.timer`:
```ini theme={null}
[Unit]
Description=Cloudstic backup timer
Requires=cloudstic-backup.service
[Timer]
OnCalendar=daily
OnCalendar=02:00
Persistent=true
[Install]
WantedBy=timers.target
```
`Persistent=true` ensures missed runs execute on next boot.
```bash theme={null}
# Reload systemd configuration
systemctl --user daemon-reload
# Enable timer (start on boot)
systemctl --user enable cloudstic-backup.timer
# Start timer now
systemctl --user start cloudstic-backup.timer
```
```bash theme={null}
# Check timer status
systemctl --user status cloudstic-backup.timer
# List all timers
systemctl --user list-timers
# View logs
journalctl --user -u cloudstic-backup.service -f
```
Trigger a backup immediately:
```bash theme={null}
systemctl --user start cloudstic-backup.service
```
### Systemd Timer Schedule Examples
```ini theme={null}
# Daily at 2 AM
OnCalendar=daily
OnCalendar=02:00
# Every 6 hours
OnCalendar=*-*-* 00,06,12,18:00:00
# Weekdays at 6 PM
OnCalendar=Mon..Fri 18:00
# Weekly on Sunday
OnCalendar=Sun *-*-* 12:00:00
# Monthly on the 1st
OnCalendar=*-*-01 03:00:00
```
## Advanced Backup Script
A production-ready script with logging, error handling, and retention:
```bash ~/bin/cloudstic-backup.sh theme={null}
#!/bin/bash
set -euo pipefail
# Configuration
source ~/.cloudstic_env
LOG_DIR="$HOME/logs"
LOG_FILE="$LOG_DIR/backup-$(date +%Y%m%d).log"
mkdir -p "$LOG_DIR"
# Logging function
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}
log "======================================"
log "Starting Cloudstic backup"
# Check connectivity (for remote backends)
if ! ping -c 1 8.8.8.8 > /dev/null 2>&1; then
log "ERROR: No network connectivity"
exit 1
fi
# Run backup
log "Backing up: $CLOUDSTIC_SOURCE"
if cloudstic backup \
-source "$CLOUDSTIC_SOURCE" \
-tag automated \
>> "$LOG_FILE" 2>&1; then
log "Backup completed successfully"
else
log "ERROR: Backup failed with exit code $?"
exit 1
fi
# Apply retention policy
log "Applying retention policy"
if cloudstic forget \
-keep-last 7 \
-keep-daily 30 \
-keep-weekly 8 \
-keep-monthly 12 \
-prune \
>> "$LOG_FILE" 2>&1; then
log "Retention policy applied successfully"
else
log "WARNING: Retention policy failed with exit code $?"
fi
# Weekly integrity check (on Sundays)
if [ "$(date +%u)" = "7" ]; then
log "Running weekly integrity check"
if cloudstic check >> "$LOG_FILE" 2>&1; then
log "Integrity check passed"
else
log "ERROR: Integrity check failed"
exit 1
fi
fi
log "Cloudstic backup completed"
log "======================================"
# Rotate old logs (keep last 30 days)
find "$LOG_DIR" -name "backup-*.log" -mtime +30 -delete
```
Make it executable:
```bash theme={null}
chmod +x ~/bin/cloudstic-backup.sh
```
## Monitoring and Alerting
### Email Notifications on Failure
Send email when backups fail:
```bash ~/bin/backup-with-email.sh theme={null}
#!/bin/bash
source ~/.cloudstic_env
LOG_FILE="$HOME/logs/backup.log"
if ! cloudstic backup -source local:~/Documents >> "$LOG_FILE" 2>&1; then
echo "Cloudstic backup failed. See attached log." | \
mail -s "Backup Failure on $(hostname)" \
-a "$LOG_FILE" \
you@example.com
exit 1
fi
```
Install `mailutils` or `sendmail` for the `mail` command:
```bash theme={null}
sudo apt install mailutils # Debian/Ubuntu
sudo yum install mailx # RHEL/CentOS
brew install mailutils # macOS
```
### Health Check Pings
Use [Healthchecks.io](https://healthchecks.io) or similar services:
```bash theme={null}
#!/bin/bash
source ~/.cloudstic_env
HEALTHCHECK_URL="https://hc-ping.com/your-uuid"
# Start ping
curl -fsS --retry 3 "$HEALTHCHECK_URL/start" > /dev/null
# Run backup
if cloudstic backup -source local:~/Documents; then
# Success ping
curl -fsS --retry 3 "$HEALTHCHECK_URL" > /dev/null
else
# Failure ping
curl -fsS --retry 3 "$HEALTHCHECK_URL/fail" > /dev/null
exit 1
fi
```
## Cloud-Specific Automation
### AWS Lambda Backup
Run backups from Lambda (e.g., backing up EFS to S3):
```python lambda_function.py theme={null}
import os
import subprocess
def lambda_handler(event, context):
os.environ['CLOUDSTIC_STORE'] = 's3:my-backup-bucket'
os.environ['CLOUDSTIC_ENCRYPTION_KEY'] = os.environ['ENCRYPTION_KEY'] # from env var
result = subprocess.run([
'/opt/cloudstic',
'backup',
'-source', 'local:/mnt/efs',
], capture_output=True, text=True)
if result.returncode != 0:
raise Exception(f"Backup failed: {result.stderr}")
return {
'statusCode': 200,
'body': result.stdout
}
```
Package Cloudstic in a Lambda layer and schedule with EventBridge.
### GitHub Actions Backup
Back up repositories to S3:
```yaml .github/workflows/backup.yml theme={null}
name: Backup Repository
on:
schedule:
- cron: '0 2 * * *' # Daily at 2 AM UTC
workflow_dispatch: # Manual trigger
jobs:
backup:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Install Cloudstic
run: |
wget https://github.com/cloudstic/cli/releases/latest/download/cloudstic_linux_amd64.tar.gz
tar xzf cloudstic_linux_amd64.tar.gz
sudo mv cloudstic /usr/local/bin/
- name: Run backup
env:
CLOUDSTIC_STORE: s3:my-backup-bucket
CLOUDSTIC_ENCRYPTION_KEY: ${{ secrets.ENCRYPTION_KEY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
cloudstic backup -source local:.
```
## Backing Up Cloud Sources
### Google Drive Automated Backup
```bash ~/bin/backup-gdrive.sh theme={null}
#!/bin/bash
source ~/.cloudstic_env
# Back up Google Drive using incremental changes API
cloudstic backup \
-source gdrive-changes \
-tag gdrive \
-tag automated
# Apply retention
cloudstic forget -keep-daily 30 -source gdrive -prune
```
### OneDrive Automated Backup
```bash ~/bin/backup-onedrive.sh theme={null}
#!/bin/bash
source ~/.cloudstic_env
# Back up OneDrive
cloudstic backup \
-source onedrive-changes \
-tag onedrive \
-tag automated
# Apply retention
cloudstic forget -keep-daily 30 -source onedrive -prune
```
For cloud sources, the first backup is slow (full scan). Subsequent backups use change APIs and are much faster.
## Structured Errors and Exit Codes for Scripting
Scripts and monitoring systems need machine-readable failure information, not just prose. Cloudstic supports this with `-json` and a small, predictable set of exit codes.
### Exit Codes
| Exit code | Meaning |
| --------- | ------------------------------------------------------------------------------------------------- |
| `0` | Success |
| `1` | Failure (any error other than an interruption) |
| `130` | Interrupted by `SIGINT` or `SIGTERM` (for example, Ctrl-C, or a process manager stopping the job) |
Check `$?` after a run to distinguish "the backup failed" from "the backup was stopped," which matters for alerting: a `130` from a deliberate service restart shouldn't page anyone the way a `1` from a real backup failure should.
```bash theme={null}
cloudstic backup -source local:~/Documents
case $? in
0) echo "backup succeeded" ;;
130) echo "backup was interrupted, not a failure" ;;
*) echo "backup failed" ; exit 1 ;;
esac
```
### JSON Error Output
Pass `-json` to any command and, on failure, Cloudstic writes a single-line JSON object to **stderr** instead of a human-readable message:
```json theme={null}
{"error": "Backup failed: store unavailable"}
```
On success, `-json` writes the command's normal structured result to stdout as before; the JSON error shape is specific to the failure path and always goes to stderr, so you can safely parse stdout and stderr independently.
```bash theme={null}
if ! output=$(cloudstic backup -source local:~/Documents -json 2>error.json); then
message=$(jq -r '.error' error.json)
echo "Backup failed: $message" >&2
exit 1
fi
```
An interrupted run with `-json` also gets a structured error, with the same `130` exit code:
```json theme={null}
{"error": "Interrupted."}
```
Combine `-json` with the exit code check above in monitoring scripts: use the exit code to decide *whether* to alert, and the JSON `error` field to say *why*.
## Troubleshooting Automated Backups
### Cron Job Doesn't Run
1. **Check cron service is running:**
```bash theme={null}
sudo systemctl status cron # Debian/Ubuntu
sudo systemctl status crond # RHEL/CentOS
```
2. **Verify crontab syntax:**
```bash theme={null}
crontab -l
```
3. **Check system logs:**
```bash theme={null}
grep CRON /var/log/syslog
```
4. **Test script manually:**
```bash theme={null}
bash -x ~/bin/backup.sh
```
### Environment Variables Not Loaded
Cron has a minimal environment. Always source your config file:
```bash theme={null}
#!/bin/bash
source ~/.cloudstic_env # Must be first
cloudstic backup ...
```
Or set variables directly in the script:
```bash theme={null}
export CLOUDSTIC_STORE=s3:my-bucket
```
### Backup Fails Silently
Redirect output to a log file:
```cron theme={null}
0 2 * * * /home/user/bin/backup.sh >> /home/user/logs/backup.log 2>&1
```
Or use systemd with `StandardOutput` and `StandardError`.
### "Repository locked" Error
A previous backup may still be running or crashed without releasing the lock.
```bash theme={null}
# Check if backup is running
ps aux | grep cloudstic
# If no process, break the lock
cloudstic break-lock
```
Prevent overlapping runs in your script:
```bash theme={null}
#!/bin/bash
LOCK_FILE="$HOME/.cloudstic-backup.lock"
if [ -f "$LOCK_FILE" ]; then
echo "Backup already running (lock file exists)"
exit 1
fi
touch "$LOCK_FILE"
trap "rm -f $LOCK_FILE" EXIT
# Run backup
cloudstic backup ...
```
## Best Practices
Run manual backups and restores to verify the setup works.
Use health check services or email alerts.
Delete old log files to save space:
```bash theme={null}
find ~/logs -name "*.log" -mtime +30 -delete
```
Add weekly `cloudstic check` runs:
```cron theme={null}
0 3 * * 0 /usr/local/bin/cloudstic check >> ~/logs/check.log 2>&1
```
Clean up old snapshots automatically:
```bash theme={null}
cloudstic backup ...
cloudstic forget -keep-daily 30 -prune
```
Keep notes on:
* Backup schedule
* Retention policy
* Storage credentials location
* Recovery procedure
## Next Steps
Manage automated snapshot lifecycle
Secure credentials for automated backups
Test your automated backups
Automate integrity verification