SFTP Source - Cloudstic CLI

The SFTP source backs up files from a remote server via SSH File Transfer Protocol. This allows you to back up remote servers without installing Cloudstic on them.

Basic Usage

Back up a remote directory via SFTP:

cloudstic backup -source sftp://myuser@example.com/home/user/data \
  -source-sftp-password "mypassword"

Configuration Options

Required Flags

Flag	Description	Environment Variable
`-source sftp://[user@]host[:port]/<path>`	Specifies SFTP source with connection details	`CLOUDSTIC_SOURCE`

Authentication Flags

Flag	Description	Environment Variable
`-source-sftp-password <pw>`	SSH password	`CLOUDSTIC_SOURCE_SFTP_PASSWORD`
`-source-sftp-key <path>`	Path to SSH private key	`CLOUDSTIC_SOURCE_SFTP_KEY`
`-source-sftp-known-hosts <path>`	Path to known_hosts file	`CLOUDSTIC_SOURCE_SFTP_KNOWN_HOSTS`
`-source-sftp-insecure`	Skip host key validation (INSECURE)	`CLOUDSTIC_SOURCE_SFTP_INSECURE`

Optional Flags

Flag	Description
`-exclude <pattern>`	Exclude pattern (gitignore syntax, repeatable)
`-exclude-file <path>`	Load exclude patterns from file
`-tag <tag>`	Tag to apply to the snapshot (repeatable)
`-dry-run`	Scan source without writing to store

Authentication Methods

Cloudstic supports three SSH authentication methods:

1. Private Key (Recommended)

Use an SSH private key for authentication:

cloudstic backup -source sftp://deploy@web-server.com/var/www \
  -source-sftp-key ~/.ssh/id_rsa

2. Password

Use password authentication:

cloudstic backup -source sftp://myuser@example.com/home/user/files \
  -source-sftp-password "mypassword"

Avoid hardcoding passwords in scripts. Use environment variables or SSH keys instead.

3. SSH Agent (Automatic)

If SSH_AUTH_SOCK is set, Cloudstic automatically uses your SSH agent:

# Start SSH agent and add your key
ssh-add ~/.ssh/id_rsa

# Cloudstic will automatically use the agent
cloudstic backup -source sftp://myuser@example.com/home/user/files

Examples

Backup Web Server

Back up a web server’s document root:

cloudstic backup -source sftp://www-data@web01.example.com/var/www/html \
  -source-sftp-key ~/.ssh/deploy_key \
  -tag "webserver" \
  -tag "production"

Backup with Custom Port

Connect to a non-standard SSH port:

cloudstic backup -source sftp://backup@server.example.com:2222/data \
  -source-sftp-key ~/.ssh/id_ed25519

Backup with Exclusions

Exclude cache and temporary files:

cloudstic backup -source sftp://appuser@app-server.com/home/user/app \
  -source-sftp-password "$SFTP_PASSWORD" \
  -exclude "*.log" \
  -exclude "cache/" \
  -exclude "tmp/"

Multiple Server Backups

Back up multiple servers sequentially:

#!/bin/bash
set -e

SERVERS=("web01" "web02" "web03")

for server in "${SERVERS[@]}"; do
  echo "Backing up $server..."
  cloudstic backup -source sftp://deploy@"$server.example.com"/var/www \
    -source-sftp-key ~/.ssh/deploy_key \
    -tag "webserver" \
    -tag "$server"
done

#!/bin/bash
# Set common SFTP credentials
export CLOUDSTIC_SOURCE_SFTP_KEY=~/.ssh/deploy_key

# Backup multiple hosts
for host in web01 web02 web03; do
  cloudstic backup \
    -source "sftp://deploy@$host.example.com/var/www" \
    -tag "$host"
done

How It Works

Connection Process

Establish SSH connection

Cloudstic connects to the SFTP server using the provided credentials.

Authenticate

Authentication is attempted in order:

Private key (if specified)
Password (if specified)
SSH agent (if available)

Initialize SFTP session

An SFTP session is created over the SSH connection.

Walk directory tree

The remote directory is recursively scanned. File permissions (mode bits) and numeric ownership (uid/gid) are captured from the SFTPv3 ATTRS response. Birth time, file flags, and extended attributes are not available over SFTP.

Stream files

Files are streamed over SFTP as they are backed up.

File Identification

SFTP files are identified by their relative path from the source root:

Source path: /home/user/data
File: /home/user/data/reports/Q1.pdf
File ID: reports/Q1.pdf

Source Information

Each snapshot records:

Type: sftp
Account: user@host (e.g., deploy@web-server.com)
Path: Remote directory path (e.g., /var/www/html)

Security Considerations

SSH Host Key Verification

By default, Cloudstic strictly validates the remote server’s SSH host key against your local known_hosts file (defaulting to ~/.ssh/known_hosts). If the host key is not found or does not match, the connection will fail. For production use, you should ensure the server’s host key is present in your known_hosts file. You can add it using ssh-keyscan:

ssh-keyscan -p 22 example.com >> ~/.ssh/known_hosts

Custom known_hosts file

If your environment uses a non-standard location for known hosts, use the -source-sftp-known-hosts flag:

cloudstic backup -source sftp://user@server/path \
  -source-sftp-known-hosts /etc/ssh/ssh_known_hosts

Insecure Mode (Not Recommended)

To skip host key verification entirely (e.g. for initial testing or internal networks where you trust the route), use the -source-sftp-insecure flag:

cloudstic backup -source sftp://user@server/path -source-sftp-insecure

Using -source-sftp-insecure makes the connection vulnerable to Man-in-the-Middle (MitM) attacks. Only use this in trusted environments.

Credential Management

Best practices for managing SFTP credentials:

Use SSH keys instead of passwords

Generate a dedicated key pair:

ssh-keygen -t ed25519 -f ~/.ssh/cloudstic_backup -C "cloudstic backup"
ssh-copy-id -i ~/.ssh/cloudstic_backup.pub user@server

Use environment variables

Avoid hardcoding credentials:

export CLOUDSTIC_SOURCE_SFTP_PASSWORD="$VAULT_SFTP_PASS"
cloudstic backup -source sftp://user@server/path

Restrict key permissions

Ensure private keys are secure:

chmod 600 ~/.ssh/cloudstic_backup

Use a dedicated backup user

Create a read-only user for backups:

# On the remote server
useradd -m -s /bin/bash backup-user
# Configure read-only access to backup directories

Performance Considerations

Network Overhead

SFTP backups are slower than local backups due to:

Network latency
SSH encryption overhead
Remote filesystem access

Typical Performance

Small files: 100-500 files/second
Large files: Limited by network bandwidth
Network: 10-100 MB/s depending on connection

Optimizing SFTP Backups

Use exclude patterns

Reduce data transfer:

-exclude "*.log" -exclude "cache/" -exclude "tmp/"

Backup during off-peak hours

Schedule backups when network usage is low.

Use compression

Cloudstic automatically compresses data before storage.

Enable packfiles

Bundle small objects to reduce round trips (enabled by default).

Common Use Cases

Web Server Backups

cloudstic backup -source sftp://www-data@web01.example.com/var/www \
  -source-sftp-key ~/.ssh/deploy_key \
  -exclude "cache/" \
  -exclude "*.log" \
  -tag "webserver"

Database Server Backups

# Backup database dumps
cloudstic backup -source sftp://backup@db01.example.com/backup/mysql \
  -source-sftp-key ~/.ssh/backup_key \
  -tag "database" \
  -tag "mysql"

Application Server Backups

cloudstic backup -source sftp://appuser@app-server.com/opt/app \
  -source-sftp-key ~/.ssh/app_key \
  -exclude "logs/" \
  -exclude "temp/" \
  -tag "application"

Troubleshooting

Connection Refused

Error: ssh dial example.com:22: connection refused

Solutions:

Check that SSH service is running on the remote server
Verify the hostname and port are correct
Check firewall rules

Authentication Failed

Error: ssh: unable to authenticate

Solutions:

Verify username and credentials
Check SSH key permissions (must be 600)
Ensure the public key is in ~/.ssh/authorized_keys on the server
Try password authentication if key auth fails

Permission Denied

Error: permission denied

Solutions:

Check that the user has read access to the source directory
Verify the source path is correct
Check filesystem permissions on the remote server

Slow Backups

If SFTP backups are taking too long:

Use exclude patterns to reduce data transfer
Check network bandwidth between source and destination
Consider local backups if possible (install Cloudstic on the remote server)
Run backups during off-peak hours

Environment Variables

Set default SFTP credentials:

export CLOUDSTIC_SOURCE=sftp://backup@example.com/var/www
export CLOUDSTIC_SOURCE_SFTP_KEY=~/.ssh/backup_key

# Now run backup with minimal flags
cloudstic backup

Next Steps

Local Source

Learn about backing up local directories

SSH Key Setup

Set up SSH keys for authentication

Scheduling Backups

Automate backups with cron or systemd

Exclude Patterns

Master exclude pattern syntax

​Basic Usage

​Configuration Options

​Required Flags

​Authentication Flags

​Optional Flags

​Authentication Methods

​1. Private Key (Recommended)

​2. Password

​3. SSH Agent (Automatic)

​Examples

​Backup Web Server

​Backup with Custom Port

​Backup with Exclusions

​Multiple Server Backups

​How It Works

​Connection Process

​File Identification

​Source Information

​Security Considerations

​SSH Host Key Verification

​Custom known_hosts file

​Insecure Mode (Not Recommended)

​Credential Management

​Performance Considerations

​Network Overhead

​Typical Performance

​Optimizing SFTP Backups

​Common Use Cases

​Web Server Backups

​Database Server Backups

​Application Server Backups

​Troubleshooting

​Connection Refused

​Authentication Failed

​Permission Denied

​Slow Backups

​Environment Variables

​Next Steps

Local Source

SSH Key Setup

Scheduling Backups

Exclude Patterns

Basic Usage

Configuration Options

Required Flags

Authentication Flags

Optional Flags

Authentication Methods

1. Private Key (Recommended)

2. Password

3. SSH Agent (Automatic)

Examples

Backup Web Server

Backup with Custom Port

Backup with Exclusions

Multiple Server Backups

How It Works

Connection Process

File Identification

Source Information

Security Considerations

SSH Host Key Verification

Custom known_hosts file

Insecure Mode (Not Recommended)

Credential Management

Performance Considerations

Network Overhead

Typical Performance

Optimizing SFTP Backups

Common Use Cases

Web Server Backups

Database Server Backups

Application Server Backups

Troubleshooting

Connection Refused

Authentication Failed

Permission Denied

Slow Backups

Environment Variables

Next Steps