Custom Source Implementation

Cloudstic supports multiple data sources through a unified Source interface. You can add custom sources to back up data from any system that can provide a file tree.

A custom source lives in its own Go module — you do not need to fork or vendor Cloudstic.The Source contract lives in github.com/cloudstic/cli/pkg/source, which depends on nothing outside the standard library, and the FileMeta / SourceInfo types it is written in are re-exported from the root github.com/cloudstic/cli package. Those re-exports are Go type aliases, so they denote the identical type — a cloudstic.FileMeta you construct in your own module satisfies the interface exactly.Contributing a source upstream is still welcome (see the Contributing Guide), but it is no longer a requirement.

Source Interface

All sources must implement the Source interface defined in pkg/source/interface.go:

type Source interface {
	Walk(ctx context.Context, callback func(core.FileMeta) error) error
	GetFileStream(fileID string) (io.ReadCloser, error)
	Info() core.SourceInfo
	Size(ctx context.Context) (*SourceSize, error)
}

From outside the module you spell those two types through the root package — cloudstic.FileMeta and cloudstic.SourceInfo alias exactly the types the interface is declared with:

import cloudstic "github.com/cloudstic/cli"

func (s *MySource) Walk(ctx context.Context, cb func(cloudstic.FileMeta) error) error
func (s *MySource) Info() cloudstic.SourceInfo

Method Descriptions

Walk

Walk(ctx context.Context, callback func(core.FileMeta) error) error

Enumerates every file and folder in the source. Parent folders MUST be emitted before their children to ensure the HAMT tree structure is built correctly. Parameters:

ctx - Context for cancellation
callback - Function called for each file/folder, receives FileMeta

Returns:

Error if enumeration fails or callback returns an error

GetFileStream

GetFileStream(fileID string) (io.ReadCloser, error)

Returns a readable stream for a file identified by its source-specific fileID. The backup engine calls this for each file that needs to be uploaded. Parameters:

fileID - Source-specific unique identifier (from FileMeta.FileID)

Returns:

io.ReadCloser containing the file data
Error if the file cannot be opened

The caller is responsible for closing the returned reader.

Info

Info() core.SourceInfo

Returns metadata about the source. This is stored in the snapshot and used to:

Find the previous snapshot from the same source (for incremental comparison)
Group snapshots in retention policies (forget --group-by source,account,path)

Returns:

SourceInfo with stable lineage fields (Identity, PathID) and display fields

Size

Size(ctx context.Context) (*SourceSize, error)

Returns the total size of the source. This is used for progress reporting during backup. Returns:

SourceSize with Bytes and Files counts
Error if size cannot be determined (implementation can return approximate values)

Existing Sources

Six source types ship today, each in its own subpackage under pkg/source/ — read one of these before writing your own; they’re the reference implementations this guide’s example is modeled on:

Type	Package	Constructor	Notes
`local`	`pkg/source/local`	`local.New`	Local filesystem; resolves `Identity` from the volume’s partition UUID
`sftp`	`pkg/source/sftp`	`sftp.New`	Remote SFTP server
`gdrive`	`pkg/source/gdrive`	`gdrive.New`	Google Drive, full scan
`gdrive-changes`	`pkg/source/gdrive`	`gdrive.NewChangeSource`	Google Drive, incremental via the Changes API (`IncrementalSource`)
`onedrive`	`pkg/source/onedrive`	`onedrive.New`	OneDrive, full scan
`onedrive-changes`	`pkg/source/onedrive`	`onedrive.NewChangeSource`	OneDrive, incremental via the Delta API (`IncrementalSource`)

Each provider is a separate package so that importing the Source contract does not drag in a provider’s SDK: pkg/source itself has no third-party dependencies, while only pkg/source/gdrive pulls in the Google API client. See docs/sources.md in the repository for each type’s exact FileID/Identity/PathID conventions.

Supporting Types

SourceInfo

type SourceInfo struct {
	Type      string // e.g. "gdrive", "local", "sftp", "onedrive", "gdrive-changes", "onedrive-changes"
	Account   string // friendly display account label (email, hostname, user@host)
	Path      string // friendly display path
	Identity  string // stable container identity for lineage matching
	PathID    string // stable selected-root identity within container
	DriveName string // friendly container label (e.g. "My Drive")
	FsType    string // source filesystem type (e.g. "apfs", "ext4", "sftp")
}

Older snapshots may have been written before Identity/PathID existed (or before a source’s volume-UUID-based identity was introduced — see RFC 0005 and RFC 0009). A local source resolves Identity from the detected partition UUID (overridable via CLOUDSTIC_VOLUME_UUID / -volume-uuid, for cross-machine incremental backup of portable drives), falling back to the hostname; PathID is the path relative to the volume’s mount point, so it stays stable if the drive is remounted elsewhere.

Snapshots from the same source are matched against the previous snapshot in three passes, falling back only when the more specific match is unavailable:

Type + Identity + PathID (preferred)
Type + Identity + Path (bridge fallback, for snapshots written before PathID existed)
Type + Account + Path (legacy fallback, for snapshots written before source identity was tracked at all)

FileMeta

type FileMeta struct {
	Version     int                    // Always 1
	FileID      string                 // Source-specific unique ID (HAMT key)
	Name        string                 // Display name
	Type        FileType               // "file" or "folder"
	Parents     []string               // Parent references: raw source FileIDs, during Walk
	Paths       []string               // Optional: a source may set this as a display-path hint; the engine recomputes it from the HAMT tree regardless
	ContentHash string                 // SHA-256 of file content (if available from source)
	ContentRef  string                 // Populated by the engine; leave empty when emitting
	Size        int64                  // File size in bytes
	Mtime       int64                  // Last modified time (Unix timestamp)
	Owner       string                 // Owner identifier (display)
	Extra       map[string]interface{} // Source-specific metadata
	Mode        uint32                 // POSIX permission bits (st_mode & 0xFFF), if the source has them
	Uid         uint32                 // POSIX user ID
	Gid         uint32                 // POSIX group ID
	Btime       int64                  // birth/creation time, Unix seconds; 0 = not available
	Flags       uint32                 // per-file flags (chflags / FS_IOC_GETFLAGS)
	Xattrs      map[string][]byte      // extended attributes: name -> raw bytes
}

Important fields:

FileID - Must be stable and unique within the source. Used as the HAMT key.
Type - Must be "file" or "folder"
Parents - List of parent FileIDs. Can be empty for root items.
ContentHash - If your source provides checksums (like Google Drive), include them here to avoid re-downloading unchanged files. Leave empty if not available.
Mode/Uid/Gid/Btime/Flags/Xattrs - POSIX-style attributes added by RFC 0004, used for permission/ownership preservation on restore. Populate them if your source exposes them (the local source does on macOS/Linux); leave the zero value otherwise — a source with no concept of Unix permissions (Google Drive, OneDrive) should simply not set them.
ContentRef - Written by the backup engine itself; leave it at its zero value when emitting FileMeta from Walk/WalkChanges.

SourceSize

type SourceSize struct {
	Bytes int64 `json:"bytes"`
	Files int64 `json:"files"`
}

Example: Simple HTTP Source

Here’s a complete example implementing a source that backs up files from an HTTP server. It lives in your own module — the only Cloudstic imports are the root package (for the FileMeta / SourceInfo aliases) and pkg/source (for SourceSize):

// go.mod: module example.com/mybackup
package httpsource

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"time"

	cloudstic "github.com/cloudstic/cli"
	"github.com/cloudstic/cli/pkg/source"
)

// HTTPSource backs up files from an HTTP API that returns a file listing.
type HTTPSource struct {
	baseURL string
	client  *http.Client
}

func NewHTTPSource(baseURL string) *HTTPSource {
	return &HTTPSource{
		baseURL: baseURL,
		client: &http.Client{
			Timeout: 30 * time.Second,
		},
	}
}

// Walk fetches the file listing from /api/files and emits each entry.
func (s *HTTPSource) Walk(ctx context.Context, callback func(cloudstic.FileMeta) error) error {
	resp, err := s.client.Get(s.baseURL + "/api/files")
	if err != nil {
		return fmt.Errorf("fetch file list: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return fmt.Errorf("server returned %d", resp.StatusCode)
	}

	var files []struct {
		ID       string `json:"id"`
		Name     string `json:"name"`
		Size     int64  `json:"size"`
		Modified int64  `json:"modified"` // Unix timestamp
		IsDir    bool   `json:"is_dir"`
		Parent   string `json:"parent"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&files); err != nil {
		return fmt.Errorf("parse file list: %w", err)
	}

	// Emit folders first, then files (ensures parents before children)
	for _, f := range files {
		if !f.IsDir {
			continue
		}
		meta := cloudstic.FileMeta{
			Version: 1,
			FileID:  f.ID,
			Name:    f.Name,
			Type:    cloudstic.FileTypeFolder,
			Mtime:   f.Modified,
		}
		if f.Parent != "" {
			meta.Parents = []string{f.Parent}
		}
		if err := callback(meta); err != nil {
			return err
		}
	}

	// Emit files
	for _, f := range files {
		if f.IsDir {
			continue
		}
		meta := cloudstic.FileMeta{
			Version: 1,
			FileID:  f.ID,
			Name:    f.Name,
			Type:    cloudstic.FileTypeFile,
			Size:    f.Size,
			Mtime:   f.Modified,
		}
		if f.Parent != "" {
			meta.Parents = []string{f.Parent}
		}
		if err := callback(meta); err != nil {
			return err
		}
	}

	return nil
}

// GetFileStream downloads the file content from /api/files/{id}.
func (s *HTTPSource) GetFileStream(fileID string) (io.ReadCloser, error) {
	resp, err := s.client.Get(s.baseURL + "/api/files/" + fileID)
	if err != nil {
		return nil, fmt.Errorf("fetch file %s: %w", fileID, err)
	}

	if resp.StatusCode != http.StatusOK {
		resp.Body.Close()
		return nil, fmt.Errorf("server returned %d for file %s", resp.StatusCode, fileID)
	}

	return resp.Body, nil
}

// Info returns source metadata.
func (s *HTTPSource) Info() cloudstic.SourceInfo {
	return cloudstic.SourceInfo{
		Type:    "http",
		Account: s.baseURL,
		Path:    "/",
	}
}

// Size returns the total size by summing all files.
func (s *HTTPSource) Size(ctx context.Context) (*source.SourceSize, error) {
	var totalBytes, totalFiles int64

	err := s.Walk(ctx, func(meta cloudstic.FileMeta) error {
		if meta.Type == cloudstic.FileTypeFile {
			totalFiles++
			totalBytes += meta.Size
		}
		return nil
	})

	if err != nil {
		return nil, err
	}

	return &source.SourceSize{
		Bytes: totalBytes,
		Files: totalFiles,
	}, nil
}

Using the Custom Source

From code elsewhere in the module (or, once merged, from the -source http CLI wiring described below):

src := httpsource.New("https://example.com")
result, err := client.Backup(ctx, src)

Incremental Source Interface

For sources that support delta-based backups (like Google Drive Changes API or OneDrive Delta API), implement the IncrementalSource interface:

type IncrementalSource interface {
	Source
	GetStartPageToken() (string, error)
	WalkChanges(ctx context.Context, token string, callback func(FileChange) error) (newToken string, err error)
}

IncrementalSource Methods

GetStartPageToken

GetStartPageToken() (string, error)

Returns an opaque token representing the current head of the change stream. The engine calls this before the first full Walk to capture the baseline state. Returns:

Token string to persist in the snapshot
Error if token cannot be retrieved

WalkChanges

WalkChanges(ctx context.Context, token string, callback func(FileChange) error) (newToken string, err error)

Emits only the entries that changed since token. Returns the new token to persist for the next run. Parameters:

ctx - Context for cancellation
token - Token from the previous snapshot (or from GetStartPageToken)
callback - Function called for each change

Returns:

newToken - New token to store in the snapshot
err - Error if change enumeration fails

FileChange

type FileChange struct {
	Type ChangeType // "upsert" or "delete"
	Meta cloudstic.FileMeta
}

type ChangeType string

const (
	ChangeUpsert ChangeType = "upsert" // File or folder created/modified
	ChangeDelete ChangeType = "delete" // File or folder removed
)

For ChangeDelete, only Meta.FileID is required. All other fields can be empty.

Example: Incremental HTTP Source

Extending the previous HTTP source to support incremental backups:

// HTTPChangeSource adds incremental support to HTTPSource.
type HTTPChangeSource struct {
	*HTTPSource
}

func NewHTTPChangeSource(baseURL string) *HTTPChangeSource {
	return &HTTPChangeSource{
		HTTPSource: NewHTTPSource(baseURL),
	}
}

// GetStartPageToken fetches the current change token from the server.
func (s *HTTPChangeSource) GetStartPageToken() (string, error) {
	resp, err := s.client.Get(s.baseURL + "/api/changes/token")
	if err != nil {
		return "", fmt.Errorf("fetch change token: %w", err)
	}
	defer resp.Body.Close()

	var result struct {
		Token string `json:"token"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		return "", fmt.Errorf("parse change token: %w", err)
	}

	return result.Token, nil
}

// WalkChanges fetches changes since the given token.
func (s *HTTPChangeSource) WalkChanges(
	ctx context.Context,
	token string,
	callback func(FileChange) error,
) (string, error) {
	resp, err := s.client.Get(s.baseURL + "/api/changes?since=" + token)
	if err != nil {
		return "", fmt.Errorf("fetch changes: %w", err)
	}
	defer resp.Body.Close()

	var result struct {
		Changes []struct {
			Type     string `json:"type"` // "upsert" or "delete"
			ID       string `json:"id"`
			Name     string `json:"name"`
			Size     int64  `json:"size"`
			Modified int64  `json:"modified"`
			IsDir    bool   `json:"is_dir"`
			Parent   string `json:"parent"`
		} `json:"changes"`
		NewToken string `json:"new_token"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		return "", fmt.Errorf("parse changes: %w", err)
	}

	// Emit folder changes first (upserts only, deletes can be in any order)
	for _, c := range result.Changes {
		if c.Type != "upsert" || !c.IsDir {
			continue
		}
		change := FileChange{
			Type: ChangeUpsert,
			Meta: cloudstic.FileMeta{
				Version: 1,
				FileID:  c.ID,
				Name:    c.Name,
				Type:    cloudstic.FileTypeFolder,
				Mtime:   c.Modified,
			},
		}
		if c.Parent != "" {
			change.Meta.Parents = []string{c.Parent}
		}
		if err := callback(change); err != nil {
			return "", err
		}
	}

	// Emit file changes and deletions
	for _, c := range result.Changes {
		if c.Type == "upsert" && c.IsDir {
			continue // Already emitted above
		}

		var change FileChange
		if c.Type == "delete" {
			change = FileChange{
				Type: ChangeDelete,
				Meta: cloudstic.FileMeta{
					FileID: c.ID,
				},
			}
		} else {
			change = FileChange{
				Type: ChangeUpsert,
				Meta: cloudstic.FileMeta{
					Version: 1,
					FileID:  c.ID,
					Name:    c.Name,
					Type:    cloudstic.FileTypeFile,
					Size:    c.Size,
					Mtime:   c.Modified,
				},
			}
			if c.Parent != "" {
				change.Meta.Parents = []string{c.Parent}
			}
		}

		if err := callback(change); err != nil {
			return "", err
		}
	}

	return result.NewToken, nil
}

How the Engine Uses Incremental Sources

First backup - Engine calls GetStartPageToken(), then Walk() for full scan, stores token in snapshot
Subsequent backups - Engine calls WalkChanges(token) with the stored token, processes only the delta
Token updated - New token from WalkChanges is stored in the snapshot for the next run

If WalkChanges fails (e.g., token expired), the engine automatically falls back to a full Walk.

Implementation Guidelines

Parent-Child Ordering

The most critical requirement is that parents must be emitted before children. This ensures the HAMT tree structure is built correctly.

// BAD: Children before parents
callback(cloudstic.FileMeta{FileID: "file.txt", Parents: []string{"folder1"}})
callback(cloudstic.FileMeta{FileID: "folder1", Type: cloudstic.FileTypeFolder})

// GOOD: Parents before children
callback(cloudstic.FileMeta{FileID: "folder1", Type: cloudstic.FileTypeFolder})
callback(cloudstic.FileMeta{FileID: "file.txt", Parents: []string{"folder1"}})

For sources with complex hierarchies, use topological sorting to ensure correct ordering.

FileID Stability

FileID must be stable across backups. If a file’s FileID changes, the engine treats it as a deletion + addition rather than a modification.

// GOOD: Use stable IDs from the source API
FileID: googleDriveFileID // Stable across renames

// BAD: Use paths that change when files move
FileID: "/home/user/documents/report.pdf" // Changes if moved

ContentHash Optimization

If your source provides checksums (like Google Drive’s SHA-256 or OneDrive’s QuickXorHash), include them in ContentHash:

meta.ContentHash = file.SHA256Checksum // From source API

This allows the engine to skip downloading unchanged files during incremental backups.

Error Handling

Return descriptive errors that help users diagnose issues:

if resp.StatusCode == 401 {
	return fmt.Errorf("authentication failed: check your API credentials")
}
if resp.StatusCode == 429 {
	return fmt.Errorf("rate limit exceeded: try again later")
}
return fmt.Errorf("fetch file list: HTTP %d: %s", resp.StatusCode, resp.Status)

Context Cancellation

Respect the context.Context passed to Walk, WalkChanges, and Size:

func (s *MySource) Walk(ctx context.Context, callback func(cloudstic.FileMeta) error) error {
	for _, file := range files {
		select {
		case <-ctx.Done():
			return ctx.Err()
		default:
		}

		if err := callback(convertFileMeta(file)); err != nil {
			return err
		}
	}
	return nil
}

Testing Your Source

Create a simple test to verify your source implementation:

func TestHTTPSource(t *testing.T) {
	source := NewHTTPSource("https://example.com")

	// Test Info
	info := source.Info()
	if info.Type != "http" {
		t.Errorf("expected type 'http', got %q", info.Type)
	}

	// Test Walk
	ctx := context.Background()
	var files []cloudstic.FileMeta
	err := source.Walk(ctx, func(meta cloudstic.FileMeta) error {
		files = append(files, meta)
		return nil
	})
	if err != nil {
		t.Fatal(err)
	}

	if len(files) == 0 {
		t.Fatal("expected files, got none")
	}

	// Test GetFileStream
	for _, f := range files {
		if f.Type != cloudstic.FileTypeFile {
			continue
		}
		stream, err := source.GetFileStream(f.FileID)
		if err != nil {
			t.Errorf("GetFileStream(%s): %v", f.FileID, err)
			continue
		}
		stream.Close()
	}
}

Registering Your Source

The CLI doesn’t dispatch source types through a hand-written flag parser. -source is a URI whose scheme selects the source: parseSourceURI (cmd/cloudstic/storeuri.go) parses the scheme and path/host out of the raw string (e.g. local:./documents, sftp://user@host/path, gdrive://Drive Name/path), and initSource (cmd/cloudstic/cmd_backup.go) switches on uri.scheme to construct the matching source.Source with its functional options:

func initSource(ctx context.Context, opts initSourceOptions) (source.Source, error) {
	uri, err := parseSourceURI(opts.sourceURI)
	if err != nil {
		return nil, err
	}

	switch uri.scheme {
	case "local":
		return source.NewLocalSource(uri.path, source.WithLocalExcludePatterns(opts.excludePatterns)), nil
	case "sftp":
		return source.NewSFTPSource(uri.host, sftpSourceOpts(opts.sourceSFTP, uri)...)
	// ... gdrive, gdrive-changes, onedrive, onedrive-changes ...
	case "http":
		return httpsource.New(uri.host), nil
	default:
		return nil, fmt.Errorf("unknown source scheme %q", uri.scheme)
	}
}

To add http as a real scheme, extend parseSourceURI’s scheme switch to recognize it (so http://example.com parses into a sourceURIParts{scheme: "http", host: "example.com"}) and add the corresponding case "http": to initSource. Any source-specific flags (credentials, exclude patterns, and so on) are declared like any other flag — see declareBackupArgs in cmd/cloudstic/cmd_backup.go and the CLI Integration section of the Contributing Guide — not parsed ad hoc inside initSource. Then use it:

cloudstic backup -source http://example.com

Architecture

Development

Custom Source Implementation

Source Interface

Method Descriptions

Walk

GetFileStream

Info

Size

Existing Sources

Supporting Types

SourceInfo

FileMeta

SourceSize

Example: Simple HTTP Source

Using the Custom Source

Incremental Source Interface

IncrementalSource Methods

GetStartPageToken

WalkChanges

FileChange

Example: Incremental HTTP Source

How the Engine Uses Incremental Sources

Implementation Guidelines

Parent-Child Ordering

FileID Stability

ContentHash Optimization

Error Handling

Context Cancellation

Testing Your Source

Registering Your Source

​Source Interface

​Method Descriptions

​Walk

​GetFileStream

​Info

​Size

​Existing Sources

​Supporting Types

​SourceInfo

​FileMeta

​SourceSize

​Example: Simple HTTP Source

​Using the Custom Source

​Incremental Source Interface

​IncrementalSource Methods

​GetStartPageToken

​WalkChanges

​FileChange

​Example: Incremental HTTP Source

​How the Engine Uses Incremental Sources

​Implementation Guidelines

​Parent-Child Ordering

​FileID Stability

​ContentHash Optimization

​Error Handling

​Context Cancellation

​Testing Your Source

​Registering Your Source

Source Interface

Method Descriptions

Walk

GetFileStream

Info

Size

Existing Sources

Supporting Types

SourceInfo

FileMeta

SourceSize

Example: Simple HTTP Source

Using the Custom Source

Incremental Source Interface

IncrementalSource Methods

GetStartPageToken

WalkChanges

FileChange

Example: Incremental HTTP Source

How the Engine Uses Incremental Sources

Implementation Guidelines

Parent-Child Ordering

FileID Stability

ContentHash Optimization

Error Handling

Context Cancellation

Testing Your Source

Registering Your Source