> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudstic.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Source Implementation

> Implement custom backup sources for Cloudstic

Cloudstic supports multiple data sources through a unified `Source` interface. You can add custom sources to back up data from any system that can provide a file tree.

<Note>
  A custom source lives in **its own Go module** — you do not need to fork or vendor Cloudstic.

  The `Source` contract lives in `github.com/cloudstic/cli/pkg/source`, which depends on nothing outside the standard library, and the `FileMeta` / `SourceInfo` types it is written in are re-exported from the root `github.com/cloudstic/cli` package. Those re-exports are Go *type aliases*, so they denote the identical type — a `cloudstic.FileMeta` you construct in your own module satisfies the interface exactly.

  Contributing a source upstream is still welcome (see the [Contributing Guide](/advanced/contributing)), but it is no longer a requirement.
</Note>

## Source Interface

All sources must implement the `Source` interface defined in `pkg/source/interface.go`:

```go theme={null}
type Source interface {
	Walk(ctx context.Context, callback func(core.FileMeta) error) error
	GetFileStream(fileID string) (io.ReadCloser, error)
	Info() core.SourceInfo
	Size(ctx context.Context) (*SourceSize, error)
}
```

From outside the module you spell those two types through the root package — `cloudstic.FileMeta` and `cloudstic.SourceInfo` alias exactly the types the interface is declared with:

```go theme={null}
import cloudstic "github.com/cloudstic/cli"

func (s *MySource) Walk(ctx context.Context, cb func(cloudstic.FileMeta) error) error
func (s *MySource) Info() cloudstic.SourceInfo
```

### Method Descriptions

#### Walk

```go theme={null}
Walk(ctx context.Context, callback func(core.FileMeta) error) error
```

Enumerates every file and folder in the source. **Parent folders MUST be emitted before their children** to ensure the HAMT tree structure is built correctly.

**Parameters:**

* `ctx` - Context for cancellation
* `callback` - Function called for each file/folder, receives `FileMeta`

**Returns:**

* Error if enumeration fails or callback returns an error

#### GetFileStream

```go theme={null}
GetFileStream(fileID string) (io.ReadCloser, error)
```

Returns a readable stream for a file identified by its source-specific `fileID`. The backup engine calls this for each file that needs to be uploaded.

**Parameters:**

* `fileID` - Source-specific unique identifier (from `FileMeta.FileID`)

**Returns:**

* `io.ReadCloser` containing the file data
* Error if the file cannot be opened

<Note>
  The caller is responsible for closing the returned reader.
</Note>

#### Info

```go theme={null}
Info() core.SourceInfo
```

Returns metadata about the source. This is stored in the snapshot and used to:

* Find the previous snapshot from the same source (for incremental comparison)
* Group snapshots in retention policies (`forget --group-by source,account,path`)

**Returns:**

* `SourceInfo` with stable lineage fields (`Identity`, `PathID`) and display fields

#### Size

```go theme={null}
Size(ctx context.Context) (*SourceSize, error)
```

Returns the total size of the source. This is used for progress reporting during backup.

**Returns:**

* `SourceSize` with `Bytes` and `Files` counts
* Error if size cannot be determined (implementation can return approximate values)

## Existing Sources

Six source types ship today, each in its own subpackage under `pkg/source/` — read one of these before writing your own; they're the reference implementations this guide's example is modeled on:

| Type               | Package               | Constructor                | Notes                                                                  |
| ------------------ | --------------------- | -------------------------- | ---------------------------------------------------------------------- |
| `local`            | `pkg/source/local`    | `local.New`                | Local filesystem; resolves `Identity` from the volume's partition UUID |
| `sftp`             | `pkg/source/sftp`     | `sftp.New`                 | Remote SFTP server                                                     |
| `gdrive`           | `pkg/source/gdrive`   | `gdrive.New`               | Google Drive, full scan                                                |
| `gdrive-changes`   | `pkg/source/gdrive`   | `gdrive.NewChangeSource`   | Google Drive, incremental via the Changes API (`IncrementalSource`)    |
| `onedrive`         | `pkg/source/onedrive` | `onedrive.New`             | OneDrive, full scan                                                    |
| `onedrive-changes` | `pkg/source/onedrive` | `onedrive.NewChangeSource` | OneDrive, incremental via the Delta API (`IncrementalSource`)          |

Each provider is a separate package so that importing the `Source` contract does not drag in a provider's SDK: `pkg/source` itself has no third-party dependencies, while only `pkg/source/gdrive` pulls in the Google API client.

See `docs/sources.md` in the repository for each type's exact `FileID`/`Identity`/`PathID` conventions.

## Supporting Types

### SourceInfo

```go theme={null}
type SourceInfo struct {
	Type      string // e.g. "gdrive", "local", "sftp", "onedrive", "gdrive-changes", "onedrive-changes"
	Account   string // friendly display account label (email, hostname, user@host)
	Path      string // friendly display path
	Identity  string // stable container identity for lineage matching
	PathID    string // stable selected-root identity within container
	DriveName string // friendly container label (e.g. "My Drive")
	FsType    string // source filesystem type (e.g. "apfs", "ext4", "sftp")
}
```

<Note>
  Older snapshots may have been written before `Identity`/`PathID` existed (or before a source's volume-UUID-based identity was introduced — see [RFC 0005](https://github.com/Cloudstic/cli/blob/main/rfcs/0005-portable-drive-identity.md) and [RFC 0009](https://github.com/Cloudstic/cli/blob/main/rfcs/0009-unified-source-identity.md)). A `local` source resolves `Identity` from the detected partition UUID (overridable via `CLOUDSTIC_VOLUME_UUID` / `-volume-uuid`, for cross-machine incremental backup of portable drives), falling back to the hostname; `PathID` is the path relative to the volume's mount point, so it stays stable if the drive is remounted elsewhere.
</Note>

Snapshots from the same source are matched against the previous snapshot in three passes, falling back only when the more specific match is unavailable:

1. `Type + Identity + PathID` (preferred)
2. `Type + Identity + Path` (bridge fallback, for snapshots written before `PathID` existed)
3. `Type + Account + Path` (legacy fallback, for snapshots written before source identity was tracked at all)

### FileMeta

```go theme={null}
type FileMeta struct {
	Version     int                    // Always 1
	FileID      string                 // Source-specific unique ID (HAMT key)
	Name        string                 // Display name
	Type        FileType               // "file" or "folder"
	Parents     []string               // Parent references: raw source FileIDs, during Walk
	Paths       []string               // Optional: a source may set this as a display-path hint; the engine recomputes it from the HAMT tree regardless
	ContentHash string                 // SHA-256 of file content (if available from source)
	ContentRef  string                 // Populated by the engine; leave empty when emitting
	Size        int64                  // File size in bytes
	Mtime       int64                  // Last modified time (Unix timestamp)
	Owner       string                 // Owner identifier (display)
	Extra       map[string]interface{} // Source-specific metadata
	Mode        uint32                 // POSIX permission bits (st_mode & 0xFFF), if the source has them
	Uid         uint32                 // POSIX user ID
	Gid         uint32                 // POSIX group ID
	Btime       int64                  // birth/creation time, Unix seconds; 0 = not available
	Flags       uint32                 // per-file flags (chflags / FS_IOC_GETFLAGS)
	Xattrs      map[string][]byte      // extended attributes: name -> raw bytes
}
```

**Important fields:**

* `FileID` - Must be stable and unique within the source. Used as the HAMT key.
* `Type` - Must be `"file"` or `"folder"`
* `Parents` - List of parent `FileID`s. Can be empty for root items.
* `ContentHash` - If your source provides checksums (like Google Drive), include them here to avoid re-downloading unchanged files. Leave empty if not available.
* `Mode`/`Uid`/`Gid`/`Btime`/`Flags`/`Xattrs` - POSIX-style attributes added by [RFC 0004](https://github.com/Cloudstic/cli/blob/main/rfcs/0004-extended-file-attributes.md), used for permission/ownership preservation on restore. Populate them if your source exposes them (the `local` source does on macOS/Linux); leave the zero value otherwise — a source with no concept of Unix permissions (Google Drive, OneDrive) should simply not set them.
* `ContentRef` - Written by the backup engine itself; leave it at its zero value when emitting `FileMeta` from `Walk`/`WalkChanges`.

### SourceSize

```go theme={null}
type SourceSize struct {
	Bytes int64 `json:"bytes"`
	Files int64 `json:"files"`
}
```

## Example: Simple HTTP Source

Here's a complete example implementing a source that backs up files from an HTTP server. It lives in **your own module** — the only Cloudstic imports are the root package (for the `FileMeta` / `SourceInfo` aliases) and `pkg/source` (for `SourceSize`):

```go theme={null}
// go.mod: module example.com/mybackup
package httpsource

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"time"

	cloudstic "github.com/cloudstic/cli"
	"github.com/cloudstic/cli/pkg/source"
)

// HTTPSource backs up files from an HTTP API that returns a file listing.
type HTTPSource struct {
	baseURL string
	client  *http.Client
}

func NewHTTPSource(baseURL string) *HTTPSource {
	return &HTTPSource{
		baseURL: baseURL,
		client: &http.Client{
			Timeout: 30 * time.Second,
		},
	}
}

// Walk fetches the file listing from /api/files and emits each entry.
func (s *HTTPSource) Walk(ctx context.Context, callback func(cloudstic.FileMeta) error) error {
	resp, err := s.client.Get(s.baseURL + "/api/files")
	if err != nil {
		return fmt.Errorf("fetch file list: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return fmt.Errorf("server returned %d", resp.StatusCode)
	}

	var files []struct {
		ID       string `json:"id"`
		Name     string `json:"name"`
		Size     int64  `json:"size"`
		Modified int64  `json:"modified"` // Unix timestamp
		IsDir    bool   `json:"is_dir"`
		Parent   string `json:"parent"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&files); err != nil {
		return fmt.Errorf("parse file list: %w", err)
	}

	// Emit folders first, then files (ensures parents before children)
	for _, f := range files {
		if !f.IsDir {
			continue
		}
		meta := cloudstic.FileMeta{
			Version: 1,
			FileID:  f.ID,
			Name:    f.Name,
			Type:    cloudstic.FileTypeFolder,
			Mtime:   f.Modified,
		}
		if f.Parent != "" {
			meta.Parents = []string{f.Parent}
		}
		if err := callback(meta); err != nil {
			return err
		}
	}

	// Emit files
	for _, f := range files {
		if f.IsDir {
			continue
		}
		meta := cloudstic.FileMeta{
			Version: 1,
			FileID:  f.ID,
			Name:    f.Name,
			Type:    cloudstic.FileTypeFile,
			Size:    f.Size,
			Mtime:   f.Modified,
		}
		if f.Parent != "" {
			meta.Parents = []string{f.Parent}
		}
		if err := callback(meta); err != nil {
			return err
		}
	}

	return nil
}

// GetFileStream downloads the file content from /api/files/{id}.
func (s *HTTPSource) GetFileStream(fileID string) (io.ReadCloser, error) {
	resp, err := s.client.Get(s.baseURL + "/api/files/" + fileID)
	if err != nil {
		return nil, fmt.Errorf("fetch file %s: %w", fileID, err)
	}

	if resp.StatusCode != http.StatusOK {
		resp.Body.Close()
		return nil, fmt.Errorf("server returned %d for file %s", resp.StatusCode, fileID)
	}

	return resp.Body, nil
}

// Info returns source metadata.
func (s *HTTPSource) Info() cloudstic.SourceInfo {
	return cloudstic.SourceInfo{
		Type:    "http",
		Account: s.baseURL,
		Path:    "/",
	}
}

// Size returns the total size by summing all files.
func (s *HTTPSource) Size(ctx context.Context) (*source.SourceSize, error) {
	var totalBytes, totalFiles int64

	err := s.Walk(ctx, func(meta cloudstic.FileMeta) error {
		if meta.Type == cloudstic.FileTypeFile {
			totalFiles++
			totalBytes += meta.Size
		}
		return nil
	})

	if err != nil {
		return nil, err
	}

	return &source.SourceSize{
		Bytes: totalBytes,
		Files: totalFiles,
	}, nil
}
```

### Using the Custom Source

From code elsewhere in the module (or, once merged, from the `-source http` CLI wiring described below):

```go theme={null}
src := httpsource.New("https://example.com")
result, err := client.Backup(ctx, src)
```

## Incremental Source Interface

For sources that support delta-based backups (like Google Drive Changes API or OneDrive Delta API), implement the `IncrementalSource` interface:

```go theme={null}
type IncrementalSource interface {
	Source
	GetStartPageToken() (string, error)
	WalkChanges(ctx context.Context, token string, callback func(FileChange) error) (newToken string, err error)
}
```

### IncrementalSource Methods

#### GetStartPageToken

```go theme={null}
GetStartPageToken() (string, error)
```

Returns an opaque token representing the current head of the change stream. The engine calls this **before** the first full `Walk` to capture the baseline state.

**Returns:**

* Token string to persist in the snapshot
* Error if token cannot be retrieved

#### WalkChanges

```go theme={null}
WalkChanges(ctx context.Context, token string, callback func(FileChange) error) (newToken string, err error)
```

Emits only the entries that changed since `token`. Returns the new token to persist for the next run.

**Parameters:**

* `ctx` - Context for cancellation
* `token` - Token from the previous snapshot (or from `GetStartPageToken`)
* `callback` - Function called for each change

**Returns:**

* `newToken` - New token to store in the snapshot
* `err` - Error if change enumeration fails

### FileChange

```go theme={null}
type FileChange struct {
	Type ChangeType // "upsert" or "delete"
	Meta cloudstic.FileMeta
}

type ChangeType string

const (
	ChangeUpsert ChangeType = "upsert" // File or folder created/modified
	ChangeDelete ChangeType = "delete" // File or folder removed
)
```

For `ChangeDelete`, only `Meta.FileID` is required. All other fields can be empty.

## Example: Incremental HTTP Source

Extending the previous HTTP source to support incremental backups:

```go theme={null}
// HTTPChangeSource adds incremental support to HTTPSource.
type HTTPChangeSource struct {
	*HTTPSource
}

func NewHTTPChangeSource(baseURL string) *HTTPChangeSource {
	return &HTTPChangeSource{
		HTTPSource: NewHTTPSource(baseURL),
	}
}

// GetStartPageToken fetches the current change token from the server.
func (s *HTTPChangeSource) GetStartPageToken() (string, error) {
	resp, err := s.client.Get(s.baseURL + "/api/changes/token")
	if err != nil {
		return "", fmt.Errorf("fetch change token: %w", err)
	}
	defer resp.Body.Close()

	var result struct {
		Token string `json:"token"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		return "", fmt.Errorf("parse change token: %w", err)
	}

	return result.Token, nil
}

// WalkChanges fetches changes since the given token.
func (s *HTTPChangeSource) WalkChanges(
	ctx context.Context,
	token string,
	callback func(FileChange) error,
) (string, error) {
	resp, err := s.client.Get(s.baseURL + "/api/changes?since=" + token)
	if err != nil {
		return "", fmt.Errorf("fetch changes: %w", err)
	}
	defer resp.Body.Close()

	var result struct {
		Changes []struct {
			Type     string `json:"type"` // "upsert" or "delete"
			ID       string `json:"id"`
			Name     string `json:"name"`
			Size     int64  `json:"size"`
			Modified int64  `json:"modified"`
			IsDir    bool   `json:"is_dir"`
			Parent   string `json:"parent"`
		} `json:"changes"`
		NewToken string `json:"new_token"`
	}

	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		return "", fmt.Errorf("parse changes: %w", err)
	}

	// Emit folder changes first (upserts only, deletes can be in any order)
	for _, c := range result.Changes {
		if c.Type != "upsert" || !c.IsDir {
			continue
		}
		change := FileChange{
			Type: ChangeUpsert,
			Meta: cloudstic.FileMeta{
				Version: 1,
				FileID:  c.ID,
				Name:    c.Name,
				Type:    cloudstic.FileTypeFolder,
				Mtime:   c.Modified,
			},
		}
		if c.Parent != "" {
			change.Meta.Parents = []string{c.Parent}
		}
		if err := callback(change); err != nil {
			return "", err
		}
	}

	// Emit file changes and deletions
	for _, c := range result.Changes {
		if c.Type == "upsert" && c.IsDir {
			continue // Already emitted above
		}

		var change FileChange
		if c.Type == "delete" {
			change = FileChange{
				Type: ChangeDelete,
				Meta: cloudstic.FileMeta{
					FileID: c.ID,
				},
			}
		} else {
			change = FileChange{
				Type: ChangeUpsert,
				Meta: cloudstic.FileMeta{
					Version: 1,
					FileID:  c.ID,
					Name:    c.Name,
					Type:    cloudstic.FileTypeFile,
					Size:    c.Size,
					Mtime:   c.Modified,
				},
			}
			if c.Parent != "" {
				change.Meta.Parents = []string{c.Parent}
			}
		}

		if err := callback(change); err != nil {
			return "", err
		}
	}

	return result.NewToken, nil
}
```

### How the Engine Uses Incremental Sources

1. **First backup** - Engine calls `GetStartPageToken()`, then `Walk()` for full scan, stores token in snapshot
2. **Subsequent backups** - Engine calls `WalkChanges(token)` with the stored token, processes only the delta
3. **Token updated** - New token from `WalkChanges` is stored in the snapshot for the next run

<Note>
  If `WalkChanges` fails (e.g., token expired), the engine automatically falls back to a full `Walk`.
</Note>

## Implementation Guidelines

### Parent-Child Ordering

The most critical requirement is that **parents must be emitted before children**. This ensures the HAMT tree structure is built correctly.

```go theme={null}
// BAD: Children before parents
callback(cloudstic.FileMeta{FileID: "file.txt", Parents: []string{"folder1"}})
callback(cloudstic.FileMeta{FileID: "folder1", Type: cloudstic.FileTypeFolder})

// GOOD: Parents before children
callback(cloudstic.FileMeta{FileID: "folder1", Type: cloudstic.FileTypeFolder})
callback(cloudstic.FileMeta{FileID: "file.txt", Parents: []string{"folder1"}})
```

<Tip>
  For sources with complex hierarchies, use topological sorting to ensure correct ordering.
</Tip>

### FileID Stability

`FileID` must be stable across backups. If a file's `FileID` changes, the engine treats it as a deletion + addition rather than a modification.

```go theme={null}
// GOOD: Use stable IDs from the source API
FileID: googleDriveFileID // Stable across renames

// BAD: Use paths that change when files move
FileID: "/home/user/documents/report.pdf" // Changes if moved
```

### ContentHash Optimization

If your source provides checksums (like Google Drive's SHA-256 or OneDrive's QuickXorHash), include them in `ContentHash`:

```go theme={null}
meta.ContentHash = file.SHA256Checksum // From source API
```

This allows the engine to skip downloading unchanged files during incremental backups.

### Error Handling

Return descriptive errors that help users diagnose issues:

```go theme={null}
if resp.StatusCode == 401 {
	return fmt.Errorf("authentication failed: check your API credentials")
}
if resp.StatusCode == 429 {
	return fmt.Errorf("rate limit exceeded: try again later")
}
return fmt.Errorf("fetch file list: HTTP %d: %s", resp.StatusCode, resp.Status)
```

### Context Cancellation

Respect the `context.Context` passed to `Walk`, `WalkChanges`, and `Size`:

```go theme={null}
func (s *MySource) Walk(ctx context.Context, callback func(cloudstic.FileMeta) error) error {
	for _, file := range files {
		select {
		case <-ctx.Done():
			return ctx.Err()
		default:
		}

		if err := callback(convertFileMeta(file)); err != nil {
			return err
		}
	}
	return nil
}
```

## Testing Your Source

Create a simple test to verify your source implementation:

```go theme={null}
func TestHTTPSource(t *testing.T) {
	source := NewHTTPSource("https://example.com")

	// Test Info
	info := source.Info()
	if info.Type != "http" {
		t.Errorf("expected type 'http', got %q", info.Type)
	}

	// Test Walk
	ctx := context.Background()
	var files []cloudstic.FileMeta
	err := source.Walk(ctx, func(meta cloudstic.FileMeta) error {
		files = append(files, meta)
		return nil
	})
	if err != nil {
		t.Fatal(err)
	}

	if len(files) == 0 {
		t.Fatal("expected files, got none")
	}

	// Test GetFileStream
	for _, f := range files {
		if f.Type != cloudstic.FileTypeFile {
			continue
		}
		stream, err := source.GetFileStream(f.FileID)
		if err != nil {
			t.Errorf("GetFileStream(%s): %v", f.FileID, err)
			continue
		}
		stream.Close()
	}
}
```

## Registering Your Source

The CLI doesn't dispatch source types through a hand-written flag parser. `-source` is a URI whose scheme selects the source: `parseSourceURI` (`cmd/cloudstic/storeuri.go`) parses the scheme and path/host out of the raw string (e.g. `local:./documents`, `sftp://user@host/path`, `gdrive://Drive Name/path`), and `initSource` (`cmd/cloudstic/cmd_backup.go`) switches on `uri.scheme` to construct the matching `source.Source` with its functional options:

```go theme={null}
func initSource(ctx context.Context, opts initSourceOptions) (source.Source, error) {
	uri, err := parseSourceURI(opts.sourceURI)
	if err != nil {
		return nil, err
	}

	switch uri.scheme {
	case "local":
		return source.NewLocalSource(uri.path, source.WithLocalExcludePatterns(opts.excludePatterns)), nil
	case "sftp":
		return source.NewSFTPSource(uri.host, sftpSourceOpts(opts.sourceSFTP, uri)...)
	// ... gdrive, gdrive-changes, onedrive, onedrive-changes ...
	case "http":
		return httpsource.New(uri.host), nil
	default:
		return nil, fmt.Errorf("unknown source scheme %q", uri.scheme)
	}
}
```

To add `http` as a real scheme, extend `parseSourceURI`'s scheme switch to recognize it (so `http://example.com` parses into a `sourceURIParts{scheme: "http", host: "example.com"}`) and add the corresponding `case "http":` to `initSource`. Any source-specific flags (credentials, exclude patterns, and so on) are declared like any other flag — see `declareBackupArgs` in `cmd/cloudstic/cmd_backup.go` and the CLI Integration section of the [Contributing Guide](/advanced/contributing) — not parsed ad hoc inside `initSource`.

Then use it:

```bash theme={null}
cloudstic backup -source http://example.com
```