# Backup & Restore Flow
Source: https://docs.cloudstic.com/advanced/backup-flow
Detailed internals of the backup and restore operations
## Backup Flow
The backup operation is orchestrated by the `BackupManager` in `internal/engine/`. Here's the complete flow:
### 1. Initialization
```go theme={null}
// From AGENTS.md and internal/engine/
1. BackupManager acquires a shared lock
2. Load the previous snapshot (if any) for this source identity
3. Extract the previous HAMT root reference
```
**Source identity** is determined by `SourceInfo`:
* `type`: e.g. "gdrive", "local", "onedrive"
* `identity`: stable container identity
* `path_id`: stable selected-root identity within the container
* `account`: friendly display account label
* `path`: friendly display path
* `drive_name`: friendly container label
Previous snapshot matching prefers the new stable fields and falls back for backward compatibility:
1. `type + identity + path_id`
2. `type + identity + path` (bridge fallback)
3. `type + volume_uuid + path` (legacy fallback)
4. `type + account + path` (legacy fallback)
### 2. Source Scanning
**Full scan** (for `local`, `sftp`, `gdrive`, `onedrive`):
```go theme={null}
source.Walk(func(entry SourceEntry) error {
// Process each file/folder
})
```
**Incremental scan** (for `gdrive-changes`, `onedrive-changes`):
```go theme={null}
changeToken := previousSnapshot.ChangeToken
source.WalkChanges(changeToken, func(entry SourceEntry) error {
// Process only changed files
})
```
For each file encountered:
1. **Look up the file ID** in the old HAMT:
```go theme={null}
oldMetaRef, err := hamtTree.Lookup(oldRoot, entry.FileID)
```
2. **Fast-check metadata** (name, size, mtime, type, parents):
* If identical and the source doesn't provide a content hash, **carry the old hash forward**
* This avoids false-positive diffs for metadata-only changes
3. **Determine action**:
* **Unchanged:** Re-insert into new HAMT by reference (structural sharing)
* **Changed or new:** Queue for upload
### 3. Upload Phase
Changed/new files are processed by concurrent workers (default: 10 workers):
```go theme={null}
for _, queuedFile := range uploadQueue {
go func(file SourceEntry) {
// 1. Stream file content
reader := source.Open(file.FileID)
// 2. Content-defined chunking
chunks := fastCDC(reader, minSize, avgSize, maxSize)
// 3. For each chunk:
for chunk := range chunks {
// Compute hash (HMAC-SHA256 or SHA-256)
hash := computeChunkHash(chunk.Data, dedupKey)
// Deduplicate
if store.Exists("chunk/" + hash) {
continue // Skip, already stored
}
// Compress and write
compressed := zstd.Compress(chunk.Data)
store.Put("chunk/" + hash, compressed)
}
// 4. Create content object
contentObj := Content{
Type: "content",
Size: file.Size,
Chunks: chunkRefs,
}
contentHash := sha256(file.RawData)
store.Put("content/" + contentHash, json(contentObj))
// 5. Create filemeta object
fileMeta := FileMeta{
FileID: file.FileID,
Name: file.Name,
Type: file.Type,
ContentHash: contentHash,
Size: file.Size,
Mtime: file.Mtime,
Parents: parentRefs,
...
}
metaRef := "filemeta/" + sha256(json(fileMeta))
store.Put(metaRef, json(fileMeta))
// 6. Insert into new HAMT
newRoot = hamtTree.Insert(newRoot, file.FileID, metaRef)
}(queuedFile)
}
```
#### FastCDC Chunking
From `internal/engine/chunker.go` and the spec:
| Parameter | Value |
| --------- | ------- |
| Min size | 512 KiB |
| Avg size | 1 MiB |
| Max size | 8 MiB |
FastCDC uses a rolling hash to find content-defined boundaries:
1. Compute a rolling hash of a 64-byte window
2. When the hash matches a pattern (e.g. last 20 bits are zero), create a boundary
3. Enforce min/max size constraints
4. The final chunk may be smaller than the minimum
Content-defined chunking ensures that inserting bytes at the start of a file doesn't invalidate all subsequent chunks. Only the chunks containing modified data change.
### 4. HAMT Flush
After all files are uploaded and inserted into the HAMT:
```go theme={null}
// TransactionalStore buffers nodes in memory
// Only flush the reachable subset from the final root
txnStore.Flush(newRoot)
```
**Flush algorithm** (BFS from root):
1. Start a queue with the root node ref
2. For each node in the queue:
* If it's in the in-memory buffer, write it to persistent storage
* If it's an internal node, add all child refs to the queue
3. Discard any buffered nodes that were never visited
This avoids uploading intermediate superseded nodes created during tree construction.
### 5. Snapshot Commit
```go theme={null}
// 1. Create snapshot object
snapshot := Snapshot{
Version: 1,
Created: time.Now().Format(time.RFC3339),
Root: newRoot,
Seq: previousSeq + 1,
Source: sourceInfo,
ChangeToken: newChangeToken, // For incremental sources
...
}
snapshotRef := "snapshot/" + sha256(json(snapshot))
store.Put(snapshotRef, json(snapshot))
// 2. Update index/latest (the commit point)
index := Index{
LatestSnapshot: snapshotRef,
Seq: snapshot.Seq,
}
store.Put("index/latest", json(index))
// 3. Update index/snapshots catalog
catalog = append(catalog, SnapshotSummary{...})
store.Put("index/snapshots", json(catalog))
// 4. Update index/packs catalog (if packfiles enabled)
// Automatically handled by the PackStore layer
```
**The commit point** is updating `index/latest`. Until this write completes, the previous snapshot remains the "latest" and the repository is in a consistent state.
### 6. Lock Release
The backup lock is released, allowing other operations to proceed.
## Restore Flow
The restore operation is orchestrated by the `RestoreManager` in `internal/engine/`.
### 1. Snapshot Resolution
```go theme={null}
// Resolve snapshot by:
// - Explicit ref ("snapshot/")
// - Sequence number (42)
// - "latest" keyword
snapshot := resolveSnapshot(snapshotID)
root := snapshot.Root
```
### 2. HAMT Traversal
Walk the HAMT to collect all file metadata entries:
```go theme={null}
var entries []FileMeta
hamtTree.Walk(root, func(key, metaRef string) error {
// Load filemeta object
metaData := store.Get(metaRef)
meta := json.Unmarshal(metaData)
entries = append(entries, meta)
})
```
### 3. Topological Sort
Ensure parent directories are created before their children:
```go theme={null}
// Build parent dependency graph
graph := buildDependencyGraph(entries)
// Topological sort
sorted := topologicalSort(graph)
```
This handles Google Drive's multi-parent semantics where a file can appear in multiple directories.
### 4. Path Reconstruction
Walk the parent chain of each entry to reconstruct the full relative path:
```go theme={null}
func buildPath(meta FileMeta) string {
if len(meta.Parents) == 0 {
return meta.Name // Root entry
}
// Load first parent (arbitrary choice if multi-parent)
parentRef := meta.Parents[0]
parentData := store.Get(parentRef)
parentMeta := json.Unmarshal(parentData)
parentPath := buildPath(parentMeta)
return parentPath + "/" + meta.Name
}
```
For files with multiple parents (Google Drive shared folders), only the first parent is used to construct the primary path. Other parents are ignored during restore.
### 5. ZIP Archive Creation
Write entries to a ZIP archive in topologically-sorted order:
```go theme={null}
zipWriter := zip.NewWriter(output)
for _, meta := range sortedEntries {
path := buildPath(meta)
if meta.Type == "folder" {
// Create directory entry
header := &zip.FileHeader{
Name: path + "/",
Method: zip.Store,
}
header.SetModTime(time.Unix(meta.Mtime, 0))
zipWriter.CreateHeader(header)
} else {
// Create file entry
header := &zip.FileHeader{
Name: path,
Method: zip.Deflate,
}
header.SetModTime(time.Unix(meta.Mtime, 0))
writer, _ := zipWriter.CreateHeader(header)
// Load content object
contentData := store.Get("content/" + meta.ContentHash)
content := json.Unmarshal(contentData)
// Stream chunks
for _, chunkRef := range content.Chunks {
// Fetch and decompress chunk
compressedData := store.Get(chunkRef)
rawData := zstd.Decompress(compressedData)
writer.Write(rawData)
}
}
}
zipWriter.Close()
```
### 6. Output
The ZIP archive is:
* Written to stdout (CLI)
* Returned as a byte stream (web API)
* Saved to a file (with `-o` flag)
## Performance Optimizations
### Concurrent Upload
The backup manager uses a worker pool (default: 10 concurrent workers) to parallelize file uploads:
```go theme={null}
workerPool := make(chan struct{}, 10) // Semaphore
for _, file := range uploadQueue {
workerPool <- struct{}{} // Acquire
go func(f SourceEntry) {
defer func() { <-workerPool }() // Release
processFile(f)
}(file)
}
```
### Chunk-Level Deduplication
Before writing a chunk:
```go theme={null}
if store.Exists("chunk/" + hash) {
continue // Skip, already stored
}
```
The `KeyCacheStore` layer caches existence checks in a local bbolt database to avoid redundant `Exists` calls:
* First check: network request → cache result
* Subsequent checks: local cache hit (0 network requests)
### Content-Level Deduplication
Before streaming a file, check if its content object already exists:
```go theme={null}
if store.Exists("content/" + file.ContentHash) {
// Entire file upload skipped!
// Only create new filemeta and HAMT nodes
}
```
For sources that provide content hashes (Google Drive MD5, OneDrive SHA1), this is a huge optimization.
### Packfile Bundling
Small objects (\< 512KB) are bundled into 8MB packfiles:
* Reduces API calls from thousands to dozens
* LRU cache (128MB) keeps hot packs in memory
* Typical metadata read: 0-1 network requests after initial pack fetch
### Structural Sharing
Unchanged files reuse their filemeta refs, and unchanged subtrees reuse their HAMT node refs:
```
Snapshot 1: 10,000 files → 10,000 filemeta + ~500 HAMT nodes
Snapshot 2: 10 files changed → 10 new filemeta + ~5 new HAMT nodes
Incremental storage: ~15 objects, not 10,510
```
## Error Handling
### Transient Errors
Network errors during upload are retried with exponential backoff:
```go theme={null}
retry.Do(
func() error { return store.Put(key, data) },
retry.Attempts(3),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay),
)
```
### Permanent Errors
Permanent errors (authentication failure, permission denied) abort the backup immediately:
```go theme={null}
if isPermanentError(err) {
return fmt.Errorf("backup failed: %w", err)
}
```
### Partial Upload Cleanup
If a backup is interrupted, orphaned objects remain in the store but are not reachable from any snapshot.
Running `prune` after an interrupted backup will:
1. Mark all reachable objects from existing snapshots
2. Sweep and delete any orphaned objects
3. Repack fragmented packfiles
Orphaned objects are harmless. They consume storage but don't affect backup correctness. Prune reclaims this space.
## Diff Operation
The `diff` command leverages the HAMT's structural diff:
```go theme={null}
hamtTree.Diff(snapshot1.Root, snapshot2.Root, func(entry DiffEntry) error {
if entry.OldValue == "" {
fmt.Printf("+ %s\n", entry.Key) // Added
} else if entry.NewValue == "" {
fmt.Printf("- %s\n", entry.Key) // Removed
} else {
fmt.Printf("M %s\n", entry.Key) // Modified
}
})
```
The diff is **structural**: if two subtrees have the same root hash, they're identical and the traversal skips them entirely. This makes diff extremely fast even for snapshots with millions of files.
# Go Client API
Source: https://docs.cloudstic.com/advanced/client-api
Use Cloudstic programmatically in your Go applications
The `github.com/cloudstic/cli` package exposes a high-level Go API that mirrors the CLI commands. Use it to embed backup, restore, and repository management directly into your Go programs.
## Installation
```bash theme={null}
go get github.com/cloudstic/cli
```
Import the root package:
```go theme={null}
import cloudstic "github.com/cloudstic/cli"
```
Storage backends and keychain helpers live in sub-packages:
```go theme={null}
import (
"github.com/cloudstic/cli/pkg/store"
"github.com/cloudstic/cli/pkg/keychain"
"github.com/cloudstic/cli/pkg/source"
)
```
Two further packages turn user-facing configuration into live objects, and the
split between them decides what importing costs you:
```go theme={null}
import (
"github.com/cloudstic/cli/pkg/config" // what did the user configure
"github.com/cloudstic/cli/pkg/open" // connect to it
)
```
`pkg/config` holds resolved configuration values and performs no I/O against a
store or a provider, so reading and validating configuration pulls in no cloud
SDK. `pkg/open` constructs stores, sources, keychains and clients from those
values, and necessarily links whichever provider SDK you use. If you only need to
inspect configuration, import the first and not the second.
## Quick Start
```go theme={null}
package main
import (
"bytes"
"context"
"fmt"
"log"
cloudstic "github.com/cloudstic/cli"
"github.com/cloudstic/cli/pkg/keychain"
localsource "github.com/cloudstic/cli/pkg/source/local"
localstore "github.com/cloudstic/cli/pkg/store/local"
)
func main() {
ctx := context.Background()
// 1. Open a storage backend
rawStore, err := localstore.New("./my-repo")
if err != nil {
log.Fatal(err)
}
// 2. Build a keychain with your credentials
kc := keychain.Chain{keychain.WithPassword("my-passphrase")}
// 3. Initialize the repository (first time only)
_, err = cloudstic.InitRepo(ctx, rawStore,
cloudstic.WithInitCredentials(kc),
)
if err != nil {
log.Fatal(err)
}
// 4. Create a client: auto-resolves encryption via the keychain
client, err := cloudstic.NewClient(ctx, rawStore, cloudstic.WithKeychain(kc))
if err != nil {
log.Fatal(err)
}
// 5. Back up a local directory
src := localsource.New("./documents")
result, err := client.Backup(ctx, src)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Snapshot: %s (%d new files)\n", result.SnapshotHash, result.FilesNew)
// 6. Restore to a ZIP archive
var buf bytes.Buffer
_, err = client.Restore(ctx, &buf, "latest")
if err != nil {
log.Fatal(err)
}
fmt.Printf("Restored %d bytes\n", buf.Len())
}
```
## Storage Backends
All backends implement `store.ObjectStore`. Pass the raw store to `InitRepo` and `NewClient`.
Each backend lives in its own package, so importing the one you use does not pull in the others' SDKs — importing the `ObjectStore` contract alone pulls in no cloud SDK at all.
```go theme={null}
import (
localstore "github.com/cloudstic/cli/pkg/store/local"
s3store "github.com/cloudstic/cli/pkg/store/s3"
b2store "github.com/cloudstic/cli/pkg/store/b2"
sftpstore "github.com/cloudstic/cli/pkg/store/sftp"
)
```
### Local
```go theme={null}
s, err := localstore.New("/path/to/repo")
```
### Amazon S3 (and S3-compatible)
```go theme={null}
s, err := s3store.New(ctx, "my-bucket",
s3store.WithRegion("us-east-1"),
// For MinIO, Cloudflare R2, Wasabi, etc.:
s3store.WithEndpoint("https://minio.example.com"),
// Explicit credentials (or use the AWS SDK default credential chain):
s3store.WithCredentials(accessKeyID, secretAccessKey),
s3store.WithPrefix("backups/"),
)
```
### Backblaze B2
```go theme={null}
s, err := b2store.New("my-bucket",
b2store.WithCredentials(keyID, appKey),
b2store.WithPrefix("backups/"),
)
```
### SFTP
```go theme={null}
s, err := sftpstore.New("backup.example.com",
sftpstore.WithBasePath("/home/backup/repo"),
sftpstore.WithUser("backupuser"),
sftpstore.WithKey("~/.ssh/id_ed25519"),
sftpstore.WithPort("22"),
)
```
## Keychain
The keychain resolves credentials to a master key. Build a `keychain.Chain` before calling `InitRepo` or `NewClient`.
```go theme={null}
// Password-based
kc := keychain.Chain{keychain.WithPassword("my-passphrase")}
// Platform key (raw 32-byte key)
platformKey, _ := hex.DecodeString("64-hex-chars...")
kc := keychain.Chain{keychain.WithPlatformKey(platformKey)}
// Recovery key (24-word BIP39 mnemonic)
kc := keychain.Chain{keychain.WithRecoveryKey("word1 word2 ... word24")}
// Multiple credentials (tried in order until one succeeds)
kc := keychain.Chain{
keychain.WithPassword("my-passphrase"),
keychain.WithRecoveryKey("word1 word2 ... word24"),
}
// AWS KMS. The client lives in pkg/crypto/kms so that pkg/crypto itself links
// no cloud SDK; keychain/kms.WithARN builds one on demand if you prefer.
kmsClient, _ := kms.New(ctx, "arn:aws:kms:us-east-1:123456789:key/abc-123")
kc := keychain.Chain{keychain.WithKMSClient(kmsClient)}
```
## Initializing a Repository
`InitRepo` is a package-level function that runs on the raw (unencrypted) store. It only needs to be called once per repository.
```go theme={null}
func InitRepo(ctx context.Context, rawStore store.ObjectStore, opts ...InitOption) (*InitResult, error)
```
### Init Options
| Option | Description |
| ---------------------------------------- | ----------------------------------------------------------------------- |
| `WithInitCredentials(kc keychain.Chain)` | Key slots to create (password, platform key, KMS) |
| `WithInitRecovery()` | Also generate a 24-word recovery key slot |
| `WithInitNoEncryption()` | Create an unencrypted repository |
| `WithInitAdoptSlots()` | Adopt existing slots if already initialized (prevents error on re-init) |
### InitResult
```go theme={null}
type InitResult struct {
Encrypted bool // whether the repository uses encryption
RecoveryKey string // 24-word mnemonic (empty if WithInitRecovery() was not requested)
AdoptedSlots bool // true if existing slots were adopted
}
```
### Example
```go theme={null}
result, err := cloudstic.InitRepo(ctx, rawStore,
cloudstic.WithInitCredentials(kc),
cloudstic.WithInitRecovery(),
)
if err != nil {
log.Fatal(err)
}
if result.RecoveryKey != "" {
fmt.Println("Recovery key:", result.RecoveryKey)
// Store this securely: it is displayed only once!
}
```
## Inspecting a Repository
These package-level functions read repository state from the raw store without requiring the encryption key — useful for deciding whether to prompt for credentials before calling `NewClient`.
```go theme={null}
func InspectRepo(ctx context.Context, rawStore store.ObjectStore) (RepoStatus, error)
```
```go theme={null}
type RepoStatus struct {
Initialized bool // whether a config marker exists at all
Encrypted bool // whether the repository uses encryption
Sealed bool // whether the marker itself is sealed (only true for newer encrypted repos)
}
```
```go theme={null}
status, err := cloudstic.InspectRepo(ctx, rawStore)
if err != nil {
log.Fatal(err)
}
if !status.Initialized {
fmt.Println("Run InitRepo first.")
} else if status.Encrypted {
fmt.Println("Repository is encrypted; resolve a keychain before NewClient.")
}
```
`LoadRepoConfig(ctx, rawStore, encryptionKey) (*RepoConfig, error)` reads the full marker (not just its status) and requires the encryption key for a sealed repository. `UpgradeRepoFormat` stamps the on-disk format version; `NewClient`/`Backup`/`Prune`/`Forget` call it for you as part of normal operation, so most callers never need it directly.
## Creating a Client
```go theme={null}
func NewClient(ctx context.Context, base store.ObjectStore, opts ...ClientOption) (*Client, error)
```
`NewClient` reads the repository config, resolves the master key via the keychain, and builds the encryption/compression/packfile decorator chain internally.
### Client Options
| Option | Description |
| --------------------------------- | -------------------------------------------------------------- |
| `WithKeychain(kc keychain.Chain)` | Keychain for automatic master key resolution |
| `WithEncryptionKey(key []byte)` | Direct 32-byte AES key. Bypasses keychain and config detection |
| `WithReporter(r Reporter)` | Progress reporter for UI feedback |
| `WithPackfile(enable bool)` | Bundle small objects into 8MB packs (default: `true`) |
```go theme={null}
client, err := cloudstic.NewClient(ctx, rawStore,
cloudstic.WithKeychain(kc),
cloudstic.WithReporter(myReporter),
cloudstic.WithPackfile(true),
)
```
`WithEncryptionKey` is intended for SaaS scenarios where the master key is resolved externally. For typical use, prefer `WithKeychain`.
`client.Store()` returns the underlying `store.ObjectStore` (the fully decorated store `NewClient` built), for callers that need lower-level access alongside the `Client` API.
## Backup
```go theme={null}
func (c *Client) Backup(ctx context.Context, src source.Source, opts ...BackupOption) (*BackupResult, error)
```
Creates a new backup snapshot from the given source.
### Backup Options
| Option | Description |
| ------------------------------ | --------------------------------------------------------------------- |
| `WithBackupDryRun()` | Scan without writing to the repository |
| `WithIgnoreEmptySnapshot()` | Skip writing a snapshot if nothing changed since the previous one |
| `WithTags(tags ...string)` | Apply tags to the snapshot |
| `WithGenerator(name string)` | Record the tool/script that produced the backup (e.g. for automation) |
| `WithMeta(key, value string)` | Attach an arbitrary key/value pair to the snapshot |
| `WithExcludeHash(hash string)` | Record the exclude pattern fingerprint in the snapshot |
### BackupResult
```go theme={null}
type BackupResult struct {
SnapshotRef string // "snapshot/"
SnapshotHash string // bare content hash
Root string // HAMT root ref
FilesNew int64
FilesChanged int64
FilesUnmodified int64
FilesRemoved int64
DirsNew int64
DirsChanged int64
DirsUnmodified int64
DirsRemoved int64
BytesAddedRaw int64 // uncompressed bytes written
BytesAddedStored int64 // bytes written to the store (post-compression and encryption)
Duration time.Duration
DryRun bool
EmptySnapshotIgnored bool // true if WithIgnoreEmptySnapshot() skipped an unchanged backup
}
```
### Example
```go theme={null}
src := localsource.New("./documents",
localsource.WithExcludePatterns([]string{"*.log", "node_modules/"}),
)
result, err := client.Backup(ctx, src,
cloudstic.WithTags("production", "weekly"),
)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Snapshot %s: %d new, %d changed files\n",
result.SnapshotHash, result.FilesNew, result.FilesChanged)
```
### Sources
Sources implement `source.Source`. Each lives in its own package, so importing one does not pull in the others' SDKs:
```go theme={null}
import (
localsource "github.com/cloudstic/cli/pkg/source/local"
sftpsource "github.com/cloudstic/cli/pkg/source/sftp"
"github.com/cloudstic/cli/pkg/source/gdrive"
"github.com/cloudstic/cli/pkg/source/onedrive"
)
// Local filesystem
src := localsource.New("/path/to/dir",
localsource.WithExcludePatterns([]string{"*.tmp"}),
)
// SFTP remote directory
src, err := sftpsource.New("backup.example.com",
sftpsource.WithBasePath("/remote/path"),
sftpsource.WithUser("user"),
sftpsource.WithKey("~/.ssh/id_rsa"),
)
// Google Drive (full scan)
src, err := gdrive.New(ctx,
gdrive.WithTokenPath("/path/to/google_token.json"),
gdrive.WithDriveID("sharedDriveID"), // omit for My Drive
gdrive.WithRootFolderID("folderID"), // omit for the entire drive
)
// Google Drive (incremental via the Changes API: recommended)
src, err := gdrive.NewChangeSource(ctx,
gdrive.WithTokenPath("/path/to/google_token.json"),
)
// OneDrive (full scan)
src, err := onedrive.New(ctx,
onedrive.WithTokenPath("/path/to/onedrive_token.json"),
)
// OneDrive (incremental via the Delta API: recommended)
src, err := onedrive.NewChangeSource(ctx,
onedrive.WithTokenPath("/path/to/onedrive_token.json"),
)
```
## Restore
```go theme={null}
func (c *Client) Restore(ctx context.Context, w io.Writer, snapshotRef string, opts ...RestoreOption) (*RestoreResult, error)
```
Writes the snapshot's file tree as a ZIP archive to `w`. Pass `io.Discard` for a dry run.
`snapshotRef` accepts `""`, `"latest"`, a bare hash or unique hash prefix, or
`"snapshot/"`.
### Restore Options
| Option | Description |
| ------------------------------ | ------------------------------------------------------------ |
| `WithRestoreDryRun()` | Count files/bytes without writing ZIP data |
| `WithRestorePath(path string)` | Restore only the given file or subtree (e.g. `"Documents/"`) |
| `WithRestoreNoVerify()` | Skip content-hash verification of restored chunks |
### RestoreResult
```go theme={null}
type RestoreResult struct {
SnapshotRef string
Root string
FilesWritten int
DirsWritten int
BytesWritten int64
Errors int // number of non-fatal errors
Warnings int // number of non-fatal warnings (e.g. skipped attributes)
DryRun bool
}
```
### Example
```go theme={null}
f, err := os.Create("restore.zip")
if err != nil {
log.Fatal(err)
}
defer f.Close()
result, err := client.Restore(ctx, f, "latest",
cloudstic.WithRestorePath("Documents/"),
)
if err != nil {
os.Remove("restore.zip")
log.Fatal(err)
}
fmt.Printf("Restored %d files (%d bytes)\n", result.FilesWritten, result.BytesWritten)
```
### Restoring directly to a directory
```go theme={null}
func (c *Client) RestoreToDir(ctx context.Context, outputDir, snapshotRef string, opts ...RestoreOption) (*RestoreResult, error)
```
Writes files directly to `outputDir` instead of a ZIP archive. Takes the same `RestoreOption`s and returns the same `RestoreResult`.
```go theme={null}
result, err := client.RestoreToDir(ctx, "/restore/target", "latest")
if err != nil {
log.Fatal(err)
}
fmt.Printf("Restored %d files to disk\n", result.FilesWritten)
```
## List
```go theme={null}
func (c *Client) List(ctx context.Context, opts ...ListOption) (*ListResult, error)
```
Lists all snapshots in the repository. Returns them sorted oldest-first.
### List Options
| Option | Description |
| ------ | ----------- |
### ListResult
```go theme={null}
type ListResult struct {
Snapshots []SnapshotEntry
}
type SnapshotEntry struct {
Ref string // "snapshot/"
Snap core.Snapshot // the decoded snapshot object (Seq, Source, Tags, Meta, ...)
Created time.Time
}
```
`SnapshotEntry.Snap` is a `core.Snapshot`, which carries `Seq int`, `Source *core.SourceInfo` (`Type`, `Account`, `Path`, ...), `Tags []string`, and `Meta map[string]string`. `SourceInfo` may be `nil` for snapshots written before source identity was tracked.
These types are declared in `github.com/cloudstic/cli/internal/core`, which external modules cannot import directly — Go's `internal/` rule blocks the import statement. You don't need to. The root `cloudstic` package re-exports each of them as a type alias, so you can name them explicitly:
```go theme={null}
var snap cloudstic.Snapshot // = core.Snapshot
var info *cloudstic.SourceInfo // = *core.SourceInfo
var meta cloudstic.FileMeta // = core.FileMeta
var entry cloudstic.SnapshotEntry // = engine.SnapshotEntry
```
Because a Go alias denotes the identical type, these are interchangeable with what the `Client` API returns — you can declare variables, write helper function signatures, and implement interfaces against them. Reading exported fields off returned values (`entry.Snap.Seq`, `meta.Name`) works without naming the types at all, as this example does.
### Example
```go theme={null}
result, err := client.List(ctx)
if err != nil {
log.Fatal(err)
}
for _, entry := range result.Snapshots {
source := "unknown"
if entry.Snap.Source != nil {
source = entry.Snap.Source.Type
}
fmt.Printf("[%d] %s %-10s %s\n",
entry.Snap.Seq, entry.Created.Format(time.DateTime), source, entry.Ref)
}
```
## LsSnapshot
```go theme={null}
func (c *Client) LsSnapshot(ctx context.Context, snapshotID string, opts ...LsSnapshotOption) (*LsSnapshotResult, error)
```
Loads all file metadata from a snapshot and returns the full directory tree.
Accepts `"latest"`, a bare hash or unique hash prefix, or
`"snapshot/"`.
### LsSnapshot Options
| Option | Description |
| ------ | ----------- |
### LsSnapshotResult
```go theme={null}
type LsSnapshotResult struct {
Ref string
Snapshot core.Snapshot
RootRefs []string // top-level entry refs
RefToMeta map[string]core.FileMeta // ref → file metadata (name, size, type, mode, owner, ...)
ChildRefs map[string][]string // parent ref → ordered child refs
}
```
See the note above about `core.Snapshot`/`core.FileMeta` and the `internal/` import restriction.
## Find
```go theme={null}
func (c *Client) Find(ctx context.Context, q FindQuery) (*FindResult, error)
```
Locates files across every snapshot in the repository without you having to know which snapshot holds them. Unlike the other read operations, `Find` takes a snapshot as *output* rather than input: by default it searches every snapshot and reports, for each matching file, the versions it has had and which snapshots each version lives in. It is a pure read path — no lock is taken and nothing is written.
The query is a value, not a list of options. That makes it serializable — you can store, log, or send a query — and a zero `FindQuery` is a valid search: every file, capped at the default result limit.
```go theme={null}
result, err := client.Find(ctx, cloudstic.FindQuery{
Name: "*.pdf",
Size: &cloudstic.SizeCompare{Op: cloudstic.SizeAtLeast, Bytes: 10 << 20},
Latest: 5,
MaxResults: 100,
})
```
### FindQuery
Entry predicates — all the ones you set must match:
| Field | Description |
| -------------------- | ----------------------------------------------------------------- |
| `Name string` | Match by basename glob |
| `Path string` | Match by full path glob |
| `Regex string` | Match by regular expression |
| `IgnoreCase bool` | Case-insensitive matching for name/path/regex |
| `FileID string` | Match a specific source file ID |
| `ContentHash string` | Match files with this exact content hash |
| `Ref string` | Match a specific `filemeta/` ref |
| `Type core.FileType` | Restrict to a file type (file, directory, ...) |
| `Size *SizeCompare` | Size predicate — see `ParseSizeCompare` below |
| `Newer string` | File `Mtime` at or after this (RFC3339 or a duration like `"7d"`) |
| `Older string` | File `Mtime` at or before this |
Snapshot selectors — which snapshots are searched:
| Field | Description |
| -------------------- | ----------------------------------------------------------------------------- |
| `Snapshots []string` | Restrict to specific snapshots (`"latest"`, full hash, or unambiguous prefix) |
| `Source string` | Restrict to snapshots from a source URI |
| `Tags []string` | Restrict to snapshots carrying any of these tags |
| `Latest int` | Restrict to the `n` newest selected snapshots |
| `Since string` | Only snapshots created at or after this |
| `Until string` | Only snapshots created at or before this |
Presentation and execution:
| Field | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `GroupByContent bool` | Group matches by content hash instead of file identity (finds duplicate content) |
| `MaxResults int` | Cap the distinct files reported. Zero means the default of 1000; scanning continues past the cap so counters stay accurate |
| `NoDelta bool` | Force a full per-snapshot walk instead of the delta scan between related snapshots |
`SetPattern` is a method rather than a field, because routing a positional
pattern is a decision rather than a value: a pattern containing a separator
constrains the full path, one without it constrains the basename. Assigning
`Path` directly when you meant a basename makes every search pay for path
reconstruction.
```go theme={null}
var q cloudstic.FindQuery
q.SetPattern("*.pdf") // no separator → Name
q.SetPattern("Documents/*.pdf") // separator → Path
```
Two helpers parse the string forms the CLI accepts, for callers building `SizeCompare`/time values programmatically:
```go theme={null}
func ParseSizeCompare(spec string) (SizeCompare, error) // "+10M", "-10M", "10M"
func ParseFindTime(spec string) (time.Time, error) // RFC3339 or a duration like "7d"
```
### FindResult
```go theme={null}
type FindResult struct {
Query FindQuery
SnapshotsSearched int
EntriesScanned int
MetaFetched int // filemeta objects actually read
Matches []FileMatch
Truncated bool // true if MaxResults was reached
Warnings []string
GroupedBy string // "file" or "content"
Elapsed string
}
// FileMatch is one file (or, with GroupByContent, one distinct
// content hash), with every version the query matched.
type FileMatch struct {
FileID string // empty when grouped by content
ContentHash string // empty unless grouped by content
Source *core.SourceInfo
Type core.FileType
Versions []FileVersion // newest first
}
// FileVersion is one immutable state of a file.
type FileVersion struct {
Ref string
FileID string
Name string
Paths []string // more than one path if the same file lives under multiple parents
ContentHash string
Type core.FileType
Size int64
Mtime int64
Mode uint32
Snapshots []SnapshotRef // every snapshot holding this exact version
FirstSeen string // ISO8601, earliest containing snapshot
LastSeen string // ISO8601, latest containing snapshot
}
type SnapshotRef struct {
Ref string
Seq int
Created string // ISO8601
}
```
`FileMatch` has two convenience methods: `Path()` returns the newest version's first path, and `LatestSnapshot()` returns the newest snapshot holding the newest version — the one a follow-up `Restore` would target.
### Example
```go theme={null}
// Find every file over 100MB modified in the last 30 days, across all snapshots.
size, err := cloudstic.ParseSizeCompare("+100M")
if err != nil {
log.Fatal(err)
}
result, err := client.Find(ctx, cloudstic.FindQuery{
Size: &size,
Newer: "30d",
})
if err != nil {
log.Fatal(err)
}
for _, m := range result.Matches {
v := m.Versions[0]
fmt.Printf("%s (%d bytes) — %d version(s), latest in %d snapshot(s)\n",
m.Path(), v.Size, len(m.Versions), len(v.Snapshots))
}
if result.Truncated {
fmt.Printf("... and more (raise MaxResults to see them)\n")
}
```
Finding duplicate content:
```go theme={null}
result, err := client.Find(ctx, cloudstic.FindQuery{GroupByContent: true})
if err != nil {
log.Fatal(err)
}
for _, m := range result.Matches {
if len(m.Versions) > 1 {
fmt.Printf("%s duplicated across %d files\n", m.ContentHash, len(m.Versions))
}
}
```
## Diff
```go theme={null}
func (c *Client) Diff(ctx context.Context, snap1, snap2 string, opts ...DiffOption) (*DiffResult, error)
```
Compares two snapshots and returns the set of added, modified, and removed
files. Each argument accepts `"latest"`, a full hash, or a unique hash prefix.
### Diff Options
| Option | Description |
| ------ | ----------- |
### Snapshot reference errors
Snapshot readers return sentinel errors that you can inspect with `errors.Is`:
```go theme={null}
result, err := client.Diff(ctx, "abc123", "latest")
switch {
case errors.Is(err, cloudstic.ErrSnapshotNotFound):
// No snapshot matched the reference.
case errors.Is(err, cloudstic.ErrSnapshotRefAmbiguous):
// More than one snapshot matched the prefix.
case err != nil:
log.Fatal(err)
}
```
The same sentinels apply to `Restore`, `RestoreToDir`, `LsSnapshot`, and
snapshot selectors passed to `Find`.
### DiffResult
```go theme={null}
type DiffResult struct {
Ref1 string
Ref2 string
Changes []Change
}
// Change.Type is one of: "added", "modified", "removed", "unchanged"
type Change struct {
Type string
Path string
}
```
### Example
```go theme={null}
result, err := client.Diff(ctx, "abc123", "latest")
if err != nil {
log.Fatal(err)
}
for _, c := range result.Changes {
if c.Type != "unchanged" {
fmt.Printf("%s %s\n", c.Type, c.Path)
}
}
```
## CopyFrom
```go theme={null}
func (c *Client) CopyFrom(ctx context.Context, src *Client, opts ...CopyOption) (*CopyResult, error)
```
Transfers snapshot history from another repository into this one. It is a method
on the **destination** — the client that will be written to — and takes the
source as another client.
Taking a `*Client` rather than a store is deliberate. The store decorator chain
is a correctness and security invariant that no caller should assemble; a
`*Client` has already had it built. It also keeps source credentials out of the
option surface — the source client was constructed with `WithKeychain` like any
other — and gates the source's repository format for free, because `NewClient`
checks it.
Nothing is written to the source, so read-only credentials are sufficient there.
### Copy Options
| Option | Description |
| --------------------------------------- | ----------------------------------------------------------------------------- |
| `WithCopySnapshotIDs(ids ...string)` | Copy only the named source snapshots (`"latest"`, a hash, or a unique prefix) |
| `WithCopyFilterSource(source string)` | Copy only snapshots of this source type |
| `WithCopyFilterPath(path string)` | Copy only snapshots of this source path |
| `WithCopyFilterAccount(account string)` | Copy only snapshots of this account |
| `WithCopyFilterTag(tag string)` | Copy only snapshots carrying this tag; repeat to require several |
| `WithCopySince(t time.Time)` | Copy only snapshots created at or after `t` |
| `WithCopyDryRun()` | Resolve and report the selection without writing |
| `WithCopyAllowCopied()` | Allow copying snapshots that were themselves produced by a copy |
### CopyResult
```go theme={null}
type CopyResult struct {
Copied []CopiedSnapshot
Skipped []SkippedSnapshot
BytesRead int64 // plaintext read through the source
BytesWritten int64 // bytes written to the destination backend
DryRun bool
SourceRepoID string
DestRepoID string
Duration time.Duration
}
type CopiedSnapshot struct {
SourceRef string // "snapshot/" in the source repository
DestRef string // "snapshot/" in this repository
Created string
Source *SourceInfo
}
type SkippedSnapshot struct {
SourceRef string
DestRef string // the snapshot it was already copied as
Created string
Reason string
Source *SourceInfo
}
```
`DestRef` is empty on a dry run: it names an object that has not been written.
### Example
```go theme={null}
dst, err := open.FromProfile(ctx, profilesPath, "remote-prod")
if err != nil {
log.Fatal(err)
}
src, err := open.FromProfile(ctx, profilesPath, "laptop-local")
if err != nil {
log.Fatal(err)
}
result, err := dst.CopyFrom(ctx, src,
cloudstic.WithCopyFilterTag("workstation"),
cloudstic.WithCopySince(time.Now().AddDate(0, -1, 0)),
)
if err != nil {
log.Fatal(err)
}
for _, s := range result.Copied {
fmt.Printf("copied %s -> %s\n", s.SourceRef, s.DestRef)
}
fmt.Printf("%d copied, %d skipped, read %d B, wrote %d B\n",
len(result.Copied), len(result.Skipped), result.BytesRead, result.BytesWritten)
```
### Notes
Copied snapshots keep their creation time, tags, source identity and file
metadata. Sequence numbers are reassigned, because they record write order
within a repository rather than snapshot identity.
Each copied snapshot records the source snapshot it came from, so calling
`CopyFrom` again skips what it already copied without re-reading the source.
That also makes an interrupted copy cheap to resume.
Copying is far more expensive than an incremental backup: object names derive
from each repository's encryption key, so every object is read and decrypted
through the source, then re-encrypted and written to the destination. Data the
destination already holds is still recognised and skipped.
Copying a repository into itself is refused — it would rewrite each snapshot as
a new one and double the history.
## Prune
```go theme={null}
func (c *Client) Prune(ctx context.Context, opts ...PruneOption) (*PruneResult, error)
```
Removes unreachable objects (mark-and-sweep garbage collection). Run after `Forget` to reclaim storage.
### Prune Options
| Option | Description |
| ------------------- | ----------------------------------------- |
| `WithPruneDryRun()` | Count deletions without removing anything |
### PruneResult
```go theme={null}
type PruneResult struct {
ObjectsScanned int
ObjectsDeleted int
BytesReclaimed int64
DryRun bool
}
```
## Forget
### Remove a specific snapshot
```go theme={null}
func (c *Client) Forget(ctx context.Context, snapshotID string, opts ...ForgetOption) (*ForgetResult, error)
```
### Apply a retention policy
```go theme={null}
func (c *Client) ForgetPolicy(ctx context.Context, opts ...ForgetOption) (*PolicyResult, error)
```
### Forget Options
| Option | Description |
| -------------------------------- | ------------------------------------------------------------------ |
| `WithForgetPrune()` | Run prune after forgetting |
| `WithForgetDryRun()` | Show what would be removed without deleting |
| `WithKeepLast(n int)` | Keep the N most recent snapshots |
| `WithKeepHourly(n int)` | Keep one snapshot per hour for the last N hours |
| `WithKeepDaily(n int)` | Keep one snapshot per day for the last N days |
| `WithKeepWeekly(n int)` | Keep one snapshot per week for the last N weeks |
| `WithKeepMonthly(n int)` | Keep one snapshot per month for the last N months |
| `WithKeepYearly(n int)` | Keep one snapshot per year for the last N years |
| `WithFilterTag(tag string)` | Only consider snapshots with this tag (repeatable) |
| `WithFilterSource(src string)` | Only consider snapshots from this source type |
| `WithFilterAccount(acct string)` | Only consider snapshots from this account |
| `WithFilterPath(path string)` | Only consider snapshots from this path |
| `WithGroupBy(fields string)` | Comma-separated grouping fields (default: `"source,account,path"`) |
### PolicyResult
```go theme={null}
type PolicyResult struct {
Groups []PolicyGroupResult
Prune *PruneResult // set only when WithForgetPrune() was passed
DryRun bool
}
type PolicyGroupResult struct {
Key GroupKey
Keep []KeepReason
Remove []SnapshotEntry
}
```
`GroupKey` and `KeepReason` are internal types you can read fields from (as in the example below) without importing them — see the note under [List](#list).
### Example
```go theme={null}
// Retention policy: keep last 10, 7 daily, 4 weekly, 12 monthly: then prune
result, err := client.ForgetPolicy(ctx,
cloudstic.WithKeepLast(10),
cloudstic.WithKeepDaily(7),
cloudstic.WithKeepWeekly(4),
cloudstic.WithKeepMonthly(12),
cloudstic.WithForgetPrune(),
)
if err != nil {
log.Fatal(err)
}
for _, group := range result.Groups {
fmt.Printf("Group %s: keep %d, remove %d\n",
group.Key, len(group.Keep), len(group.Remove))
}
if result.Prune != nil {
fmt.Printf("Reclaimed %d bytes\n", result.Prune.BytesReclaimed)
}
```
## Check
```go theme={null}
func (c *Client) Check(ctx context.Context, opts ...CheckOption) (*CheckResult, error)
```
Verifies repository integrity by walking the full reference chain (snapshots → HAMT nodes → filemeta → content → chunks).
### Check Options
| Option | Description |
| ----------------------------- | -------------------------------------------------- |
| `WithReadData()` | Re-hash all chunk data for byte-level verification |
| `WithSnapshotRef(ref string)` | Check only the specified snapshot (default: all) |
### CheckResult
```go theme={null}
type CheckResult struct {
SnapshotsChecked int
ObjectsVerified int
Errors []CheckError
}
type CheckError struct {
Type string // "missing", "corrupt", "unreadable"
Key string // object key
Message string
}
```
### Example
```go theme={null}
result, err := client.Check(ctx, cloudstic.WithReadData())
if err != nil {
log.Fatal(err)
}
if len(result.Errors) > 0 {
for _, e := range result.Errors {
fmt.Printf("[%s] %s: %s\n", e.Type, e.Key, e.Message)
}
}
```
## BreakLock
```go theme={null}
func (c *Client) BreakLock(ctx context.Context) ([]*RepoLock, error)
```
Removes stale repository lock files. Returns the list of removed locks (empty slice if none found).
```go theme={null}
removed, err := client.BreakLock(ctx)
if err != nil {
log.Fatal(err)
}
for _, lock := range removed {
fmt.Printf("Removed %s lock (held by %s)\n", lock.Operation, lock.Holder)
}
```
### ErrRepoLocked
`Backup`, `Restore`, and `Prune` return `ErrRepoLocked` when the repository is already held by another operation. Check for it with `errors.Is` and prompt the caller toward `BreakLock` rather than treating it as a generic failure:
```go theme={null}
result, err := client.Backup(ctx, src)
if errors.Is(err, cloudstic.ErrRepoLocked) {
// Another operation holds the repository. Only break the lock if you are
// certain nothing is actually running against it.
removed, breakErr := client.BreakLock(ctx)
// ...
}
```
## Cat
```go theme={null}
func (c *Client) Cat(ctx context.Context, keys ...string) ([]*CatResult, error)
```
Fetches the raw (decrypted, decompressed) data for one or more object keys. Useful for debugging and inspection.
### CatResult
```go theme={null}
type CatResult struct {
Key string // the object key requested
Data []byte // raw object data (typically JSON)
}
```
### Object Key Namespaces
| Key | Description |
| ----------------- | ------------------------------------- |
| `config` | Repository configuration marker |
| `index/latest` | Latest snapshot pointer |
| `index/snapshots` | Snapshot catalog |
| `snapshot/` | Snapshot manifest |
| `node/` | HAMT tree node |
| `filemeta/` | File metadata object |
| `content/` | Content manifest (list of chunk refs) |
| `chunk/` | Raw data chunk |
| `keys/` | Encryption key slot |
| `lock/` | Repository lock file |
### Example
```go theme={null}
results, err := client.Cat(ctx, "config", "index/latest")
if err != nil {
log.Fatal(err)
}
for _, r := range results {
fmt.Printf("=== %s ===\n%s\n\n", r.Key, r.Data)
}
```
## Key Management
These package-level functions operate on the raw (unencrypted) store.
### List key slots
```go theme={null}
func ListKeySlots(ctx context.Context, rawStore store.ObjectStore) ([]KeySlot, error)
```
Returns metadata for all key slots. Does not require authentication. Slot metadata is stored unencrypted.
```go theme={null}
slots, err := cloudstic.ListKeySlots(ctx, rawStore)
for _, s := range slots {
fmt.Printf("Type: %-12s Label: %s\n", s.SlotType, s.Label)
}
```
### Change password
```go theme={null}
func ChangePassword(ctx context.Context, rawStore store.ObjectStore, kc keychain.Chain, pwd PasswordProvider) error
```
Replaces the password key slot. `kc` must unlock the current master key. `pwd` supplies the new password.
Use `PasswordString` for a known value at call time:
```go theme={null}
err := cloudstic.ChangePassword(ctx, rawStore, kc,
cloudstic.PasswordString("new-passphrase"),
)
```
Use `PasswordProviderFunc` when the new password must be obtained lazily (e.g. interactive prompt):
```go theme={null}
err := cloudstic.ChangePassword(ctx, rawStore, kc,
cloudstic.PasswordProviderFunc(func(ctx context.Context) (string, error) {
return promptUser("New password: ")
}),
)
```
### Add recovery key
```go theme={null}
func AddRecoveryKey(ctx context.Context, rawStore store.ObjectStore, kc keychain.Chain, opts AddRecoveryKeyOptions) (string, error)
```
Generates a BIP39 recovery key slot and returns the 24-word mnemonic. `kc` must unlock the current master key.
```go theme={null}
type AddRecoveryKeyOptions struct {
Label string // names the slot; empty means the default slot
Replace bool // overwrite an existing slot with the same label
}
```
A distinct `Label` lets a repository hold several recovery keys, all valid at once. Without `Replace: true`, adding a recovery key under a label that already exists returns a `*keychain.SlotExistsError` and writes nothing — this protects a previously issued mnemonic from being silently invalidated.
```go theme={null}
mnemonic, err := cloudstic.AddRecoveryKey(ctx, rawStore, kc, cloudstic.AddRecoveryKeyOptions{})
if err != nil {
log.Fatal(err)
}
fmt.Println("Recovery key:", mnemonic)
// Store this securely: it is returned only once!
```
## Workstation Setup and Source Discovery
These are no longer part of the client library. Workstation onboarding —
discovering local backup candidates, proposing a set of profiles, and applying
that plan — moved to an internal package, because it is a CLI wizard rather
than a library capability: none of it needed a repository client, and the
entry points were package-level functions rather than `Client` methods.
Use the CLI instead:
```bash theme={null}
cloudstic source discover # list local backup candidates
cloudstic setup # plan and apply a set of profiles
```
The profiles file those commands write is still readable and writable
programmatically — see [Backup Profiles](#backup-profiles) below.
## Backup Profiles
The profiles YAML format lives in `github.com/cloudstic/cli/pkg/profile`. It is
a separate package from the client precisely so that reading or writing a
user's profiles does not require opening a repository:
```go theme={null}
import "github.com/cloudstic/cli/pkg/profile"
cfg, err := profile.Load(profilesPath) // or profile.LoadOrEmpty, which
if err != nil { // treats a missing file as empty
log.Fatal(err)
}
profile.EnsureMaps(cfg)
cfg.Profiles["documents"] = profile.Profile{
Source: "local:/Users/me/Documents",
Store: "primary",
Tags: []string{"workstation"},
}
if err := profile.Save(profilesPath, cfg); err != nil {
log.Fatal(err)
}
```
`Save` validates before writing, so a malformed secret reference is rejected
there rather than at backup time. The failure is a `*secretref.Error`, so you
can branch on `Kind` rather than matching on the message:
```go theme={null}
var refErr *secretref.Error
if errors.As(err, &refErr) && refErr.Kind == secretref.KindInvalidRef {
// the scheme://path in a profile field is malformed
}
```
| Type | Purpose |
| ----------------- | ------------------------------------------------------- |
| `profile.Config` | The whole file: `Version`, `Stores`, `Profiles`, `Auth` |
| `profile.Profile` | One backup job — source, store, tags, exclusions |
| `profile.Store` | A named store definition, including secret references |
| `profile.Auth` | A reusable cloud auth entry |
`profile.DefaultPath("")` returns where the file lives when the user names no
path — `profiles.yaml` inside the config directory, honouring
`CLOUDSTIC_CONFIG_DIR`. Use it so your program reads the same file the
`cloudstic` CLI does rather than a second one that silently disagrees. It
resolves a path without touching the filesystem, so asking the question creates
nothing.
### Acting on a Profile
Reading a profile is one thing; *using* one is another. `pkg/open` turns a named
profile into a connected client in one call:
```go theme={null}
import "github.com/cloudstic/cli/pkg/open"
// An empty path means the default location.
client, err := open.FromProfile(ctx, "", "documents")
if err != nil {
log.Fatal(err)
}
```
This is the one-call form of the explicit sequence, which you want instead
whenever something has to happen in between — displaying the resolved
configuration, diffing it, or deciding something from it before connecting:
```go theme={null}
cfg, err := profile.Load(profilesPath)
if err != nil {
log.Fatal(err)
}
// Which store does this profile select? nil means the profile names none,
// which is legal — the CLI then takes it from -store or CLOUDSTIC_STORE.
storeCfg, err := cfg.StoreFor("documents")
if err != nil {
log.Fatal(err)
}
clientCfg, err := config.FromProfileStore(ctx, *storeCfg, resolver)
if err != nil {
log.Fatal(err)
}
fmt.Printf("connecting to %s\n", clientCfg.Store.URI)
client, err := open.Client(ctx, clientCfg)
```
A client, once returned, is already connected — so anything you want to override
must be decided before this point, not after.
### Layering Your Own Configuration
If your program has a configuration mechanism of its own — command-line flags,
its own file, a form — `WithDecided` layers it over the profile, with the same
precedence the `cloudstic` CLI applies to its flags. Every field you name is
taken from your value, and the profile supplies the rest:
```go theme={null}
mine := config.Client{Store: config.Store{URI: "local:/my-own-choice"}}
client, err := open.FromProfile(ctx, "", "documents",
open.WithDecided(mine, config.FieldsSetIn(mine)))
```
`config.FieldsSetIn` reports which fields your value is non-empty for, which is
the right answer when your mechanism has no notion of "present but empty".
When empty *is* a choice you need to keep — `-password ""` meaning "no
password", as the CLI treats it — name the fields explicitly instead:
```go theme={null}
decided := config.NewFieldSet(config.FieldPassword, config.FieldS3AccessKey)
```
The field names are typed constants rather than strings, because the profiles
file and the CLI spell the same field differently — `s3_access_key` against
`-s3-access-key` — and a misspelled string silently meant "not decided", which
would connect you with a credential you had not chosen. `config.StoreFields()`
and `config.BackupFields()` enumerate the complete set, so iterating them keeps
your mapping current as fields are added; `Field.ProfileKey()` renders the
profiles-file spelling for an error message.
Two groups of fields behave differently on purpose. Location and KMS settings
are taken only when the profile actually names one, so a silent profile leaves
what you had. Credentials are taken whenever you have not decided them, *empty
included* — selecting a profile clears an ambient credential, because a profile
is an explicit choice of which store to talk to, and reaching it with half a
credential set inherited from the environment would be worse than failing to
reach it.
A field you have decided is never resolved, so a broken secret reference on a
field you are about to replace is not an error.
### Backing Up From a Profile
The same layering applies to what a profile says to back up. `config.Backup`
holds the resolved form, and `open.Backup` constructs both the source and the
options a backup runs with:
```go theme={null}
bcfg, err := config.MergeProfileBackup(config.Backup{}, nil, "documents", cfg)
if err != nil {
log.Fatal(err)
}
job, err := open.Backup(ctx, bcfg)
if err != nil {
log.Fatal(err)
}
result, err := client.Backup(ctx, job.Source, job.Options...)
```
`open.Backup` returns the source and the options **together** rather than as two
calls, because they share a derived value. The snapshot records a hash of the
active exclude patterns, and the next backup compares its own against it to
decide between an incremental scan and a full rescan. The patterns come from two
places — `Excludes` and the file named by `ExcludeFile` — so a hash computed
without reading that file describes a different exclude set than the one in
force, and every subsequent run reads the difference as "the patterns changed".
Getting the source and the options from one call is what keeps them consistent.
If you supply your own `source.Source` implementation, use
`open.BackupOptions(cfg)` for the options and apply `cfg.Source.Excludes`
yourself — otherwise the recorded hash describes filtering that is not happening.
`source.ExcludeHash` is exported if you need to compute it directly.
| Function | Purpose |
| ----------------------------------------------------- | ------------------------------------------------- |
| `open.FromProfile(ctx, path, name, opts...)` | Profile → connected client, in one call |
| `open.Client(ctx, config.Client, opts...)` | Resolved configuration → connected client |
| `open.Store(ctx, config.Store, opts...)` | Store URI + credentials → raw `store.ObjectStore` |
| `open.Source(ctx, config.Source, opts...)` | Source URI + credentials → `source.Source` |
| `open.Backup(ctx, config.Backup, opts...)` | → `BackupJob{Source, Options}` |
| `config.FromProfileStore(ctx, s, r)` | Profile store definition → `config.Client` |
| `config.MergeProfileStore(ctx, base, decided, s, r)` | The same, layered under your own values |
| `config.MergeProfileBackup(base, decided, name, cfg)` | Profile backup settings → `config.Backup` |
## Secret References
Store and source credentials can be written as `scheme://path` references
instead of inline secrets. `github.com/cloudstic/cli/pkg/secretref` resolves
them, and `pkg/secretref/backends` holds the schemes Cloudstic ships with:
`env://`, `file://`, `config-token://`, `keychain://`, `secret-service://` and
`wincred://`.
```go theme={null}
import (
"github.com/cloudstic/cli/pkg/secretref"
"github.com/cloudstic/cli/pkg/secretref/backends"
)
resolver := backends.NewDefaultResolver()
password, err := resolver.Resolve(ctx, "keychain://cloudstic/store/prod")
```
Failures are typed, so you can tell a malformed reference from a missing value
from a backend that is not available on this platform:
| `Kind` | Meaning |
| ---------------------------------- | ------------------------------------------------------------------- |
| `secretref.KindInvalidRef` | The `scheme://path` is malformed |
| `secretref.KindNotFound` | The backend works, but has no value at that path |
| `secretref.KindBackendUnavailable` | No backend is registered for that scheme, or it is unsupported here |
### Adding your own scheme
`Backend` is a single method, so a custom scheme — Vault, a cloud KMS, your own
service — is small. `backends.Default()` returns a **fresh map on every call**,
which is what lets you *extend* the built-in set rather than replace it:
```go theme={null}
type vaultBackend struct{ client *vault.Client }
func (v *vaultBackend) Resolve(ctx context.Context, ref secretref.Ref) (string, error) {
secret, err := v.client.Read(ctx, ref.Path)
if err != nil {
return "", secretref.NewError(secretref.KindNotFound, ref.Raw, "no such vault secret", err)
}
return secret, nil
}
b := backends.Default()
b["vault"] = &vaultBackend{client: c}
resolver := secretref.NewResolver(b)
```
Use `secretref.NewError` for failures so callers can branch on `Kind` the same
way they do for the built-in backends.
Implement `WritableBackend` as well (adding `Scheme`, `DisplayName`,
`WriteSupported`, `DefaultRef`, `Exists` and `Store`) if the scheme should
also be offered when the CLI and TUI *write* a secret, rather than only when
resolving one.
### Choosing where managed tokens live
The `config-token://` backend stores tokens itself, encrypted at rest, and by
default puts them in Cloudstic's own config directory — `CLOUDSTIC_CONFIG_DIR`
or the OS default. For an embedding program that is usually the wrong place,
and an environment variable is a poor way for a library to be configured:
```go theme={null}
b := backends.Default()
b["config-token"] = backends.NewConfigTokenBackend(
backends.WithConfigDir("/var/lib/myapp/secrets"),
)
resolver := secretref.NewResolver(b)
```
The salt that derives the at-rest encryption key lives in that directory too.
Two backends pointed at different directories therefore hold different keys —
tokens written under one are **not** readable under the other. Changing the
directory of an existing deployment orphans whatever was already stored there.
## Progress Reporting
`cloudstic.Reporter` receives progress events during long-running operations:
```go theme={null}
type Reporter interface {
// total is 0 when the size is not known in advance; isBytes selects
// byte units over a plain count.
StartPhase(name string, total int64, isBytes bool) Phase
}
type Phase interface {
Increment(n int64)
Log(msg string)
Logf(level Detail, format string, args ...any)
Done()
Error()
}
```
### Detail
How much an operation says about what it is doing is **your** decision, not the operation's:
```go theme={null}
const (
DetailNormal Detail = iota // milestones: phases, counts, warnings
DetailVerbose // per-item: every object verified, every file restored
)
```
`Logf` formats its arguments only if your reporter wants that level, which matters because per-item logging runs once per object — formatting a string you will discard is work proportional to repository size.
Operations have no verbosity option. Each one used to carry its own
(`WithVerbose`, `WithCheckVerbose`, `WithFindVerbose`, and six more), which put
a presentation decision inside the engine — nine names for one idea, decided by
the code producing the events rather than the code displaying them. Discard the
levels you do not want in your `Phase` implementation.
Operations that have no phases — `list`, `ls`, `diff` and `find` are queries, not long-running work — report their detail through the logger instead, which you supply with `WithLogger`.
Pass your reporter to `NewClient`:
```go theme={null}
client, err := cloudstic.NewClient(ctx, rawStore,
cloudstic.WithKeychain(kc),
cloudstic.WithReporter(myReporter),
)
```
If `WithReporter` is omitted, all progress output is suppressed (no-op default).
## Complete Example: Automated Backup
```go theme={null}
package main
import (
"context"
"encoding/hex"
"fmt"
"log"
"os"
cloudstic "github.com/cloudstic/cli"
"github.com/cloudstic/cli/pkg/keychain"
localsource "github.com/cloudstic/cli/pkg/source/local"
localstore "github.com/cloudstic/cli/pkg/store/local"
)
func main() {
ctx := context.Background()
// Load platform key from environment
encKeyHex := os.Getenv("CLOUDSTIC_ENCRYPTION_KEY")
encKey, err := hex.DecodeString(encKeyHex)
if err != nil {
log.Fatalf("invalid CLOUDSTIC_ENCRYPTION_KEY: %v", err)
}
// Open S3 backend
rawStore, err := s3store.New(ctx, "my-backup-bucket",
s3store.WithRegion("us-east-1"),
)
if err != nil {
log.Fatal(err)
}
// Build keychain
kc := keychain.Chain{keychain.WithPlatformKey(encKey)}
// Initialize if not already done (adopt-slots makes it idempotent)
if _, err := cloudstic.InitRepo(ctx, rawStore,
cloudstic.WithInitCredentials(kc),
cloudstic.WithInitAdoptSlots(),
); err != nil {
log.Fatal(err)
}
// Create client
client, err := cloudstic.NewClient(ctx, rawStore, cloudstic.WithKeychain(kc))
if err != nil {
log.Fatal(err)
}
// Backup
src := localsource.New("/data",
localsource.WithExcludePatterns([]string{"*.tmp", ".cache/"}),
)
result, err := client.Backup(ctx, src, cloudstic.WithTags("automated"))
if err != nil {
log.Fatalf("backup failed: %v", err)
}
fmt.Printf("Backup complete: %s (%d new, %d changed files)\n",
result.SnapshotHash, result.FilesNew, result.FilesChanged)
// Apply retention policy and prune
_, err = client.ForgetPolicy(ctx,
cloudstic.WithKeepLast(7),
cloudstic.WithKeepWeekly(4),
cloudstic.WithKeepMonthly(12),
cloudstic.WithForgetPrune(),
)
if err != nil {
log.Fatalf("retention policy failed: %v", err)
}
}
```
## See Also
* [Source Interface](/advanced/source-interface): Implement a custom backup source
* [Storage Model](/advanced/storage-model): How objects are stored and addressed
* [HAMT Structure](/advanced/hamt-structure): The file index data structure
* [Backup Flow](/advanced/backup-flow): End-to-end walkthrough of the backup process
# Repository Compatibility
Source: https://docs.cloudstic.com/advanced/compatibility
What Cloudstic guarantees about reading old repositories and refusing repositories that are too new
Cloudstic is a backup tool, and restoring an old backup is not a nice-to-have — it is the product. A repository written years ago, by a version of Cloudstic nobody runs any more, must still restore today. This page explains the compatibility guarantee behind that promise, and what happens if you ever point an older Cloudstic build at a newer repository.
## Backward compatibility is permanent
**Any repository written by any released version of Cloudstic remains readable by every later version, forever.**
There is no deprecation window and no expiry date on this guarantee. Support for old repository layouts may be refactored or made faster internally, but it is never removed, and it never changes behavior. Concretely, this means `list`, `ls`, `check`, `cat`, `diff`, and `restore` all keep working against a repository no matter how old it is. Writing to an old repository (`backup`, `prune`, `forget`) is also supported, and may upgrade parts of the repository's on-disk format in place as a side effect.
You never need to run a migration command. Opening an old repository with a current build of Cloudstic just works.
## Newer repositories, older builds
The reverse direction is not guaranteed: an older Cloudstic build is not guaranteed to fully understand a repository written by a newer one. Formats do change over time, and that's expected.
What Cloudstic guarantees instead is that this situation **fails safely**. An older build that encounters a repository format it doesn't recognize will refuse to operate on it, rather than silently misreading it. If you try, you'll see an error like:
```text theme={null}
repository format version 2 is newer than this build supports (up to 1):
upgrade cloudstic to work with this repository
```
**The fix is simple: upgrade Cloudstic.** Once you're running a build that supports the repository's format, everything works normally again.
Never work around this error by rolling back a repository or hand-editing its `config` marker. The version gate exists to stop an operation from destroying data it can't fully see — bypassing it removes that protection.
## Why the gate matters: don't confuse "unreadable" with "empty"
The reason Cloudstic refuses outright, instead of doing its best with what it can parse, is a specific and serious failure mode: treating an index that failed to load as if it were an empty index.
An empty index legitimately means "nothing is referenced." But a *failed-to-load* index means "we don't know what's referenced" — and those are very different statements. If a garbage collector (`prune`) ever confused the two, it would see "nothing is referenced" and delete everything, including live, healthy data it simply failed to read.
This isn't hypothetical. An early Cloudstic release, pointed at a repository containing a data structure it couldn't decode, reported `0 snapshots` with no error — and running `prune` against it deleted the repository's packfiles. That incident is why the version gate exists: any build that cannot fully understand a repository now refuses to touch it at all, rather than guessing.
## How repositories actually get upgraded
There is no separate "migrate this repository" command, and no moment where a repository becomes "fully upgraded." Upgrades happen **in place and opportunistically**, as a byproduct of normal write operations:
* The first time a newer Cloudstic build runs `backup`, `prune`, or `forget` against an older repository, it may write some data in the newer format alongside data still in the old format.
* Only the parts of the repository actually touched by that write get upgraded. Everything else stays as it was.
That means a long-lived, frequently-used repository is, indefinitely, a **mixture of format eras** — and that's the intended steady state, not a transitional phase you need to "finish." Because backward compatibility never expires, nothing ever needs that mixture to fully resolve.
If you run a fleet of machines against a shared repository, keep every machine's Cloudstic build reasonably current. A machine that writes to the repository can raise its recorded format version — which is a real, useful signal that the repository now contains something newer builds understand and older ones might not. Letting one machine lag far behind risks it hitting the "please upgrade" error the next time it tries to write.
## Summary
| Direction | Guarantee |
| --------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Older repository, newer Cloudstic | Always fully readable and writable. No expiry, no migration step required. |
| Newer repository, older Cloudstic | Not guaranteed to work. Fails cleanly with an "upgrade Cloudstic" error rather than misreading data. |
This is a deliberate design tradeoff: Cloudstic would rather stop you with a clear error than risk destroying your backups by guessing at a format it doesn't fully understand.
## See also
* [Storage Model](/advanced/storage-model): how objects, packfiles, and indexes are laid out on disk
* [cloudstic check](/commands/check): verify repository integrity
# Contributing Guide
Source: https://docs.cloudstic.com/advanced/contributing
Guidelines for contributing to Cloudstic CLI development
Welcome! We appreciate your help in making Cloudstic better. This guide covers development setup, testing, debugging, and contribution workflows.
## Development Setup
### Prerequisites
* Go 1.26 or later (see `go.mod` for the exact minimum)
* Docker (for hermetic E2E tests using Testcontainers)
* `golangci-lint` (for linting)
* Node.js (for `markdownlint-cli2`, run via `npx` by `scripts/check.sh`)
### Clone the Repository
```bash theme={null}
git clone https://github.com/cloudstic/cli.git
cd cli
```
### Build the Binary
```bash theme={null}
go build -o bin/cloudstic ./cmd/cloudstic
```
The binary will be created at `bin/cloudstic`.
## Project Structure
Cloudstic CLI is organized into clear package boundaries:
* **`client.go`** (root) - Public `Client` API for programmatic use. Re-exports types from internal packages via Go type aliases.
* **`cmd/cloudstic/`** - CLI entry point (`package main`). `main.go` is a thin dispatcher; each command lives in its own `cmd_.go` file, registered in `commands.go`'s `commandRegistry()`. See [CLI Integration](#4-cli-integration) below.
* **`internal/engine/`** - Business logic for operations (backup, restore, prune, forget, diff, list, find). Each operation has a `*Manager` struct.
* **`internal/core/`** - Repository-format types: `Snapshot`, `Content`, `HAMTNode`, `RepoConfig`, plus `ComputeJSONHash`. `FileMeta`, `SourceInfo` and `FileType` are defined in `pkg/source` and aliased here, so the public Source contract does not depend on an internal package.
* **`internal/hamt/`** - Persistent Merkle Hash Array Mapped Trie backed by the object store.
* **`pkg/source/`** - The `Source` and `IncrementalSource` contract, plus shared helpers (`ExcludeMatcher`, `TopoSortFolderChanges`). Depends on nothing outside the standard library, so implementing a source pulls in no provider SDK. Implementations live in their own subpackages: `pkg/source/{local,sftp,gdrive,onedrive}`. See the [Source Interface](/advanced/source-interface) guide.
* **`internal/sourceoauth/`** - OAuth2 machinery shared by the Google Drive and OneDrive sources. Also holds the `-X` ldflags targets for the default OAuth client IDs; those symbol paths are mirrored in `.goreleaser.yml` and must move together, since the linker silently ignores `-X` for a symbol that does not exist.
* **`pkg/store/`** - The `ObjectStore` contract, its capability interfaces (`RangeGetter`, `ConcurrencyHinter`, `Unwrapper`) and the order-independent wrappers `QuotaStore`/`DebugStore`. Depends on nothing outside the standard library, so implementing a custom backend pulls in no vendor SDK. Backends live in `pkg/store/{local,s3,b2,sftp}/` (`local.New`, `s3.New`, …); only `pkg/store/s3` carries the AWS SDK. `pkg/store/storetest` holds shared test doubles.
* **`internal/storelayer/`** - The repository-format decorator chain (compression, encryption, metering, packfiles, key cache). Internal because its composition order is a security invariant — see the Store Decorator Stack below.
* **`pkg/crypto/`** - AES-256-GCM encryption, HKDF key derivation, BIP39 mnemonic recovery keys.
* **`pkg/keychain/`** - OS keychain integration and encryption key-slot helpers.
* **`internal/app/`** - Orchestration layer shared by the CLI and TUI (profile listing, health checks, backup actions).
* **`internal/tui/`** - The interactive terminal dashboard (Bubble Tea).
* **`pkg/secretref/`** - The `scheme://path` secret-reference contract: `Ref`, `Parse`, `Backend`, `Resolver`, `NewResolver`, `Error`. Public so a third party can register a custom backend (Vault, a cloud KMS) from another module.
* **`pkg/secretref/backends/`** - The built-in backends (`env://`, `file://`, `config-token://`, `keychain://`, `secret-service://`, `wincred://`) plus `Default()`, which returns a fresh map so callers *extend* the built-in set rather than replacing it.
* **`pkg/profile/`** - The backup-profiles YAML format (`Config`, `Profile`, `Store`, `Auth`, `Load`, `Save`). Separate from the client so profiles can be read and written without opening a repository.
* **`internal/workstation/`** - Workstation onboarding (`Plan`, `Apply`, `Setup`) and local source discovery. Internal: it is a CLI wizard, not a library capability.
* **`internal/paths/`** - Config-directory and token-path resolution.
* **`internal/pathmatch/`** - Glob matching for slash-separated paths, including `**`.
* **`internal/logger/`, `internal/retry/`, `internal/sftp/`** - Structured logging, retry/backoff helpers, and the shared SFTP client used by both the SFTP source and store.
* **`internal/ui/`** - Non-interactive console progress reporting and terminal helpers.
See `AGENTS.md` in the repository root for the full, actively-maintained architecture documentation — it's the canonical reference this page is kept in sync with.
## Build & Test Commands
### Run All Tests
```bash theme={null}
go test -v -race -count=1 ./...
```
This runs unit tests and hermetic E2E tests (using Testcontainers for MinIO and SFTP).
Docker is required for hermetic E2E tests. Tests will be skipped if `/var/run/docker.sock` is not available.
### Run a Single Test
```bash theme={null}
go test -v -run TestName ./path/to/package
```
Example:
```bash theme={null}
go test -v -run TestCLI_Feature_BackupRestoreLatest ./e2e
```
### Run with Race Detector
```bash theme={null}
go test -v -race -count=1 ./...
```
The race detector catches concurrency bugs. Always run tests with `-race` during development.
### Run the Full Check Script
```bash theme={null}
./scripts/check.sh
```
This runs:
1. `go fmt` - Format check
2. `golangci-lint run` - Linting
3. `markdownlint-cli2` (via `npx`) - Markdown lint, including this guide and every other `.md` file in the repo
4. `go test -race -count=1 ./...` - Tests with race detection
5. Coverage report generation
### Format Code
```bash theme={null}
go fmt ./...
```
### Lint Code
```bash theme={null}
golangci-lint run ./...
```
## E2E Test Modes
E2E tests in `e2e/` are controlled by the `CLOUDSTIC_E2E_MODE` environment variable:
* **`hermetic`** (default) - Local filesystem + Testcontainers (MinIO, SFTP). Requires Docker.
* **`live`** - Real cloud vendor APIs (requires secrets in environment variables).
* **`all`** - Runs both hermetic and live tests.
### Running Hermetic Tests
```bash theme={null}
go test -v ./e2e
```
or explicitly:
```bash theme={null}
CLOUDSTIC_E2E_MODE=hermetic go test -v ./e2e
```
### Running Live Tests
Live tests require cloud provider credentials (AWS, Backblaze B2, Google Drive, OneDrive, SFTP servers) configured via environment variables.
```bash theme={null}
CLOUDSTIC_E2E_MODE=live go test -v ./e2e
```
### Running All Tests
```bash theme={null}
CLOUDSTIC_E2E_MODE=all go test -v ./e2e
```
## Debugging
### Enable Debug Logging
Append the `-debug` flag to any CLI command to enable verbose internal logging:
```bash theme={null}
cloudstic backup -source local:./data -debug
```
This outputs:
* Detailed timings for every `GET`, `PUT`, `LIST`, and `DELETE` operation
* Cache hits/misses
* Memory management decisions
* Engine operation traces
Debug logging is extremely useful for tracing API calls, caching behaviors, and performance bottlenecks.
### Attach a Debugger
You can use `dlv` (Delve) to debug the CLI:
```bash theme={null}
go install github.com/go-delve/delve/cmd/dlv@latest
dlv debug ./cmd/cloudstic -- backup -source local:./data
```
Or attach to a running process:
```bash theme={null}
dlv attach
```
## Profiling
Cloudstic supports standard Go profiling via hidden flags on any command:
### CPU Profiling
```bash theme={null}
cloudstic backup -source local:./data -cpuprofile cpu.prof
go tool pprof -http=:8080 cpu.prof
```
The CPU profile flag also automatically generates:
* `cpu.prof.goroutine` - Goroutine dump
* `cpu.prof.block` - Block profile
* `cpu.prof.mutex` - Mutex profile
### Memory Profiling
```bash theme={null}
cloudstic backup -source local:./data -memprofile mem.prof
go tool pprof -http=:8080 mem.prof
```
### View Profiles
Use `go tool pprof` to analyze profiles:
```bash theme={null}
# Interactive mode
go tool pprof cpu.prof
# Web UI
go tool pprof -http=:8080 cpu.prof
# Generate flame graph
go tool pprof -http=:8080 -flame cpu.prof
```
## Development Best Practices
### When Adding New Features
Always consider the following:
#### 1. Documentation
Check if user-facing documentation needs updates:
* `docs/user-guide.md` - Add command documentation with usage examples, flags, and descriptions.
* `README.md` - Update if the feature changes the quick start or high-level overview.
* Code comments - Document public APIs, especially in `client.go` and package interfaces.
#### 2. Unit Tests
Add test coverage when it makes sense:
* Always add tests for new public API methods (e.g., `Client.*()` methods).
* Test both success and error cases.
* Test integration with encryption/compression if applicable.
* Use existing test patterns (see `client_test.go`, `internal/engine/*_test.go`).
* Mock stores are available in `internal/engine/mock_test.go` for testing.
#### 3. Client API
For new operations, expose them via the `Client` struct:
* CLI commands should use `Client` methods, not directly access stores.
* This allows library users to programmatically use the functionality.
* Follow the pattern: define types/options, add a `Client.*()` method, implement in `internal/engine/` if complex.
#### 4. CLI Integration
For new commands, don't hand-write a flag parser or edit `main.go`. Instead:
* Add a `cmd_.go` file next to the code it implements, following the pattern every existing command uses (`cmd_backup.go`, `cmd_check.go`, `cmd_breaklock.go`, ...):
* `declareArgs(g *globalFlags) (*Args, commandInput)` declares the command's complete input surface — flags via `stringFlag`/`boolFlag`/`intFlag`/`valueFlag`, positionals via `requiredPositional`/`optionalPositional`. This also carries environment, secret, help, and completion metadata.
* `run(r *runner, ctx context.Context, a *Args) int` is the command body. It writes output via `r.out`/`r.errOut` (never `fmt.Print`) so it's capturable in tests.
* A `Command() command` function declares the runnable leaf: `leaf(name, summary, groups, declareArgs, run)`. Pass `repoCommandGroups` (or `backupCommandGroups` for a command that reads a source) to opt into only the global flags the command needs.
* Add the new `Command()` to the ordered list in `commandRegistry()` (`cmd/cloudstic/commands.go`) — that's the single source of truth `runCmd()` dispatches from, `printUsage()` renders `COMMANDS` from, and shell completion is generated from. Never edit `main.go`, `usage.go`, or `completion.go` directly for a new command.
* Mark any flag carrying a credential with `asSecret()` so it never leaks into `-h` output (`TestSecretEnvValuesNeverAppearInHelp` enforces this).
* If the command changes root or per-command help text, regenerate the golden files: `go test ./cmd/cloudstic -run 'TestRootUsageGolden|TestCommandHelpGolden' -update`.
#### 5. Error Handling
Return descriptive errors:
* Wrap errors with context using `fmt.Errorf("context: %w", err)`.
* Provide actionable error messages to users.
* Distinguish between user errors and system errors.
* If a caller (CLI or library) needs to programmatically detect a specific failure mode — not just display it — define a sentinel error (`var ErrX = errors.New(...)`) and wrap it with `%w`, rather than making the caller match on error text. See `engine.ErrRepoLocked` / `cloudstic.ErrRepoLocked` for a recent example wired through to a CLI hint.
### Example: Adding a New Command
Let's say you want to add a `stats` command that shows repository statistics.
**Step 1: Add Client Method**
```go theme={null}
// client.go
type StatsResult struct {
TotalSnapshots int64
TotalObjects int64
TotalBytes int64
}
func (c *Client) Stats(ctx context.Context) (*StatsResult, error) {
mgr := engine.NewStatsManager(c.store)
return mgr.Run(ctx)
}
```
**Step 2: Implement Engine Logic**
```go theme={null}
// internal/engine/stats.go
type StatsManager struct {
store store.ObjectStore
}
func NewStatsManager(s store.ObjectStore) *StatsManager {
return &StatsManager{store: s}
}
func (m *StatsManager) Run(ctx context.Context) (*StatsResult, error) {
// Implementation here
}
```
**Step 3: Declare and Implement the CLI Command**
```go theme={null}
// cmd/cloudstic/cmd_stats.go
package main
import (
"context"
"fmt"
"io"
cloudstic "github.com/cloudstic/cli"
)
type statsArgs struct {
*globalFlags
verbose bool
}
func declareStatsArgs(g *globalFlags) (*statsArgs, commandInput) {
a := &statsArgs{globalFlags: g}
return a, commandInput{
flags: []flagSpec{
boolFlag(&a.verbose, "verbose", false, "Log per-object detail"),
},
}
}
func runStats(r *runner, ctx context.Context, a *statsArgs) int {
if err := r.openClient(ctx, a.globalFlags); err != nil {
return r.fail("Failed to init store: %v", err)
}
result, err := r.client.Stats(ctx)
if err != nil {
return r.fail("Stats failed: %v", err)
}
if a.jsonEnabled() {
return r.writeJSON(result)
}
printStatsResult(r.out, result)
return 0
}
func printStatsResult(out io.Writer, result *cloudstic.StatsResult) {
fmt.Fprintf(out, "Snapshots: %d\n", result.TotalSnapshots)
fmt.Fprintf(out, "Objects: %d\n", result.TotalObjects)
fmt.Fprintf(out, "Size: %d bytes\n", result.TotalBytes)
}
// statsCommand declares the `stats` command.
func statsCommand() command {
return leaf("stats", "Show repository statistics",
repoCommandGroups, declareStatsArgs, runStats)
}
```
Note the shape: `run` takes the shared `runner` (so output is capturable and `r.client` is injectable in tests) instead of building its own client and flag set; presentation lives in a separate `print*` function taking `io.Writer` first, never given access to `runner`.
**Step 4: Register the Command**
```go theme={null}
// cmd/cloudstic/commands.go - add one entry to commandRegistry()
func commandRegistry() []command {
return []command{
initCommand(),
backupCommand(),
// ... existing commands ...
statsCommand(),
}
}
```
That single entry is enough: dispatch (`runCmd()` in `main.go`), the `cloudstic help` listing, and shell completion all derive from this list.
**Step 5: Add Tests**
```go theme={null}
// client_test.go — public API
func TestStats(t *testing.T) {
store := newMockStore()
client, _ := cloudstic.NewClient(context.Background(), store)
result, err := client.Stats(context.Background())
if err != nil {
t.Fatal(err)
}
if result.TotalSnapshots < 0 {
t.Errorf("expected non-negative snapshots, got %d", result.TotalSnapshots)
}
}
```
```go theme={null}
// cmd/cloudstic/cmd_stats_test.go — CLI flow, via stubClient (no real repository)
func TestRunStats(t *testing.T) {
var out bytes.Buffer
r := newRunner(nil)
r.out = &out
r.client = &stubClient{statsResult: &cloudstic.StatsResult{TotalSnapshots: 3}}
if code := statsCommand().execute(r, context.Background(), "stats"); code != 0 {
t.Fatalf("exit code = %d, want 0", code)
}
if !strings.Contains(out.String(), "Snapshots: 3") {
t.Fatalf("output = %q, missing snapshot count", out.String())
}
}
```
**Step 6: Update Documentation**
Add command documentation to `docs/user-guide.md` and (if `stats` has notable flags) run `go test ./cmd/cloudstic -run TestCommandHelpGolden -update` to refresh its golden `-h` output.
## Testing Guidelines
Choose the smallest test style that covers the behavior — the four styles below cover different layers, and picking the wrong one for a given change is a common review comment:
* **Unit tests** for argument parsing, orchestration branches, domain logic, and individual error cases. Inject a `stubClient` (`cmd/cloudstic/stub_client_test.go`) to test CLI command flow without a real repository.
* **Golden-file tests** for deterministic `print*`/`render*` presentation output, where the exact full text is the contract.
* **Testscript tests** for whole-command behavior that crosses the process boundary: flag ordering, stdout/stderr separation, exit codes, filesystem effects.
* **E2E tests** (`e2e/`, Testcontainers-backed) for behavior that needs a real backend (MinIO, SFTP) end to end.
Don't reach for a golden file when a value is inherently unstable (timestamps, hashes), and don't replace a focused unit test with a testscript when a direct assertion gives a clearer failure message.
### Test Coverage
Aim for high test coverage, especially for:
* Public API methods in `client.go`
* Engine logic in `internal/engine/`
* Store implementations in `pkg/store/`
* Crypto operations in `pkg/crypto/`
### Test Patterns
#### Unit Tests
Use mock stores for isolated engine testing:
```go theme={null}
func TestBackupManager(t *testing.T) {
store := &MockStore{}
source := &MockSource{}
mgr := engine.NewBackupManager(source, store, ui.NewNoOpReporter(), nil)
result, err := mgr.Run(context.Background())
if err != nil {
t.Fatal(err)
}
if result.FilesNew == 0 {
t.Error("expected files to be added")
}
}
```
For CLI command flow, inject a `stubClient` instead of a real repository — see `TestRunStats` in the command example above.
#### Golden-File Tests
Write deterministic output to a buffer and compare it against `cmd/cloudstic/testdata/*.golden` with `assertGolden`:
```go theme={null}
func TestPrintFindResult(t *testing.T) {
var buf bytes.Buffer
printFindResult(&buf, someFindResult)
assertGolden(t, "print_find_result", buf.String())
}
```
Regenerate an intentionally changed golden file with `-update`, and review the diff as part of the change:
```bash theme={null}
go test ./cmd/cloudstic -run TestPrintFindResult -update
```
#### Testscript Tests
Add a hermetic `.txtar` script under `cmd/cloudstic/testdata/scripts/` for whole-process behavior. Prefer local stores/sources so the script needs no network access, credentials, or Docker:
```text theme={null}
# cmd/cloudstic/testdata/scripts/find_across_snapshots.txtar
exec cloudstic init -store local:$WORK/repo -no-encryption -no-prompt
stderr 'Repository initialized \(encrypted: false\)'
exec cloudstic backup -store local:$WORK/repo -source local:$WORK/source -no-prompt
stdout 'Snapshot .* saved'
exec cloudstic find 'vault.kdbx' -store local:$WORK/repo -no-prompt
stdout 'vault\.kdbx'
-- source/vault.kdbx --
first version
```
#### Integration Tests
Use real stores with temporary directories:
```go theme={null}
func TestBackupRestore(t *testing.T) {
tmpDir := t.TempDir()
rawStore, _ := localstore.New(tmpDir)
client, _ := cloudstic.NewClient(context.Background(), rawStore)
// Run backup
src := localsource.New("testdata")
backupResult, err := client.Backup(context.Background(), src)
if err != nil {
t.Fatal(err)
}
// Run restore
var buf bytes.Buffer
restoreResult, err := client.Restore(context.Background(), &buf, backupResult.SnapshotHash)
if err != nil {
t.Fatal(err)
}
if int64(restoreResult.FilesWritten) != backupResult.FilesNew {
t.Errorf("expected %d files restored, got %d",
backupResult.FilesNew, restoreResult.FilesWritten)
}
}
```
#### E2E Tests
Use Testcontainers for hermetic E2E tests:
```go theme={null}
func TestBackupToS3(t *testing.T) {
if os.Getenv("CLOUDSTIC_E2E_MODE") == "" {
os.Setenv("CLOUDSTIC_E2E_MODE", "hermetic")
}
if os.Getenv("CLOUDSTIC_E2E_MODE") == "live" {
t.Skip("skipping hermetic test in live mode")
}
// Start MinIO container
ctx := context.Background()
minioC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: testcontainers.ContainerRequest{
Image: "minio/minio:latest",
ExposedPorts: []string{"9000/tcp"},
Cmd: []string{"server", "/data"},
Env: map[string]string{
"MINIO_ROOT_USER": "minioadmin",
"MINIO_ROOT_PASSWORD": "minioadmin",
},
WaitingFor: wait.ForHTTP("/minio/health/live").WithPort("9000/tcp"),
},
Started: true,
})
if err != nil {
t.Fatal(err)
}
defer minioC.Terminate(ctx)
// Get endpoint
endpoint, _ := minioC.Endpoint(ctx, "")
// Test backup to MinIO
rawStore, _ := s3store.New(ctx, "test-bucket",
s3store.WithEndpoint(endpoint),
s3store.WithCredentials("minioadmin", "minioadmin"),
s3store.WithRegion("us-east-1"),
)
client, _ := cloudstic.NewClient(ctx, rawStore)
src := localsource.New("testdata")
result, err := client.Backup(ctx, src)
if err != nil {
t.Fatal(err)
}
if result.FilesNew == 0 {
t.Error("expected files to be added")
}
}
```
### Before Committing
Always run the full check script:
```bash theme={null}
./scripts/check.sh
```
This ensures:
* Code is formatted correctly
* No linting errors
* All tests pass
* Race conditions are detected
* Coverage is adequate
## Architecture Overview
### Store Layering
Stores are composed as a decorator chain (from outermost to innermost):
```
CompressedStore → EncryptedStore → MeteredStore → [PackStore] → KeyCacheStore →
```
* **CompressedStore** - zstd compression on write, auto-detects zstd/gzip/raw on read.
* **EncryptedStore** - AES-256-GCM. Passes through objects under `keys/` prefix unencrypted.
* **MeteredStore** - Tracks bytes written for reporting.
* **PackStore** (optional) - Bundles small objects (less than 512KB) into 8MB packfiles to reduce API calls. Each packfile ends with a self-describing footer, so its catalog (`index/packs`) is a rebuildable cache rather than the sole source of truth — see [RFC 0018](https://github.com/Cloudstic/cli/blob/main/rfcs/0018-self-describing-packfiles.md) if you're touching this layer.
* **KeyCacheStore** - Caches key existence in a temporary bbolt database.
* **Backend** - `local.Store`, `s3.Store`, `b2.Store`, or `sftp.Store`, each in its own subpackage under `pkg/store/`.
Do not assemble this chain yourself. `PackStore` sits **below**
`EncryptedStore`, so its catalog and footers never pass through encryption and
need a separately derived key; a hand-built chain missing it produces a
repository whose pack index is plaintext, with no error at any layer. That is
why these types live in `internal/storelayer`. To add your own wrapper,
implement `store.ObjectStore` and pass it to `NewClient` — it layers the chain
on top, exactly as the CLI does with `DebugStore`.
## Repository Compatibility
This is the most important constraint in the codebase for anything that touches what gets written to a store. Read `docs/compatibility.md` in the repository before changing on-disk format — it's normative and takes precedence over convenience.
* **Backward compatibility is permanent.** A repository written by any released version must stay readable by every later version (`list`, `ls`, `check`, `cat`, `diff`, `restore`). There is no deprecation window for reads.
* **Forward compatibility isn't guaranteed, but failure must be safe.** An older build may be unable to read a newer repository, but it must never *misread* it as empty or valid — "cannot decode" must always surface as an error, never as "no entries," because that's exactly the condition that lets a garbage collector delete a live repository.
* **The version gate** (`core.RepoFormatVersion` / `core.MaxSupportedRepoFormat` in `internal/core/models.go`) gates every repository open. Raise it only when a change would make a repository unreadable or misreadable by earlier builds.
* **Changing the on-disk format** requires: keeping older layouts readable, upgrading only opportunistically (never requiring a migration to *read*), committing a fixture from the last release in the old format, deciding on the version gate, adding the baseline to `docs/compatibility.md`'s table, and stating in the PR what older builds do when they meet the new format — verified by running an old binary, not by reasoning about it. `e2e/feature_legacy_repo_test.go` enforces the fixture and doc-table requirements.
### Backup Flow
1. `BackupManager` acquires a shared lock, loads the previous snapshot (if any) for its source identity.
2. Source is scanned via `Walk()` (full) or `WalkChanges()` (incremental).
3. New/changed files are chunked using FastCDC, content-addressed, and uploaded.
4. The HAMT tree is updated with new filemeta refs. `TransactionalStore` buffers all intermediate HAMT nodes and only flushes reachable ones from the final root.
5. A new `Snapshot` object is written, and `index/latest` is updated.
### Encryption Model
* On `init`, a random 32-byte master key is generated and wrapped into key slots (password-based via scrypt, platform key, KMS-wrapped platform key, or BIP39 recovery key).
* Key slots are stored under `keys/` prefix, which the `EncryptedStore` passes through unencrypted.
* An HMAC dedup key is derived from the encryption key via HKDF for content-addressing without exposing plaintext hashes.
## Naming Conventions
**Commits & PR titles** use a Conventional Commit prefix — `type: imperative summary`, or `type(scope): …`. Lowercase the summary after the colon, no trailing period, keep it short (\~72 chars) and specific about what changed. PRs are squash-merged, so **the PR title becomes the commit subject** — give both the same form.
* Types: `feat`, `fix`, `refactor`, `test`, `docs`, `chore`, `perf`, `ci`.
* An optional scope names the area: `feat(tui):`, `fix(completion):`.
**Branch names** are `/` (e.g. `feat/tui-profile-history`), matching the commit type.
**Issue titles** carry no conventional-commit prefix — the type lives in the label. Lead with an imperative verb (`Add …`, `Convert …`) or an `Area:` scanning prefix, no trailing period.
**RFCs**: a substantial feature gets a design record under `rfcs/NNNN-kebab-slug.md` (zero-padded, next free number). The proposal PR/commit is `rfc: (RFC NNNN)`; the implementation PR/commit is a standard `type:` prefix with a trailing `(RFC NNNN)` reference, e.g. `feat: unified source identity (RFC 0009)`. See `rfcs/README.md` for the index and RFC 0010 as a template.
## Pull Request Guidelines
### Before Submitting
1. **Run tests**: `./scripts/check.sh`
2. **Update documentation**: Add/update user guide, README, and code comments
3. **Add tests**: Cover new functionality with unit/golden-file/testscript tests as appropriate
4. **Format code**: `go fmt ./...`
5. **Lint code**: `golangci-lint run ./...`
### PR Description
Two sections:
```markdown theme={null}
## Summary
- bullet list of the concrete changes, imperative voice
Closes #NNN
## Verification
-
-
```
Keep the `Summary` bullets high-signal — what changed and why, not a file-by-file diff. Under `Verification`, paste the exact commands you ran, e.g.:
```bash theme={null}
go test -count=1 ./cmd/cloudstic ./internal/engine
golangci-lint run ./cmd/cloudstic ./internal/engine
```
### Creating an Issue
Issues use four sections — `## Context` (current state, with backtick paths to concrete files/functions), `## Goal` (desired end state, one or two sentences), `## Scope` (bullet list of concrete changes), and `## Acceptance Criteria` (verifiable outcomes, always ending with the exact test/lint commands that must pass). Apply exactly one type label (`bug`, `enhancement`, `refactor`, `tech debt`, `chore`, `test`, `documentation`, `rfc`, `tracking`) plus one or more `area/*` labels.
## Getting Help
If you have questions or need help:
* Check `AGENTS.md` for architecture details
* Review existing code for patterns
* Open a GitHub issue for discussion
* Join our community chat (if available)
## License
By contributing, you agree that your contributions will be licensed under the same license as the project.
# HAMT Structure
Source: https://docs.cloudstic.com/advanced/hamt-structure
Hash Array Mapped Trie implementation for scalable directory indexing
## Overview
Cloudstic uses a **Merkle Hash Array Mapped Trie (HAMT)** to map file IDs to their metadata references. This data structure provides:
* **Efficient lookups**: O(log₃₂ n) depth due to 32-way branching
* **Structural sharing**: unchanged subtrees are reused by reference between snapshots
* **Persistent immutability**: all mutations return a new root while preserving the old one
* **Content-addressable nodes**: nodes are identified by the hash of their content
## Configuration
From `internal/hamt/hamt.go`:
```go theme={null}
const (
bitsPerLevel = 5 // 5 bits per level → 32-way branching
branching = 32 // 2^bitsPerLevel
maxDepth = 6 // Maximum tree depth
maxLeafSize = 32 // Maximum entries per leaf node
)
```
### Design Rationale
* **5 bits per level** provides 32-way branching, balancing tree depth with node size
* **Max depth of 6** supports 32^6 = 1,073,741,824 entries before hitting depth limit
* **Max leaf size of 32** keeps leaf nodes small enough to serialize efficiently
## Node Types
### Internal Node
Internal nodes use a **bitmap** to compactly represent which of the 32 child slots are populated:
```json theme={null}
{
"type": "internal",
"bitmap": 2348810305,
"children": ["node/", "node/"]
}
```
```go theme={null}
// From internal/core/models.go
type HAMTNode struct {
Type ObjectType `json:"type"` // "internal" or "leaf"
Bitmap uint32 `json:"bitmap,omitempty"`
Children []string `json:"children,omitempty"` // ["node/", ...]
Entries []LeafEntry `json:"entries,omitempty"`
}
```
**Bitmap encoding:**
* Each bit position (0-31) corresponds to a child slot
* A set bit indicates that slot is populated
* The `children` array contains only the populated slots, in order
* Child position is computed via `popcount(bitmap & (bit - 1))`
The bitmap compression reduces storage: a node with only 2 children stores 2 refs instead of a 32-element array.
### Leaf Node
Leaf nodes store actual key-value entries:
```json theme={null}
{
"type": "leaf",
"entries": [
{ "key": "", "filemeta": "filemeta/" },
{ "key": "", "filemeta": "filemeta/" }
]
}
```
```go theme={null}
// From internal/core/models.go
type LeafEntry struct {
Key string `json:"key"` // FileID
FileMeta string `json:"filemeta"` // "filemeta/"
}
```
Entries are **sorted by key** for deterministic hashing. Two leaves with the same entries in different orders produce the same hash.
## Path Key Computation
File IDs are hashed to produce a "path key" that determines the traversal path through the tree:
```go theme={null}
// From internal/hamt/hamt.go
func computePathKey(id string) string {
return core.ComputeHash([]byte(id))
}
func indexForLevel(keyHex string, level int) (int, error) {
// Parse first 8 hex chars (32 bits) of the hash
val, err := strconv.ParseUint(keyHex[:8], 16, 32)
if err != nil {
return 0, err
}
// Extract 5 bits for this level
shift := 32 - (level+1)*bitsPerLevel
mask := uint64((1 << bitsPerLevel) - 1)
return int((val >> shift) & mask), nil
}
```
**Example traversal:**
1. Hash the file ID → `a7f3c92e...`
2. Extract bits 27-31 for level 0 → index 20
3. Extract bits 22-26 for level 1 → index 15
4. Continue until reaching a leaf or empty slot
## Affinity Model (Locality-Preserving Keys)
Implemented in [PR #61](https://github.com/cloudstic/cloudstic-cli/pull/61) as HAMTv2.
By default, `SHA-256(fileID)` produces uniformly distributed keys, which means files sharing the same parent directory scatter across all 32 top-level trie buckets. On an incremental backup of a directory with `N` changed files, this causes `O(N · depth)` intermediate node rewrites.
The **affinity model** biases the routing key so that siblings share a common trie subtree:
```
AffinityKey(parentID, fileID) = SHA256(parentID)[:4] + SHA256(fileID)[4:]
```
* The first **4 hex chars** (16 bits) come from the parent directory's hash, pinning all siblings to the same top-3 trie levels.
* The remaining **28 hex chars** come from the file's own hash, uniquely distributing siblings within that subtree.
The total key length remains 32 hex characters. The routing machinery is unchanged.
### Locality Guarantee
With the affinity model, a backup of a flat directory with `N` changed files rewrites `O(maxDepth)` nodes instead of `O(N · maxDepth)`. Siblings converge to a single subtree root.
### Snapshot Version
New snapshots are tagged with `hamt_version: 2`. Older snapshots (no field or `hamt_version: 1`) continue to use plain `SHA-256` keys and remain fully readable.
### File Moves
When a file moves to a new parent, its affinity key changes. The engine issues an explicit `Delete(oldKey)` + `Insert(newKey)` pair. Sources that emit move events (e.g. Google Drive change tokens) handle this naturally.
## Operations
The `Tree` type exposes a functional API where all mutations return a new root:
```go theme={null}
// From internal/hamt/hamt.go
type Tree struct {
store store.ObjectStore
}
func (t *Tree) Insert(root, key, value string) (string, error)
func (t *Tree) Lookup(root, key string) (string, error)
func (t *Tree) Delete(root, key string) (string, error)
func (t *Tree) Walk(root string, fn func(key, value string) error) error
func (t *Tree) Diff(root1, root2 string, fn func(DiffEntry) error) error
func (t *Tree) NodeRefs(root string, fn func(ref string) error) error
```
### Insert Algorithm
1. **Lookup existing node** at the current level
2. If node is a **leaf**:
* If entry exists, update it in place (copy-on-write)
* If leaf has room (\< 32 entries), append the new entry
* If leaf is full and we're not at max depth, **split** into an internal node
3. If node is **internal**:
* Compute the child index for this level
* Recursively insert into the appropriate child
* Update the parent's child reference
4. Save the modified node and return its new reference
```go theme={null}
// From internal/hamt/hamt.go:insertIntoInternal
bit := uint32(1 << idx)
exists := node.Bitmap&bit != 0
childPos := popcount(node.Bitmap & (bit - 1))
if !exists {
newNode.Bitmap |= bit // Set the bit
// Insert new child at computed position
} else {
// Update existing child at position
}
```
### Delete Algorithm
1. Traverse to the leaf containing the key
2. Remove the entry from the leaf
3. If leaf becomes empty, return an empty ref
4. Propagate changes up the tree:
* If a child becomes empty, remove its bit from the parent's bitmap
* If an internal node is left with only one child that is a leaf, **collapse** it by promoting the leaf
### Structural Sharing
Only nodes along the path of a modified entry change:
```
root_old root_new
/ | \ / | \
A B C A B' C ← only B' is new
| |
... (modified)
```
Nodes A and C are reused by reference. Only B and its ancestors up to the root are copied.
## Diff Algorithm
The `Diff` operation performs a parallel traversal of two HAMT roots:
```go theme={null}
// From internal/hamt/hamt.go
type DiffEntry struct {
Key string
OldValue string // Empty for additions
NewValue string // Empty for deletions
}
```
**Algorithm:**
1. If both nodes are leaves, compare entries directly
2. If nodes are internal, iterate through all 32 buckets:
* If bucket exists in only one tree, collect all entries as added/removed
* If bucket exists in both, recursively diff the child nodes
3. Yield a `DiffEntry` for each difference found
The diff operation is **structural**. If two subtrees have the same root hash, they are identical and can be skipped entirely.
## TransactionalStore
During a backup, thousands of HAMT nodes are created as the tree is built incrementally. Many of these intermediate nodes become unreachable once the tree is fully constructed.
The `TransactionalStore` buffers new nodes in memory and only flushes the **reachable subset** from the final root:
```go theme={null}
// From internal/hamt/ (conceptual interface)
type TransactionalStore interface {
Put(key string, data []byte) error // Buffer in memory
Get(key string) ([]byte, error) // Check buffer, then backing store
Flush(root string) error // BFS from root, flush reachable nodes
}
```
**Flush algorithm:**
1. Start a BFS from the root node
2. For each node visited, check if it's in the in-memory buffer
3. If buffered, write it to the persistent store
4. Add all child refs to the BFS queue
5. Discard any buffered nodes that were never visited
This optimization avoids uploading superseded nodes that were created during tree construction but are no longer reachable.
## Garbage Collection
The `NodeRefs` method walks the entire tree and yields every node reference:
```go theme={null}
// From internal/hamt/hamt.go
func (t *Tree) NodeRefs(root string, fn func(ref string) error) error {
if root == "" {
return nil
}
return t.nodeRefs(root, fn)
}
func (t *Tree) nodeRefs(ref string, fn func(string) error) error {
if err := fn(ref); err != nil {
return err
}
node, err := t.loadNode(ref)
if err != nil {
return err
}
if node.Type == core.ObjectTypeInternal {
for _, childRef := range node.Children {
if err := t.nodeRefs(childRef, fn); err != nil {
return err
}
}
}
return nil
}
```
This is used during the **mark phase** of garbage collection to identify all reachable HAMT nodes from a snapshot root.
## Performance Characteristics
| Operation | Complexity | Notes |
| --------- | ---------- | ------------------------------- |
| Insert | O(log₃₂ n) | Maximum 6 levels for 1B entries |
| Lookup | O(log₃₂ n) | Typically 2-3 network requests |
| Delete | O(log₃₂ n) | Includes potential collapse |
| Walk | O(n) | Visits every leaf entry once |
| Diff | O(m) | Where m = changed entries |
With packfiles enabled, multiple HAMT nodes are bundled into 8MB packs. An LRU cache reduces the average lookup to 0-1 network requests after the initial pack fetch.
# Repository Locking
Source: https://docs.cloudstic.com/advanced/locking
How Cloudstic coordinates concurrent access using distributed locks stored in the repository
Cloudstic uses a distributed lock protocol stored directly inside the repository (under `index/`) to prevent concurrent writes from corrupting data. This page explains the lock types, which operations hold them, the TTL and refresh mechanism, and how to recover from a stuck lock.
## Lock Types
There are two lock types, implementing a standard reader-writer protocol:
| Type | Storage key | Rules |
| ------------- | ------------------------------- | ---------------------------------------------------------------------- |
| **Shared** | `index/lock.shared/` | Multiple shared locks can coexist. Acquired by `backup` and `restore`. |
| **Exclusive** | `index/lock.exclusive` | Only one at a time. Blocks all shared locks. Acquired by `prune`. |
## Which Operation Holds Which Lock
| Command | Lock type | Acquired | Released |
| --------- | --------- | ------------------------------------------- | --------------- |
| `backup` | Shared | Start of run (skipped for `-dry-run`) | On process exit |
| `restore` | Shared | Start of run (always, including `-dry-run`) | On process exit |
| `prune` | Exclusive | Start of run (skipped for `-dry-run`) | On process exit |
| `forget` | None | N/A | N/A |
| `check` | None | N/A | N/A |
## Lock Payload
Each lock is a JSON object written to the repository store:
```json theme={null}
{
"operation": "backup",
"holder": "my-hostname (pid 12345)",
"acquired_at": "2026-03-07T09:00:00.000000000Z",
"expires_at": "2026-03-07T09:01:00.000000000Z",
"is_shared": true
}
```
`holder` is `" (pid )"` of the process that acquired the lock.
## TTL and Automatic Refresh
Locks are designed to be short-lived so a crashed process never blocks access for long:
* **TTL:** 1 minute from acquisition.
* **Refresh:** While the process is alive, a background goroutine rewrites the lock every 30 seconds, extending `expires_at` by another minute.
* **Crash recovery:** If the process is killed, the refresh goroutine stops. The lock expires after at most 1 minute. The next operation sees the stale `expires_at` and proceeds automatically: no manual intervention required.
* **Refresh failure:** If the backing store becomes unreachable, the goroutine gives up after 3 consecutive failures and lets the TTL expire naturally.
## Conflict Rules
| Trying to acquire → | Shared (backup/restore) | Exclusive (prune) |
| ------------------------- | ----------------------- | ----------------- |
| **Shared lock active** | ✅ Allowed | ❌ Blocked |
| **Exclusive lock active** | ❌ Blocked | ❌ Blocked |
When blocked, the CLI exits immediately with an error. It does **not** wait for the lock to be released:
```
repository is exclusively locked by my-hostname (pid 12345) (operation: prune, acquired: ..., expires: ...)
```
## TOCTOU Mitigation
Object stores like S3 and B2 don't support atomic conditional writes. To reduce the risk of two processes claiming a lock simultaneously:
* **Exclusive lock:** After writing `index/lock.exclusive`, the engine immediately re-reads it and verifies `holder + acquired_at` still match. If another process won the race, the acquire fails.
* **Shared lock:** After writing `index/lock.shared/`, the engine re-checks `index/lock.exclusive`. If an exclusive lock appeared concurrently, the shared lock entry is deleted and the acquire fails.
This mitigation reduces but does not eliminate races on eventually-consistent stores. In practice, the acquire-then-verify pattern makes collisions vanishingly rare.
## Stale Lock Recovery
A lock is stale when its `expires_at` is in the past. Stale locks are ignored automatically. No manual action is needed.
If you cannot wait for the 1-minute TTL (e.g. you need to unblock a deployment immediately), use `break-lock`:
```bash theme={null}
cloudstic break-lock
```
This unconditionally deletes `index/lock.exclusive` and all `index/lock.shared/*` entries, regardless of TTL or holder. See the [`break-lock` reference](/commands/break-lock) for full usage.
Only run `break-lock` when you are certain no `backup`, `restore`, or `prune` process is actively running. Removing a lock held by an active process can corrupt the repository.
## Concurrency Semantics
Because `backup` and `restore` use shared locks, they can run concurrently against the same repository without conflict. Each backup run writes its own snapshot independently. `prune` requires an exclusive lock and must wait for all active backups to finish (or fail fast if they're running).
This means:
* **Two simultaneous backups:** ✅ Both succeed; each creates its own snapshot.
* **Backup + restore simultaneously:** ✅ Both succeed.
* **Backup while prune is running:** ❌ Backup fails immediately with a lock error.
* **Prune while backup is running:** ❌ Prune fails immediately with a lock error.
* **Two simultaneous prunes:** ❌ Second prune fails immediately.
## See Also
* [`break-lock` command](/commands/break-lock): Force-remove stale locks
* [`prune` command](/commands/prune): Exclusive lock holder
* [`backup` command](/commands/backup): Shared lock holder
# Object Type Specifications
Source: https://docs.cloudstic.com/advanced/object-types
Detailed format specifications for all object types in the Cloudstic storage model
## Overview
This page documents the exact structure of each object type in the Cloudstic storage model. All objects are stored as JSON (except chunks, which are raw binary) and keyed by their hash.
## Chunk
**Object key:** `chunk/` or `chunk/`
**Format:** Raw binary (zstd-compressed bytes)
Chunks are the **only non-JSON objects** in the system. They contain raw file data compressed with zstd.
### Content-Defined Chunking
Chunks are produced by **FastCDC** (Fast Content-Defined Chunking):
| Parameter | Value |
| --------- | ------- |
| Min size | 512 KiB |
| Avg size | 1 MiB |
| Max size | 8 MiB |
The final chunk of a file may be smaller than the minimum.
### Hash Function
* **When encrypted:** `HMAC-SHA256(dedup_key, uncompressed_data)`
* The dedup key is derived from the encryption key via HKDF
* Prevents the storage provider from confirming file contents by hashing known plaintext
* **When unencrypted:** `SHA-256(uncompressed_data)`
The hash is computed on the **uncompressed** data, not the stored zstd-compressed bytes. This ensures consistent deduplication regardless of compression settings.
### Storage Format
```
Chunk object:
┌──────────────────────┐
│ zstd-compressed data │ ← Raw bytes, no JSON wrapper
└──────────────────────┘
```
Reading a chunk:
1. Fetch the object bytes
2. Decompress with zstd
3. Return the raw file data
## Content
**Object key:** `content/`
**Format:** JSON object
Content objects list the ordered chunks that make up a file's content.
```json theme={null}
{
"type": "content",
"size": 10485760,
"chunks": [
"chunk/a7f3c92e...",
"chunk/b4e1d83f...",
"chunk/c9a2e75d..."
]
}
```
### Go Struct Definition
From `internal/core/models.go`:
```go theme={null}
type Content struct {
Type ObjectType `json:"type"` // "content"
Size int64 `json:"size"`
Chunks []string `json:"chunks,omitempty"` // List of "chunk/"
DataInlineB64 []byte `json:"data_inline_b64,omitempty"` // For small files
}
```
### Inline Data Optimization
Very small files (\< 512 KiB) may use `data_inline_b64` instead of `chunks` to avoid creating a separate chunk object:
```json theme={null}
{
"type": "content",
"size": 142,
"data_inline_b64": "SGVsbG8sIHdvcmxkIQ=="
}
```
This reduces object count and API calls for small files.
## FileMeta
**Object key:** `filemeta/`
**Format:** JSON object
FileMeta objects contain immutable metadata about a file or folder.
```json theme={null}
{
"version": 1,
"fileId": "1a2b3c4d5e6f",
"name": "invoice.pdf",
"type": "file",
"parents": ["filemeta/e7f8a9b0..."],
"content_hash": "b4e1d83f9a2c...",
"content_ref": "c5f2e94g0b3d...",
"size": 21733,
"mtime": 1710000000,
"owner": "user@example.com",
"extra": {
"mimeType": "application/pdf",
"trashed": false
},
"mode": 33188,
"uid": 501,
"gid": 20,
"btime": 1710000000,
"flags": 0,
"xattrs": {
"user.tag": "cHJvamVjdA=="
}
}
```
### Go Struct Definition
From `internal/core/models.go`:
```go theme={null}
type FileMeta struct {
Version int `json:"version"`
FileID string `json:"fileId"` // HAMT key
Name string `json:"name"`
Type FileType `json:"type"` // "file" or "folder"
Parents []string `json:"parents"` // List of "filemeta/" refs
Paths []string `json:"paths,omitempty"`
ContentHash string `json:"content_hash"` // SHA256 of raw content
ContentRef string `json:"content_ref,omitempty"` // HMAC(dedupKey, ContentHash) for secure backend lookup
Size int64 `json:"size"`
Mtime int64 `json:"mtime"` // Unix timestamp
Owner string `json:"owner"`
Extra map[string]interface{} `json:"extra,omitempty"`
Mode uint32 `json:"mode,omitempty"` // POSIX permission bits
Uid uint32 `json:"uid,omitempty"` // POSIX user ID
Gid uint32 `json:"gid,omitempty"` // POSIX group ID
Btime int64 `json:"btime,omitempty"` // birth/creation time, Unix seconds
Flags uint32 `json:"flags,omitempty"` // per-file flags (chflags / FS_IOC_GETFLAGS)
Xattrs map[string][]byte `json:"xattrs,omitempty"` // extended attributes: name → raw bytes
}
func (f *FileMeta) Ref() (string, []byte, error) {
hash, data, err := ComputeJSONHash(f)
if err != nil {
return "", data, err
}
return "filemeta/" + hash, data, nil
}
```
### Field Descriptions
| Field | Description |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| `fileId` | Source-specific unique identifier (Google Drive ID, relative path) |
| `type` | `"file"` or `"folder"` |
| `parents` | List of `filemeta/` refs pointing to parent metadata objects |
| `content_hash` | SHA-256 of the raw file content |
| `content_ref` | HMAC of the content hash (used to key the Content object securely) |
| `paths` | Optional legacy compatibility field. New snapshots usually omit it and derive display paths from `parents` + `name`. |
| `extra` | Source-specific metadata (e.g. MIME type, trashed status) |
| `mode` | POSIX file mode bits (e.g. `0644` = `420`). Omitted if zero. |
| `uid` | Numeric owner user ID. Omitted if zero. |
| `gid` | Numeric owner group ID. Omitted if zero. |
| `btime` | File creation (birth) time as Unix epoch seconds. Omitted if zero. |
| `flags` | OS-specific file flags (macOS `UF_*`/`SF_*`, Linux `FS_*_FL`). Omitted if zero. |
| `xattrs` | Extended attributes as `name → base64(value)` map. Omitted if empty. |
**Important:** `fileId` is the **HAMT key** used to look up this file's metadata. It must be unique within a snapshot.
### Folder Representation
Folders are represented with:
* `type: "folder"`
* `content_hash: ""` (empty string)
* `content_ref: ""` (empty string)
* `size: 0`
* `chunks: []` in the content object (if created)
### Parent References
The `parents` field contains refs to **parent FileMeta objects**, not raw file IDs. This allows reconstructing the full directory path by walking the parent chain, which is now the primary restore/listing model for new snapshots.
## HAMT Node
**Object key:** `node/`
**Format:** JSON object
See [HAMT Structure](/advanced/hamt-structure) for detailed documentation.
### Internal Node
```json theme={null}
{
"type": "internal",
"bitmap": 2348810305,
"children": [
"node/a7f3c92e...",
"node/b4e1d83f..."
]
}
```
### Leaf Node
```json theme={null}
{
"type": "leaf",
"entries": [
{
"key": "1a2b3c4d5e6f",
"filemeta": "filemeta/e7f8a9b0..."
},
{
"key": "2b3c4d5e6f7g",
"filemeta": "filemeta/f8a9b0c1..."
}
]
}
```
### Go Struct Definition
From `internal/core/models.go`:
```go theme={null}
type HAMTNode struct {
Type ObjectType `json:"type"` // "internal" or "leaf"
Bitmap uint32 `json:"bitmap,omitempty"`
Children []string `json:"children,omitempty"` // ["node/", ...]
Entries []LeafEntry `json:"entries,omitempty"`
}
type LeafEntry struct {
Key string `json:"key"` // FileID
FileMeta string `json:"filemeta"` // "filemeta/"
}
```
## Snapshot
**Object key:** `snapshot/`
**Format:** JSON object
Snapshots are point-in-time backup checkpoints referencing a HAMT root.
```json theme={null}
{
"version": 1,
"created": "2025-12-01T12:00:00Z",
"root": "node/a7f3c92e...",
"seq": 42,
"source": {
"type": "gdrive",
"account": "user@gmail.com",
"path": "my-drive://"
},
"meta": {
"generator": "cloudstic-cli",
"hostname": "workstation-01"
},
"tags": ["daily", "important"],
"change_token": "12345",
"exclude_hash": "d4c3b2a1..."
}
```
### Go Struct Definition
From `internal/core/models.go`:
```go theme={null}
type Snapshot struct {
Version int `json:"version"`
Created string `json:"created"` // ISO8601
Root string `json:"root"` // "node/"
Seq int `json:"seq"`
Source *SourceInfo `json:"source,omitempty"`
Meta map[string]string `json:"meta,omitempty"`
Tags []string `json:"tags,omitempty"`
ChangeToken string `json:"change_token,omitempty"`
ExcludeHash string `json:"exclude_hash,omitempty"`
}
type SourceInfo struct {
Type string `json:"type"` // e.g. "gdrive", "local"
Account string `json:"account,omitempty"` // friendly display account
Path string `json:"path,omitempty"` // friendly display path
Identity string `json:"identity,omitempty"` // stable container identity
PathID string `json:"path_id,omitempty"` // stable selected-root identity
DriveName string `json:"drive_name,omitempty"` // friendly container label
// Legacy compatibility fields (older snapshots)
VolumeUUID string `json:"volume_uuid,omitempty"`
VolumeLabel string `json:"volume_label,omitempty"`
}
```
### Field Descriptions
| Field | Description |
| -------------- | ------------------------------------------------------------------------------------ |
| `seq` | Monotonically increasing sequence number |
| `source` | Origin and lineage identity of the backup (type, identity, path\_id, display labels) |
| `meta` | Free-form key-value metadata (generator, hostname, etc.) |
| `tags` | User-defined labels for retention policies |
| `change_token` | Opaque token for incremental sources (omitted when not applicable) |
| `exclude_hash` | Hash of the exclude patterns used for this snapshot |
Every snapshot is a **complete checkpoint**, with no delta replay needed. Structural sharing via the HAMT minimizes the number of new nodes.
### Change Tokens
Incremental sources (`gdrive-changes`, `onedrive-changes`) record an opaque `change_token` in each snapshot. On the next backup:
1. Read the token from the previous snapshot
2. Pass it to the source to get only changed files since that token
3. Save the new token in the new snapshot
If no previous token exists (first backup or after switching from a full-scan source), the source performs a full scan and saves the initial token.
## Index Objects
### index/latest
**Object key:** `index/latest`
**Format:** JSON object
A mutable pointer to the most recent snapshot.
```json theme={null}
{
"latest_snapshot": "snapshot/a7f3c92e...",
"seq": 42
}
```
```go theme={null}
// From internal/core/models.go
type Index struct {
LatestSnapshot string `json:"latest_snapshot"` // "snapshot/"
Seq int `json:"seq"`
}
```
### index/snapshots
**Object key:** `index/snapshots`
**Format:** JSON array
A catalog of lightweight snapshot summaries, used to avoid fetching each full snapshot object:
```json theme={null}
[
{
"ref": "snapshot/a7f3c92e...",
"seq": 42,
"created": "2025-12-01T12:00:00Z",
"root": "node/e7f8a9b0...",
"source": {
"type": "gdrive",
"account": "user@gmail.com",
"path": "my-drive://"
},
"tags": ["daily"],
"change_token": "12345"
}
]
```
```go theme={null}
// From internal/core/models.go
type SnapshotSummary struct {
Ref string `json:"ref"` // "snapshot/"
Seq int `json:"seq"`
Created string `json:"created"` // ISO8601
Root string `json:"root"` // "node/"
Source *SourceInfo `json:"source,omitempty"`
Tags []string `json:"tags,omitempty"`
ChangeToken string `json:"change_token,omitempty"`
ExcludeHash string `json:"exclude_hash,omitempty"`
}
```
The catalog self-heals via reconciliation with `LIST snapshot/` on load. If the catalog is missing or stale, it's rebuilt automatically.
### index/packs
**Object key:** `index/packs`
**Format:** bbolt database
When packfiles are enabled, the pack catalog is a bbolt key-value database mapping logical object keys to their location within packfiles:
```
Logical key: "filemeta/a7f3c92e..."
↓
Pack entry: { PackID: "pack/b4e1d83f...", Offset: 1024, Length: 256 }
```
The catalog is stored as a single object in the store and loaded into memory on startup.
## Encryption Key Slots
**Object key:** `keys/-