> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cloudstic.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Go Client API
> Use Cloudstic programmatically in your Go applications
The `github.com/cloudstic/cli` package exposes a high-level Go API that mirrors the CLI commands. Use it to embed backup, restore, and repository management directly into your Go programs.
## Installation
```bash theme={null}
go get github.com/cloudstic/cli
```
Import the root package:
```go theme={null}
import cloudstic "github.com/cloudstic/cli"
```
Storage backends and keychain helpers live in sub-packages:
```go theme={null}
import (
"github.com/cloudstic/cli/pkg/store"
"github.com/cloudstic/cli/pkg/keychain"
"github.com/cloudstic/cli/pkg/source"
)
```
Two further packages turn user-facing configuration into live objects, and the
split between them decides what importing costs you:
```go theme={null}
import (
"github.com/cloudstic/cli/pkg/config" // what did the user configure
"github.com/cloudstic/cli/pkg/open" // connect to it
)
```
`pkg/config` holds resolved configuration values and performs no I/O against a
store or a provider, so reading and validating configuration pulls in no cloud
SDK. `pkg/open` constructs stores, sources, keychains and clients from those
values, and necessarily links whichever provider SDK you use. If you only need to
inspect configuration, import the first and not the second.
## Quick Start
```go theme={null}
package main
import (
"bytes"
"context"
"fmt"
"log"
cloudstic "github.com/cloudstic/cli"
"github.com/cloudstic/cli/pkg/keychain"
"github.com/cloudstic/cli/pkg/source"
"github.com/cloudstic/cli/pkg/store"
)
func main() {
ctx := context.Background()
// 1. Open a storage backend
rawStore, err := store.NewLocalStore("./my-repo")
if err != nil {
log.Fatal(err)
}
// 2. Build a keychain with your credentials
kc := keychain.Chain{keychain.WithPassword("my-passphrase")}
// 3. Initialize the repository (first time only)
_, err = cloudstic.InitRepo(ctx, rawStore,
cloudstic.WithInitCredentials(kc),
)
if err != nil {
log.Fatal(err)
}
// 4. Create a client: auto-resolves encryption via the keychain
client, err := cloudstic.NewClient(ctx, rawStore, cloudstic.WithKeychain(kc))
if err != nil {
log.Fatal(err)
}
// 5. Back up a local directory
src := source.NewLocalSource("./documents")
result, err := client.Backup(ctx, src)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Snapshot: %s (%d new files)\n", result.SnapshotHash, result.FilesNew)
// 6. Restore to a ZIP archive
var buf bytes.Buffer
_, err = client.Restore(ctx, &buf, "latest")
if err != nil {
log.Fatal(err)
}
fmt.Printf("Restored %d bytes\n", buf.Len())
}
```
## Storage Backends
All backends implement `store.ObjectStore`. Pass the raw store to `InitRepo` and `NewClient`.
### Local
```go theme={null}
s, err := store.NewLocalStore("/path/to/repo")
```
### Amazon S3 (and S3-compatible)
```go theme={null}
s, err := store.NewS3Store(ctx, "my-bucket",
store.WithS3Region("us-east-1"),
// For MinIO, Cloudflare R2, Wasabi, etc.:
store.WithS3Endpoint("https://minio.example.com"),
// Explicit credentials (or use the AWS SDK default credential chain):
store.WithS3Credentials(accessKeyID, secretAccessKey),
store.WithS3Prefix("backups/"),
)
```
### Backblaze B2
```go theme={null}
s, err := store.NewB2Store("my-bucket",
store.WithCredentials(keyID, appKey),
store.WithPrefix("backups/"),
)
```
### SFTP
```go theme={null}
s, err := store.NewSFTPStore("backup.example.com",
store.WithSFTPBasePath("/home/backup/repo"),
store.WithSFTPUser("backupuser"),
store.WithSFTPKey("~/.ssh/id_ed25519"),
store.WithSFTPPort("22"),
)
```
## Keychain
The keychain resolves credentials to a master key. Build a `keychain.Chain` before calling `InitRepo` or `NewClient`.
```go theme={null}
// Password-based
kc := keychain.Chain{keychain.WithPassword("my-passphrase")}
// Platform key (raw 32-byte key)
platformKey, _ := hex.DecodeString("64-hex-chars...")
kc := keychain.Chain{keychain.WithPlatformKey(platformKey)}
// Recovery key (24-word BIP39 mnemonic)
kc := keychain.Chain{keychain.WithRecoveryKey("word1 word2 ... word24")}
// Multiple credentials (tried in order until one succeeds)
kc := keychain.Chain{
keychain.WithPassword("my-passphrase"),
keychain.WithRecoveryKey("word1 word2 ... word24"),
}
// AWS KMS
kmsClient, _ := crypto.NewAWSKMSClient(ctx, "arn:aws:kms:us-east-1:123456789:key/abc-123")
kc := keychain.Chain{keychain.WithKMSClient(kmsClient)}
```
## Initializing a Repository
`InitRepo` is a package-level function that runs on the raw (unencrypted) store. It only needs to be called once per repository.
```go theme={null}
func InitRepo(ctx context.Context, rawStore store.ObjectStore, opts ...InitOption) (*InitResult, error)
```
### Init Options
| Option | Description |
| ---------------------------------------- | ----------------------------------------------------------------------- |
| `WithInitCredentials(kc keychain.Chain)` | Key slots to create (password, platform key, KMS) |
| `WithInitRecovery()` | Also generate a 24-word recovery key slot |
| `WithInitNoEncryption()` | Create an unencrypted repository |
| `WithInitAdoptSlots()` | Adopt existing slots if already initialized (prevents error on re-init) |
### InitResult
```go theme={null}
type InitResult struct {
Encrypted bool // whether the repository uses encryption
RecoveryKey string // 24-word mnemonic (empty if WithInitRecovery() was not requested)
AdoptedSlots bool // true if existing slots were adopted
}
```
### Example
```go theme={null}
result, err := cloudstic.InitRepo(ctx, rawStore,
cloudstic.WithInitCredentials(kc),
cloudstic.WithInitRecovery(),
)
if err != nil {
log.Fatal(err)
}
if result.RecoveryKey != "" {
fmt.Println("Recovery key:", result.RecoveryKey)
// Store this securely: it is displayed only once!
}
```
## Inspecting a Repository
These package-level functions read repository state from the raw store without requiring the encryption key — useful for deciding whether to prompt for credentials before calling `NewClient`.
```go theme={null}
func InspectRepo(ctx context.Context, rawStore store.ObjectStore) (RepoStatus, error)
```
```go theme={null}
type RepoStatus struct {
Initialized bool // whether a config marker exists at all
Encrypted bool // whether the repository uses encryption
Sealed bool // whether the marker itself is sealed (only true for newer encrypted repos)
}
```
```go theme={null}
status, err := cloudstic.InspectRepo(ctx, rawStore)
if err != nil {
log.Fatal(err)
}
if !status.Initialized {
fmt.Println("Run InitRepo first.")
} else if status.Encrypted {
fmt.Println("Repository is encrypted; resolve a keychain before NewClient.")
}
```
`LoadRepoConfig(ctx, rawStore, encryptionKey) (*RepoConfig, error)` reads the full marker (not just its status) and requires the encryption key for a sealed repository. `UpgradeRepoFormat` stamps the on-disk format version; `NewClient`/`Backup`/`Prune`/`Forget` call it for you as part of normal operation, so most callers never need it directly.
## Creating a Client
```go theme={null}
func NewClient(ctx context.Context, base store.ObjectStore, opts ...ClientOption) (*Client, error)
```
`NewClient` reads the repository config, resolves the master key via the keychain, and builds the encryption/compression/packfile decorator chain internally.
### Client Options
| Option | Description |
| --------------------------------- | -------------------------------------------------------------- |
| `WithKeychain(kc keychain.Chain)` | Keychain for automatic master key resolution |
| `WithEncryptionKey(key []byte)` | Direct 32-byte AES key. Bypasses keychain and config detection |
| `WithReporter(r Reporter)` | Progress reporter for UI feedback |
| `WithPackfile(enable bool)` | Bundle small objects into 8MB packs (default: `true`) |
```go theme={null}
client, err := cloudstic.NewClient(ctx, rawStore,
cloudstic.WithKeychain(kc),
cloudstic.WithReporter(myReporter),
cloudstic.WithPackfile(true),
)
```
`WithEncryptionKey` is intended for SaaS scenarios where the master key is resolved externally. For typical use, prefer `WithKeychain`.
`client.Store()` returns the underlying `store.ObjectStore` (the fully decorated store `NewClient` built), for callers that need lower-level access alongside the `Client` API.
## Backup
```go theme={null}
func (c *Client) Backup(ctx context.Context, src source.Source, opts ...BackupOption) (*BackupResult, error)
```
Creates a new backup snapshot from the given source.
### Backup Options
| Option | Description |
| ------------------------------ | --------------------------------------------------------------------- |
| `WithVerbose()` | Log per-file operations |
| `WithBackupDryRun()` | Scan without writing to the repository |
| `WithIgnoreEmptySnapshot()` | Skip writing a snapshot if nothing changed since the previous one |
| `WithTags(tags ...string)` | Apply tags to the snapshot |
| `WithGenerator(name string)` | Record the tool/script that produced the backup (e.g. for automation) |
| `WithMeta(key, value string)` | Attach an arbitrary key/value pair to the snapshot |
| `WithExcludeHash(hash string)` | Record the exclude pattern fingerprint in the snapshot |
### BackupResult
```go theme={null}
type BackupResult struct {
SnapshotRef string // "snapshot/"
SnapshotHash string // bare content hash
Root string // HAMT root ref
FilesNew int64
FilesChanged int64
FilesUnmodified int64
FilesRemoved int64
DirsNew int64
DirsChanged int64
DirsUnmodified int64
DirsRemoved int64
BytesAddedRaw int64 // uncompressed bytes written
BytesAddedStored int64 // bytes written to the store (post-compression and encryption)
Duration time.Duration
DryRun bool
EmptySnapshotIgnored bool // true if WithIgnoreEmptySnapshot() skipped an unchanged backup
}
```
### Example
```go theme={null}
src := source.NewLocalSource("./documents",
source.WithLocalExcludePatterns([]string{"*.log", "node_modules/"}),
)
result, err := client.Backup(ctx, src,
cloudstic.WithTags("production", "weekly"),
)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Snapshot %s: %d new, %d changed files\n",
result.SnapshotHash, result.FilesNew, result.FilesChanged)
```
### Sources
Sources implement `source.Source`. Available implementations:
```go theme={null}
// Local filesystem
src := source.NewLocalSource("/path/to/dir",
source.WithLocalExcludePatterns([]string{"*.tmp"}),
)
// SFTP remote directory
src, err := source.NewSFTPSource("backup.example.com",
source.WithSFTPSourceBasePath("/remote/path"),
source.WithSFTPSourceUser("user"),
source.WithSFTPSourceKey("~/.ssh/id_rsa"),
)
// Google Drive (full scan)
src, err := source.NewGDriveSource(ctx,
source.WithTokenPath("/path/to/google_token.json"),
source.WithDriveID("sharedDriveID"), // omit for My Drive
source.WithRootFolderID("folderID"), // omit for entire drive
)
// Google Drive (incremental via Changes API: recommended)
src, err := source.NewGDriveChangeSource(ctx,
source.WithTokenPath("/path/to/google_token.json"),
)
// OneDrive (full scan)
src, err := source.NewOneDriveSource(ctx,
source.WithOneDriveTokenPath("/path/to/onedrive_token.json"),
)
// OneDrive (incremental via Delta API: recommended)
src, err := source.NewOneDriveChangeSource(ctx,
source.WithOneDriveTokenPath("/path/to/onedrive_token.json"),
)
```
## Restore
```go theme={null}
func (c *Client) Restore(ctx context.Context, w io.Writer, snapshotRef string, opts ...RestoreOption) (*RestoreResult, error)
```
Writes the snapshot's file tree as a ZIP archive to `w`. Pass `io.Discard` for a dry run.
`snapshotRef` accepts `""`, `"latest"`, a bare hash or unique hash prefix, or
`"snapshot/"`.
### Restore Options
| Option | Description |
| ------------------------------ | ------------------------------------------------------------ |
| `WithRestoreDryRun()` | Count files/bytes without writing ZIP data |
| `WithRestoreVerbose()` | Log per-file operations |
| `WithRestorePath(path string)` | Restore only the given file or subtree (e.g. `"Documents/"`) |
| `WithRestoreNoVerify()` | Skip content-hash verification of restored chunks |
### RestoreResult
```go theme={null}
type RestoreResult struct {
SnapshotRef string
Root string
FilesWritten int
DirsWritten int
BytesWritten int64
Errors int // number of non-fatal errors
Warnings int // number of non-fatal warnings (e.g. skipped attributes)
DryRun bool
}
```
### Example
```go theme={null}
f, err := os.Create("restore.zip")
if err != nil {
log.Fatal(err)
}
defer f.Close()
result, err := client.Restore(ctx, f, "latest",
cloudstic.WithRestorePath("Documents/"),
)
if err != nil {
os.Remove("restore.zip")
log.Fatal(err)
}
fmt.Printf("Restored %d files (%d bytes)\n", result.FilesWritten, result.BytesWritten)
```
### Restoring directly to a directory
```go theme={null}
func (c *Client) RestoreToDir(ctx context.Context, outputDir, snapshotRef string, opts ...RestoreOption) (*RestoreResult, error)
```
Writes files directly to `outputDir` instead of a ZIP archive. Takes the same `RestoreOption`s and returns the same `RestoreResult`.
```go theme={null}
result, err := client.RestoreToDir(ctx, "/restore/target", "latest")
if err != nil {
log.Fatal(err)
}
fmt.Printf("Restored %d files to disk\n", result.FilesWritten)
```
## List
```go theme={null}
func (c *Client) List(ctx context.Context, opts ...ListOption) (*ListResult, error)
```
Lists all snapshots in the repository. Returns them sorted oldest-first.
### List Options
| Option | Description |
| ------------------- | ------------------------------------------------- |
| `WithListVerbose()` | Log progress while the snapshot catalog is loaded |
### ListResult
```go theme={null}
type ListResult struct {
Snapshots []SnapshotEntry
}
type SnapshotEntry struct {
Ref string // "snapshot/"
Snap core.Snapshot // the decoded snapshot object (Seq, Source, Tags, Meta, ...)
Created time.Time
}
```
`SnapshotEntry.Snap` is a `core.Snapshot`, which carries `Seq int`, `Source *core.SourceInfo` (`Type`, `Account`, `Path`, ...), `Tags []string`, and `Meta map[string]string`. `SourceInfo` may be `nil` for snapshots written before source identity was tracked.
These types are declared in `github.com/cloudstic/cli/internal/core`, which external modules cannot import directly — Go's `internal/` rule blocks the import statement. You don't need to. The root `cloudstic` package re-exports each of them as a type alias, so you can name them explicitly:
```go theme={null}
var snap cloudstic.Snapshot // = core.Snapshot
var info *cloudstic.SourceInfo // = *core.SourceInfo
var meta cloudstic.FileMeta // = core.FileMeta
var entry cloudstic.SnapshotEntry // = engine.SnapshotEntry
```
Because a Go alias denotes the identical type, these are interchangeable with what the `Client` API returns — you can declare variables, write helper function signatures, and implement interfaces against them. Reading exported fields off returned values (`entry.Snap.Seq`, `meta.Name`) works without naming the types at all, as this example does.
### Example
```go theme={null}
result, err := client.List(ctx)
if err != nil {
log.Fatal(err)
}
for _, entry := range result.Snapshots {
source := "unknown"
if entry.Snap.Source != nil {
source = entry.Snap.Source.Type
}
fmt.Printf("[%d] %s %-10s %s\n",
entry.Snap.Seq, entry.Created.Format(time.DateTime), source, entry.Ref)
}
```
## LsSnapshot
```go theme={null}
func (c *Client) LsSnapshot(ctx context.Context, snapshotID string, opts ...LsSnapshotOption) (*LsSnapshotResult, error)
```
Loads all file metadata from a snapshot and returns the full directory tree.
Accepts `"latest"`, a bare hash or unique hash prefix, or
`"snapshot/"`.
### LsSnapshot Options
| Option | Description |
| ----------------- | ------------------------------------- |
| `WithLsVerbose()` | Log progress while the tree is loaded |
### LsSnapshotResult
```go theme={null}
type LsSnapshotResult struct {
Ref string
Snapshot core.Snapshot
RootRefs []string // top-level entry refs
RefToMeta map[string]core.FileMeta // ref → file metadata (name, size, type, mode, owner, ...)
ChildRefs map[string][]string // parent ref → ordered child refs
}
```
See the note above about `core.Snapshot`/`core.FileMeta` and the `internal/` import restriction.
## Find
```go theme={null}
func (c *Client) Find(ctx context.Context, opts ...FindOption) (*FindResult, error)
```
Locates files across every snapshot in the repository without you having to know which snapshot holds them. Unlike the other read operations, `Find` takes a snapshot as *output* rather than input: by default it searches every snapshot and reports, for each matching file, the versions it has had and which snapshots each version lives in. It is a pure read path — no lock is taken and nothing is written.
### Find Options
Entry predicates (all given predicates must match):
| Option | Description |
| ---------------------------------- | ------------------------------------------------------------------------------------------------ |
| `WithFindPattern(pattern string)` | Positional pattern; routed to `Name` or `Path` depending on whether it contains a path separator |
| `WithFindName(pattern string)` | Match by basename glob |
| `WithFindPath(pattern string)` | Match by full path glob |
| `WithFindRegex(expr string)` | Match by regular expression |
| `WithFindIgnoreCase()` | Case-insensitive matching for name/path/regex |
| `WithFindFileID(id string)` | Match a specific source file ID |
| `WithFindContentHash(hash string)` | Match files with this exact content hash |
| `WithFindRef(ref string)` | Match a specific `filemeta/` ref |
| `WithFindType(t core.FileType)` | Restrict to a file type (file, directory, ...) |
| `WithFindSize(cmp SizeCompare)` | Size predicate — see `ParseSizeCompare` below |
| `WithFindNewer(spec string)` | File `Mtime` at or after `spec` (RFC3339 or a duration like `"7d"`) |
| `WithFindOlder(spec string)` | File `Mtime` at or before `spec` |
Snapshot selectors (which snapshots are searched):
| Option | Description |
| ----------------------------------- | ----------------------------------------------------------------------------- |
| `WithFindSnapshots(refs ...string)` | Restrict to specific snapshots (`"latest"`, full hash, or unambiguous prefix) |
| `WithFindSource(uri string)` | Restrict to snapshots from a source URI |
| `WithFindTags(tags ...string)` | Restrict to snapshots carrying any of these tags |
| `WithFindLatest(n int)` | Restrict to the `n` newest selected snapshots |
| `WithFindSince(spec string)` | Only snapshots created at or after `spec` |
| `WithFindUntil(spec string)` | Only snapshots created at or before `spec` |
Presentation and execution:
| Option | Description |
| --------------------------- | ------------------------------------------------------------------------------------------------------ |
| `WithFindGroupByContent()` | Group matches by content hash instead of file identity (finds duplicate content) |
| `WithFindMaxResults(n int)` | Cap the number of distinct files reported (default 1000; scanning continues so counters stay accurate) |
| `WithFindNoDelta()` | Force a full per-snapshot walk instead of the delta scan between related snapshots |
| `WithFindVerbose()` | Log scan progress |
Two helpers parse the string forms the CLI accepts, for callers building `SizeCompare`/time values programmatically:
```go theme={null}
func ParseSizeCompare(spec string) (SizeCompare, error) // "+10M", "-10M", "10M"
func ParseFindTime(spec string) (time.Time, error) // RFC3339 or a duration like "7d"
```
### FindResult
```go theme={null}
type FindResult struct {
Query FindQuery
SnapshotsSearched int
EntriesScanned int
MetaFetched int // filemeta objects actually read
Matches []FileMatch
Truncated bool // true if MaxResults was reached
Warnings []string
GroupedBy string // "file" or "content"
Elapsed string
}
// FileMatch is one file (or, under WithFindGroupByContent, one distinct
// content hash), with every version the query matched.
type FileMatch struct {
FileID string // empty when grouped by content
ContentHash string // empty unless grouped by content
Source *core.SourceInfo
Type core.FileType
Versions []FileVersion // newest first
}
// FileVersion is one immutable state of a file.
type FileVersion struct {
Ref string
FileID string
Name string
Paths []string // more than one path if the same file lives under multiple parents
ContentHash string
Type core.FileType
Size int64
Mtime int64
Mode uint32
Snapshots []SnapshotRef // every snapshot holding this exact version
FirstSeen string // ISO8601, earliest containing snapshot
LastSeen string // ISO8601, latest containing snapshot
}
type SnapshotRef struct {
Ref string
Seq int
Created string // ISO8601
}
```
`FileMatch` has two convenience methods: `Path()` returns the newest version's first path, and `LatestSnapshot()` returns the newest snapshot holding the newest version — the one a follow-up `Restore` would target.
### Example
```go theme={null}
// Find every file over 100MB modified in the last 30 days, across all snapshots.
size, err := cloudstic.ParseSizeCompare("+100M")
if err != nil {
log.Fatal(err)
}
result, err := client.Find(ctx,
cloudstic.WithFindSize(size),
cloudstic.WithFindNewer("30d"),
)
if err != nil {
log.Fatal(err)
}
for _, m := range result.Matches {
v := m.Versions[0]
fmt.Printf("%s (%d bytes) — %d version(s), latest in %d snapshot(s)\n",
m.Path(), v.Size, len(m.Versions), len(v.Snapshots))
}
if result.Truncated {
fmt.Printf("... and more (raise WithFindMaxResults to see them)\n")
}
```
Finding duplicate content:
```go theme={null}
result, err := client.Find(ctx, cloudstic.WithFindGroupByContent())
if err != nil {
log.Fatal(err)
}
for _, m := range result.Matches {
if len(m.Versions) > 1 {
fmt.Printf("%s duplicated across %d files\n", m.ContentHash, len(m.Versions))
}
}
```
## Diff
```go theme={null}
func (c *Client) Diff(ctx context.Context, snap1, snap2 string, opts ...DiffOption) (*DiffResult, error)
```
Compares two snapshots and returns the set of added, modified, and removed
files. Each argument accepts `"latest"`, a full hash, or a unique hash prefix.
### Diff Options
| Option | Description |
| ------------------- | -------------------------------------------- |
| `WithDiffVerbose()` | Log progress while both snapshots are loaded |
### Snapshot reference errors
Snapshot readers return sentinel errors that you can inspect with `errors.Is`:
```go theme={null}
result, err := client.Diff(ctx, "abc123", "latest")
switch {
case errors.Is(err, cloudstic.ErrSnapshotNotFound):
// No snapshot matched the reference.
case errors.Is(err, cloudstic.ErrSnapshotRefAmbiguous):
// More than one snapshot matched the prefix.
case err != nil:
log.Fatal(err)
}
```
The same sentinels apply to `Restore`, `RestoreToDir`, `LsSnapshot`, and
snapshot selectors passed to `Find`.
### DiffResult
```go theme={null}
type DiffResult struct {
Ref1 string
Ref2 string
Changes []Change
}
// Change.Type is one of: "added", "modified", "removed", "unchanged"
type Change struct {
Type string
Path string
}
```
### Example
```go theme={null}
result, err := client.Diff(ctx, "abc123", "latest")
if err != nil {
log.Fatal(err)
}
for _, c := range result.Changes {
if c.Type != "unchanged" {
fmt.Printf("%s %s\n", c.Type, c.Path)
}
}
```
## Prune
```go theme={null}
func (c *Client) Prune(ctx context.Context, opts ...PruneOption) (*PruneResult, error)
```
Removes unreachable objects (mark-and-sweep garbage collection). Run after `Forget` to reclaim storage.
### Prune Options
| Option | Description |
| -------------------- | ----------------------------------------- |
| `WithPruneDryRun()` | Count deletions without removing anything |
| `WithPruneVerbose()` | Log each deleted object |
### PruneResult
```go theme={null}
type PruneResult struct {
ObjectsScanned int
ObjectsDeleted int
BytesReclaimed int64
DryRun bool
}
```
## Forget
### Remove a specific snapshot
```go theme={null}
func (c *Client) Forget(ctx context.Context, snapshotID string, opts ...ForgetOption) (*ForgetResult, error)
```
### Apply a retention policy
```go theme={null}
func (c *Client) ForgetPolicy(ctx context.Context, opts ...ForgetOption) (*PolicyResult, error)
```
### Forget Options
| Option | Description |
| -------------------------------- | ------------------------------------------------------------------ |
| `WithPrune()` | Run prune after forgetting |
| `WithDryRun()` | Show what would be removed without deleting |
| `WithForgetVerbose()` | Verbose logging |
| `WithKeepLast(n int)` | Keep the N most recent snapshots |
| `WithKeepHourly(n int)` | Keep one snapshot per hour for the last N hours |
| `WithKeepDaily(n int)` | Keep one snapshot per day for the last N days |
| `WithKeepWeekly(n int)` | Keep one snapshot per week for the last N weeks |
| `WithKeepMonthly(n int)` | Keep one snapshot per month for the last N months |
| `WithKeepYearly(n int)` | Keep one snapshot per year for the last N years |
| `WithFilterTag(tag string)` | Only consider snapshots with this tag (repeatable) |
| `WithFilterSource(src string)` | Only consider snapshots from this source type |
| `WithFilterAccount(acct string)` | Only consider snapshots from this account |
| `WithFilterPath(path string)` | Only consider snapshots from this path |
| `WithGroupBy(fields string)` | Comma-separated grouping fields (default: `"source,account,path"`) |
### PolicyResult
```go theme={null}
type PolicyResult struct {
Groups []PolicyGroupResult
Prune *PruneResult // set only when WithPrune() was passed
DryRun bool
}
type PolicyGroupResult struct {
Key GroupKey
Keep []KeepReason
Remove []SnapshotEntry
}
```
`GroupKey` and `KeepReason` are internal types you can read fields from (as in the example below) without importing them — see the note under [List](#list).
### Example
```go theme={null}
// Retention policy: keep last 10, 7 daily, 4 weekly, 12 monthly: then prune
result, err := client.ForgetPolicy(ctx,
cloudstic.WithKeepLast(10),
cloudstic.WithKeepDaily(7),
cloudstic.WithKeepWeekly(4),
cloudstic.WithKeepMonthly(12),
cloudstic.WithPrune(),
)
if err != nil {
log.Fatal(err)
}
for _, group := range result.Groups {
fmt.Printf("Group %s: keep %d, remove %d\n",
group.Key, len(group.Keep), len(group.Remove))
}
if result.Prune != nil {
fmt.Printf("Reclaimed %d bytes\n", result.Prune.BytesReclaimed)
}
```
## Check
```go theme={null}
func (c *Client) Check(ctx context.Context, opts ...CheckOption) (*CheckResult, error)
```
Verifies repository integrity by walking the full reference chain (snapshots → HAMT nodes → filemeta → content → chunks).
### Check Options
| Option | Description |
| ----------------------------- | -------------------------------------------------- |
| `WithReadData()` | Re-hash all chunk data for byte-level verification |
| `WithCheckVerbose()` | Log each object as it is verified |
| `WithSnapshotRef(ref string)` | Check only the specified snapshot (default: all) |
### CheckResult
```go theme={null}
type CheckResult struct {
SnapshotsChecked int
ObjectsVerified int
Errors []CheckError
}
type CheckError struct {
Type string // "missing", "corrupt", "unreadable"
Key string // object key
Message string
}
```
### Example
```go theme={null}
result, err := client.Check(ctx, cloudstic.WithReadData())
if err != nil {
log.Fatal(err)
}
if len(result.Errors) > 0 {
for _, e := range result.Errors {
fmt.Printf("[%s] %s: %s\n", e.Type, e.Key, e.Message)
}
}
```
## BreakLock
```go theme={null}
func (c *Client) BreakLock(ctx context.Context) ([]*RepoLock, error)
```
Removes stale repository lock files. Returns the list of removed locks (empty slice if none found).
```go theme={null}
removed, err := client.BreakLock(ctx)
if err != nil {
log.Fatal(err)
}
for _, lock := range removed {
fmt.Printf("Removed %s lock (held by %s)\n", lock.Operation, lock.Holder)
}
```
### ErrRepoLocked
`Backup`, `Restore`, and `Prune` return `ErrRepoLocked` when the repository is already held by another operation. Check for it with `errors.Is` and prompt the caller toward `BreakLock` rather than treating it as a generic failure:
```go theme={null}
result, err := client.Backup(ctx, src)
if errors.Is(err, cloudstic.ErrRepoLocked) {
// Another operation holds the repository. Only break the lock if you are
// certain nothing is actually running against it.
removed, breakErr := client.BreakLock(ctx)
// ...
}
```
## Cat
```go theme={null}
func (c *Client) Cat(ctx context.Context, keys ...string) ([]*CatResult, error)
```
Fetches the raw (decrypted, decompressed) data for one or more object keys. Useful for debugging and inspection.
### CatResult
```go theme={null}
type CatResult struct {
Key string // the object key requested
Data []byte // raw object data (typically JSON)
}
```
### Object Key Namespaces
| Key | Description |
| ----------------- | ------------------------------------- |
| `config` | Repository configuration marker |
| `index/latest` | Latest snapshot pointer |
| `index/snapshots` | Snapshot catalog |
| `snapshot/` | Snapshot manifest |
| `node/` | HAMT tree node |
| `filemeta/` | File metadata object |
| `content/` | Content manifest (list of chunk refs) |
| `chunk/` | Raw data chunk |
| `keys/` | Encryption key slot |
| `lock/` | Repository lock file |
### Example
```go theme={null}
results, err := client.Cat(ctx, "config", "index/latest")
if err != nil {
log.Fatal(err)
}
for _, r := range results {
fmt.Printf("=== %s ===\n%s\n\n", r.Key, r.Data)
}
```
## Key Management
These package-level functions operate on the raw (unencrypted) store.
### List key slots
```go theme={null}
func ListKeySlots(ctx context.Context, rawStore store.ObjectStore) ([]KeySlot, error)
```
Returns metadata for all key slots. Does not require authentication. Slot metadata is stored unencrypted.
```go theme={null}
slots, err := cloudstic.ListKeySlots(ctx, rawStore)
for _, s := range slots {
fmt.Printf("Type: %-12s Label: %s\n", s.SlotType, s.Label)
}
```
### Change password
```go theme={null}
func ChangePassword(ctx context.Context, rawStore store.ObjectStore, kc keychain.Chain, pwd PasswordProvider) error
```
Replaces the password key slot. `kc` must unlock the current master key. `pwd` supplies the new password.
Use `PasswordString` for a known value at call time:
```go theme={null}
err := cloudstic.ChangePassword(ctx, rawStore, kc,
cloudstic.PasswordString("new-passphrase"),
)
```
Use `PasswordProviderFunc` when the new password must be obtained lazily (e.g. interactive prompt):
```go theme={null}
err := cloudstic.ChangePassword(ctx, rawStore, kc,
cloudstic.PasswordProviderFunc(func(ctx context.Context) (string, error) {
return promptUser("New password: ")
}),
)
```
### Add recovery key
```go theme={null}
func AddRecoveryKey(ctx context.Context, rawStore store.ObjectStore, kc keychain.Chain, opts AddRecoveryKeyOptions) (string, error)
```
Generates a BIP39 recovery key slot and returns the 24-word mnemonic. `kc` must unlock the current master key.
```go theme={null}
type AddRecoveryKeyOptions struct {
Label string // names the slot; empty means the default slot
Replace bool // overwrite an existing slot with the same label
}
```
A distinct `Label` lets a repository hold several recovery keys, all valid at once. Without `Replace: true`, adding a recovery key under a label that already exists returns a `*keychain.SlotExistsError` and writes nothing — this protects a previously issued mnemonic from being silently invalidated.
```go theme={null}
mnemonic, err := cloudstic.AddRecoveryKey(ctx, rawStore, kc, cloudstic.AddRecoveryKeyOptions{})
if err != nil {
log.Fatal(err)
}
fmt.Println("Recovery key:", mnemonic)
// Store this securely: it is returned only once!
```
## Workstation Setup and Source Discovery
These are no longer part of the client library. Workstation onboarding —
discovering local backup candidates, proposing a set of profiles, and applying
that plan — moved to an internal package, because it is a CLI wizard rather
than a library capability: none of it needed a repository client, and the
entry points were package-level functions rather than `Client` methods.
Use the CLI instead:
```bash theme={null}
cloudstic source discover # list local backup candidates
cloudstic setup # plan and apply a set of profiles
```
The profiles file those commands write is still readable and writable
programmatically — see [Backup Profiles](#backup-profiles) below.
## Backup Profiles
The profiles YAML format lives in `github.com/cloudstic/cli/pkg/profile`. It is
a separate package from the client precisely so that reading or writing a
user's profiles does not require opening a repository:
```go theme={null}
import "github.com/cloudstic/cli/pkg/profile"
cfg, err := profile.Load(profilesPath) // or profile.LoadOrEmpty, which
if err != nil { // treats a missing file as empty
log.Fatal(err)
}
profile.EnsureMaps(cfg)
cfg.Profiles["documents"] = profile.Profile{
Source: "local:/Users/me/Documents",
Store: "primary",
Tags: []string{"workstation"},
}
if err := profile.Save(profilesPath, cfg); err != nil {
log.Fatal(err)
}
```
`Save` validates before writing, so a malformed secret reference is rejected
there rather than at backup time. The failure is a `*secretref.Error`, so you
can branch on `Kind` rather than matching on the message:
```go theme={null}
var refErr *secretref.Error
if errors.As(err, &refErr) && refErr.Kind == secretref.KindInvalidRef {
// the scheme://path in a profile field is malformed
}
```
| Type | Purpose |
| ----------------- | ------------------------------------------------------- |
| `profile.Config` | The whole file: `Version`, `Stores`, `Profiles`, `Auth` |
| `profile.Profile` | One backup job — source, store, tags, exclusions |
| `profile.Store` | A named store definition, including secret references |
| `profile.Auth` | A reusable cloud auth entry |
`profile.DefaultPath("")` returns where the file lives when the user names no
path — `profiles.yaml` inside the config directory, honouring
`CLOUDSTIC_CONFIG_DIR`. Use it so your program reads the same file the
`cloudstic` CLI does rather than a second one that silently disagrees. It
resolves a path without touching the filesystem, so asking the question creates
nothing.
### Acting on a Profile
Reading a profile is one thing; *using* one is another. `pkg/open` turns a named
profile into a connected client in one call:
```go theme={null}
import "github.com/cloudstic/cli/pkg/open"
// An empty path means the default location.
client, err := open.FromProfile(ctx, "", "documents")
if err != nil {
log.Fatal(err)
}
```
This is the one-call form of the explicit sequence, which you want instead
whenever something has to happen in between — displaying the resolved
configuration, diffing it, or deciding something from it before connecting:
```go theme={null}
cfg, err := profile.Load(profilesPath)
if err != nil {
log.Fatal(err)
}
// Which store does this profile select? nil means the profile names none,
// which is legal — the CLI then takes it from -store or CLOUDSTIC_STORE.
storeCfg, err := cfg.StoreFor("documents")
if err != nil {
log.Fatal(err)
}
clientCfg, err := config.FromProfileStore(ctx, *storeCfg, resolver)
if err != nil {
log.Fatal(err)
}
fmt.Printf("connecting to %s\n", clientCfg.Store.URI)
client, err := open.Client(ctx, clientCfg)
```
A client, once returned, is already connected — so anything you want to override
must be decided before this point, not after.
### Layering Your Own Configuration
If your program has a configuration mechanism of its own — command-line flags,
its own file, a form — `WithDecided` layers it over the profile, with the same
precedence the `cloudstic` CLI applies to its flags. Every field you name is
taken from your value, and the profile supplies the rest:
```go theme={null}
mine := config.Client{Store: config.Store{URI: "local:/my-own-choice"}}
client, err := open.FromProfile(ctx, "", "documents",
open.WithDecided(mine, config.FieldsSetIn(mine)))
```
`config.FieldsSetIn` reports which fields your value is non-empty for, which is
the right answer when your mechanism has no notion of "present but empty".
When empty *is* a choice you need to keep — `-password ""` meaning "no
password", as the CLI treats it — name the fields explicitly instead:
```go theme={null}
decided := config.NewFieldSet(config.FieldPassword, config.FieldS3AccessKey)
```
The field names are typed constants rather than strings, because the profiles
file and the CLI spell the same field differently — `s3_access_key` against
`-s3-access-key` — and a misspelled string silently meant "not decided", which
would connect you with a credential you had not chosen. `config.StoreFields()`
and `config.BackupFields()` enumerate the complete set, so iterating them keeps
your mapping current as fields are added; `Field.ProfileKey()` renders the
profiles-file spelling for an error message.
Two groups of fields behave differently on purpose. Location and KMS settings
are taken only when the profile actually names one, so a silent profile leaves
what you had. Credentials are taken whenever you have not decided them, *empty
included* — selecting a profile clears an ambient credential, because a profile
is an explicit choice of which store to talk to, and reaching it with half a
credential set inherited from the environment would be worse than failing to
reach it.
A field you have decided is never resolved, so a broken secret reference on a
field you are about to replace is not an error.
### Backing Up From a Profile
The same layering applies to what a profile says to back up. `config.Backup`
holds the resolved form, and `open.Backup` constructs both the source and the
options a backup runs with:
```go theme={null}
bcfg, err := config.MergeProfileBackup(config.Backup{}, nil, "documents", cfg)
if err != nil {
log.Fatal(err)
}
job, err := open.Backup(ctx, bcfg)
if err != nil {
log.Fatal(err)
}
result, err := client.Backup(ctx, job.Source, job.Options...)
```
`open.Backup` returns the source and the options **together** rather than as two
calls, because they share a derived value. The snapshot records a hash of the
active exclude patterns, and the next backup compares its own against it to
decide between an incremental scan and a full rescan. The patterns come from two
places — `Excludes` and the file named by `ExcludeFile` — so a hash computed
without reading that file describes a different exclude set than the one in
force, and every subsequent run reads the difference as "the patterns changed".
Getting the source and the options from one call is what keeps them consistent.
If you supply your own `source.Source` implementation, use
`open.BackupOptions(cfg)` for the options and apply `cfg.Source.Excludes`
yourself — otherwise the recorded hash describes filtering that is not happening.
`source.ExcludeHash` is exported if you need to compute it directly.
| Function | Purpose |
| ----------------------------------------------------- | ------------------------------------------------- |
| `open.FromProfile(ctx, path, name, opts...)` | Profile → connected client, in one call |
| `open.Client(ctx, config.Client, opts...)` | Resolved configuration → connected client |
| `open.Store(ctx, config.Store, opts...)` | Store URI + credentials → raw `store.ObjectStore` |
| `open.Source(ctx, config.Source, opts...)` | Source URI + credentials → `source.Source` |
| `open.Backup(ctx, config.Backup, opts...)` | → `BackupJob{Source, Options}` |
| `config.FromProfileStore(ctx, s, r)` | Profile store definition → `config.Client` |
| `config.MergeProfileStore(ctx, base, decided, s, r)` | The same, layered under your own values |
| `config.MergeProfileBackup(base, decided, name, cfg)` | Profile backup settings → `config.Backup` |
## Secret References
Store and source credentials can be written as `scheme://path` references
instead of inline secrets. `github.com/cloudstic/cli/pkg/secretref` resolves
them, and `pkg/secretref/backends` holds the schemes Cloudstic ships with:
`env://`, `file://`, `config-token://`, `keychain://`, `secret-service://` and
`wincred://`.
```go theme={null}
import (
"github.com/cloudstic/cli/pkg/secretref"
"github.com/cloudstic/cli/pkg/secretref/backends"
)
resolver := backends.NewDefaultResolver()
password, err := resolver.Resolve(ctx, "keychain://cloudstic/store/prod")
```
Failures are typed, so you can tell a malformed reference from a missing value
from a backend that is not available on this platform:
| `Kind` | Meaning |
| ---------------------------------- | ------------------------------------------------------------------- |
| `secretref.KindInvalidRef` | The `scheme://path` is malformed |
| `secretref.KindNotFound` | The backend works, but has no value at that path |
| `secretref.KindBackendUnavailable` | No backend is registered for that scheme, or it is unsupported here |
### Adding your own scheme
`Backend` is a single method, so a custom scheme — Vault, a cloud KMS, your own
service — is small. `backends.Default()` returns a **fresh map on every call**,
which is what lets you *extend* the built-in set rather than replace it:
```go theme={null}
type vaultBackend struct{ client *vault.Client }
func (v *vaultBackend) Resolve(ctx context.Context, ref secretref.Ref) (string, error) {
secret, err := v.client.Read(ctx, ref.Path)
if err != nil {
return "", secretref.NewError(secretref.KindNotFound, ref.Raw, "no such vault secret", err)
}
return secret, nil
}
b := backends.Default()
b["vault"] = &vaultBackend{client: c}
resolver := secretref.NewResolver(b)
```
Use `secretref.NewError` for failures so callers can branch on `Kind` the same
way they do for the built-in backends.
Implement `WritableBackend` as well (adding `Scheme`, `DisplayName`,
`WriteSupported`, `DefaultRef`, `Exists` and `Store`) if the scheme should
also be offered when the CLI and TUI *write* a secret, rather than only when
resolving one.
### Choosing where managed tokens live
The `config-token://` backend stores tokens itself, encrypted at rest, and by
default puts them in Cloudstic's own config directory — `CLOUDSTIC_CONFIG_DIR`
or the OS default. For an embedding program that is usually the wrong place,
and an environment variable is a poor way for a library to be configured:
```go theme={null}
b := backends.Default()
b["config-token"] = backends.NewConfigTokenBackend(
backends.WithConfigDir("/var/lib/myapp/secrets"),
)
resolver := secretref.NewResolver(b)
```
The salt that derives the at-rest encryption key lives in that directory too.
Two backends pointed at different directories therefore hold different keys —
tokens written under one are **not** readable under the other. Changing the
directory of an existing deployment orphans whatever was already stored there.
## Progress Reporting
`cloudstic.Reporter` receives progress events during long-running operations:
```go theme={null}
type Reporter interface {
StartPhase(name string, total int64) Phase
}
type Phase interface {
Increment(n int64)
Done()
}
```
Pass your reporter to `NewClient`:
```go theme={null}
client, err := cloudstic.NewClient(ctx, rawStore,
cloudstic.WithKeychain(kc),
cloudstic.WithReporter(myReporter),
)
```
If `WithReporter` is omitted, all progress output is suppressed (no-op default).
## Complete Example: Automated Backup
```go theme={null}
package main
import (
"context"
"encoding/hex"
"fmt"
"log"
"os"
cloudstic "github.com/cloudstic/cli"
"github.com/cloudstic/cli/pkg/keychain"
"github.com/cloudstic/cli/pkg/source"
"github.com/cloudstic/cli/pkg/store"
)
func main() {
ctx := context.Background()
// Load platform key from environment
encKeyHex := os.Getenv("CLOUDSTIC_ENCRYPTION_KEY")
encKey, err := hex.DecodeString(encKeyHex)
if err != nil {
log.Fatalf("invalid CLOUDSTIC_ENCRYPTION_KEY: %v", err)
}
// Open S3 backend
rawStore, err := store.NewS3Store(ctx, "my-backup-bucket",
store.WithS3Region("us-east-1"),
)
if err != nil {
log.Fatal(err)
}
// Build keychain
kc := keychain.Chain{keychain.WithPlatformKey(encKey)}
// Initialize if not already done (adopt-slots makes it idempotent)
if _, err := cloudstic.InitRepo(ctx, rawStore,
cloudstic.WithInitCredentials(kc),
cloudstic.WithInitAdoptSlots(),
); err != nil {
log.Fatal(err)
}
// Create client
client, err := cloudstic.NewClient(ctx, rawStore, cloudstic.WithKeychain(kc))
if err != nil {
log.Fatal(err)
}
// Backup
src := source.NewLocalSource("/data",
source.WithLocalExcludePatterns([]string{"*.tmp", ".cache/"}),
)
result, err := client.Backup(ctx, src, cloudstic.WithTags("automated"))
if err != nil {
log.Fatalf("backup failed: %v", err)
}
fmt.Printf("Backup complete: %s (%d new, %d changed files)\n",
result.SnapshotHash, result.FilesNew, result.FilesChanged)
// Apply retention policy and prune
_, err = client.ForgetPolicy(ctx,
cloudstic.WithKeepLast(7),
cloudstic.WithKeepWeekly(4),
cloudstic.WithKeepMonthly(12),
cloudstic.WithPrune(),
)
if err != nil {
log.Fatalf("retention policy failed: %v", err)
}
}
```
## See Also
* [Source Interface](/advanced/source-interface): Implement a custom backup source
* [Storage Model](/advanced/storage-model): How objects are stored and addressed
* [HAMT Structure](/advanced/hamt-structure): The file index data structure
* [Backup Flow](/advanced/backup-flow): End-to-end walkthrough of the backup process