Skip to main content
Welcome! We appreciate your help in making Cloudstic better. This guide covers development setup, testing, debugging, and contribution workflows.

Development Setup

Prerequisites

  • Go 1.26 or later (see go.mod for the exact minimum)
  • Docker (for hermetic E2E tests using Testcontainers)
  • golangci-lint (for linting)
  • Node.js (for markdownlint-cli2, run via npx by scripts/check.sh)

Clone the Repository

Build the Binary

The binary will be created at bin/cloudstic.

Project Structure

Cloudstic CLI is organized into clear package boundaries:
  • client.go (root) - Public Client API for programmatic use. Re-exports types from internal packages via Go type aliases.
  • cmd/cloudstic/ - CLI entry point (package main). main.go is a thin dispatcher; each command lives in its own cmd_<name>.go file, registered in commands.go’s commandRegistry(). See CLI Integration below.
  • internal/engine/ - Business logic for operations (backup, restore, prune, forget, diff, list, find). Each operation has a *Manager struct.
  • internal/core/ - Repository-format types: Snapshot, Content, HAMTNode, RepoConfig, plus ComputeJSONHash. FileMeta, SourceInfo and FileType are defined in pkg/source and aliased here, so the public Source contract does not depend on an internal package.
  • internal/hamt/ - Persistent Merkle Hash Array Mapped Trie backed by the object store.
  • pkg/source/ - The Source and IncrementalSource contract, plus shared helpers (ExcludeMatcher, TopoSortFolderChanges). Depends on nothing outside the standard library, so implementing a source pulls in no provider SDK. Implementations live in their own subpackages: pkg/source/{local,sftp,gdrive,onedrive}. See the Source Interface guide.
  • internal/sourceoauth/ - OAuth2 machinery shared by the Google Drive and OneDrive sources. Also holds the -X ldflags targets for the default OAuth client IDs; those symbol paths are mirrored in .goreleaser.yml and must move together, since the linker silently ignores -X for a symbol that does not exist.
  • pkg/store/ - The ObjectStore contract, its capability interfaces (RangeGetter, ConcurrencyHinter, Unwrapper) and the order-independent wrappers QuotaStore/DebugStore. Depends on nothing outside the standard library, so implementing a custom backend pulls in no vendor SDK. Backends live in pkg/store/{local,s3,b2,sftp}/ (local.New, s3.New, …); only pkg/store/s3 carries the AWS SDK. pkg/store/storetest holds shared test doubles.
  • internal/storelayer/ - The repository-format decorator chain (compression, encryption, metering, packfiles, key cache). Internal because its composition order is a security invariant — see the Store Decorator Stack below.
  • pkg/crypto/ - AES-256-GCM encryption, HKDF key derivation, BIP39 mnemonic recovery keys.
  • pkg/keychain/ - OS keychain integration and encryption key-slot helpers.
  • internal/app/ - Orchestration layer shared by the CLI and TUI (profile listing, health checks, backup actions).
  • internal/tui/ - The interactive terminal dashboard (Bubble Tea).
  • pkg/secretref/ - The scheme://path secret-reference contract: Ref, Parse, Backend, Resolver, NewResolver, Error. Public so a third party can register a custom backend (Vault, a cloud KMS) from another module.
  • pkg/secretref/backends/ - The built-in backends (env://, file://, config-token://, keychain://, secret-service://, wincred://) plus Default(), which returns a fresh map so callers extend the built-in set rather than replacing it.
  • pkg/profile/ - The backup-profiles YAML format (Config, Profile, Store, Auth, Load, Save). Separate from the client so profiles can be read and written without opening a repository.
  • internal/workstation/ - Workstation onboarding (Plan, Apply, Setup) and local source discovery. Internal: it is a CLI wizard, not a library capability.
  • internal/paths/ - Config-directory and token-path resolution.
  • internal/pathmatch/ - Glob matching for slash-separated paths, including **.
  • internal/logger/, internal/retry/, internal/sftp/ - Structured logging, retry/backoff helpers, and the shared SFTP client used by both the SFTP source and store.
  • internal/ui/ - Non-interactive console progress reporting and terminal helpers.
See AGENTS.md in the repository root for the full, actively-maintained architecture documentation — it’s the canonical reference this page is kept in sync with.

Build & Test Commands

Run All Tests

This runs unit tests and hermetic E2E tests (using Testcontainers for MinIO and SFTP).
Docker is required for hermetic E2E tests. Tests will be skipped if /var/run/docker.sock is not available.

Run a Single Test

Example:

Run with Race Detector

The race detector catches concurrency bugs. Always run tests with -race during development.

Run the Full Check Script

This runs:
  1. go fmt - Format check
  2. golangci-lint run - Linting
  3. markdownlint-cli2 (via npx) - Markdown lint, including this guide and every other .md file in the repo
  4. go test -race -count=1 ./... - Tests with race detection
  5. Coverage report generation

Format Code

Lint Code

E2E Test Modes

E2E tests in e2e/ are controlled by the CLOUDSTIC_E2E_MODE environment variable:
  • hermetic (default) - Local filesystem + Testcontainers (MinIO, SFTP). Requires Docker.
  • live - Real cloud vendor APIs (requires secrets in environment variables).
  • all - Runs both hermetic and live tests.

Running Hermetic Tests

or explicitly:

Running Live Tests

Live tests require cloud provider credentials (AWS, Backblaze B2, Google Drive, OneDrive, SFTP servers) configured via environment variables.

Running All Tests

Debugging

Enable Debug Logging

Append the -debug flag to any CLI command to enable verbose internal logging:
This outputs:
  • Detailed timings for every GET, PUT, LIST, and DELETE operation
  • Cache hits/misses
  • Memory management decisions
  • Engine operation traces
Debug logging is extremely useful for tracing API calls, caching behaviors, and performance bottlenecks.

Attach a Debugger

You can use dlv (Delve) to debug the CLI:
Or attach to a running process:

Profiling

Cloudstic supports standard Go profiling via hidden flags on any command:

CPU Profiling

The CPU profile flag also automatically generates:
  • cpu.prof.goroutine - Goroutine dump
  • cpu.prof.block - Block profile
  • cpu.prof.mutex - Mutex profile

Memory Profiling

View Profiles

Use go tool pprof to analyze profiles:

Development Best Practices

When Adding New Features

Always consider the following:

1. Documentation

Check if user-facing documentation needs updates:
  • docs/user-guide.md - Add command documentation with usage examples, flags, and descriptions.
  • README.md - Update if the feature changes the quick start or high-level overview.
  • Code comments - Document public APIs, especially in client.go and package interfaces.

2. Unit Tests

Add test coverage when it makes sense:
  • Always add tests for new public API methods (e.g., Client.*() methods).
  • Test both success and error cases.
  • Test integration with encryption/compression if applicable.
  • Use existing test patterns (see client_test.go, internal/engine/*_test.go).
  • Mock stores are available in internal/engine/mock_test.go for testing.

3. Client API

For new operations, expose them via the Client struct:
  • CLI commands should use Client methods, not directly access stores.
  • This allows library users to programmatically use the functionality.
  • Follow the pattern: define types/options, add a Client.*() method, implement in internal/engine/ if complex.

4. CLI Integration

For new commands, don’t hand-write a flag parser or edit main.go. Instead:
  • Add a cmd_<name>.go file next to the code it implements, following the pattern every existing command uses (cmd_backup.go, cmd_check.go, cmd_breaklock.go, …):
    • declare<Name>Args(g *globalFlags) (*<name>Args, commandInput) declares the command’s complete input surface — flags via stringFlag/boolFlag/intFlag/valueFlag, positionals via requiredPositional/optionalPositional. This also carries environment, secret, help, and completion metadata.
    • run<Name>(r *runner, ctx context.Context, a *<name>Args) int is the command body. It writes output via r.out/r.errOut (never fmt.Print) so it’s capturable in tests.
    • A <name>Command() command function declares the runnable leaf: leaf(name, summary, groups, declare<Name>Args, run<Name>). Pass repoCommandGroups (or backupCommandGroups for a command that reads a source) to opt into only the global flags the command needs.
  • Add the new <name>Command() to the ordered list in commandRegistry() (cmd/cloudstic/commands.go) — that’s the single source of truth runCmd() dispatches from, printUsage() renders COMMANDS from, and shell completion is generated from. Never edit main.go, usage.go, or completion.go directly for a new command.
  • Mark any flag carrying a credential with asSecret() so it never leaks into -h output (TestSecretEnvValuesNeverAppearInHelp enforces this).
  • If the command changes root or per-command help text, regenerate the golden files: go test ./cmd/cloudstic -run 'TestRootUsageGolden|TestCommandHelpGolden' -update.

5. Error Handling

Return descriptive errors:
  • Wrap errors with context using fmt.Errorf("context: %w", err).
  • Provide actionable error messages to users.
  • Distinguish between user errors and system errors.
  • If a caller (CLI or library) needs to programmatically detect a specific failure mode — not just display it — define a sentinel error (var ErrX = errors.New(...)) and wrap it with %w, rather than making the caller match on error text. See engine.ErrRepoLocked / cloudstic.ErrRepoLocked for a recent example wired through to a CLI hint.

Example: Adding a New Command

Let’s say you want to add a stats command that shows repository statistics. Step 1: Add Client Method
Step 2: Implement Engine Logic
Step 3: Declare and Implement the CLI Command
Note the shape: run<Name> takes the shared runner (so output is capturable and r.client is injectable in tests) instead of building its own client and flag set; presentation lives in a separate print* function taking io.Writer first, never given access to runner. Step 4: Register the Command
That single entry is enough: dispatch (runCmd() in main.go), the cloudstic help listing, and shell completion all derive from this list. Step 5: Add Tests
Step 6: Update Documentation Add command documentation to docs/user-guide.md and (if stats has notable flags) run go test ./cmd/cloudstic -run TestCommandHelpGolden -update to refresh its golden -h output.

Testing Guidelines

Choose the smallest test style that covers the behavior — the four styles below cover different layers, and picking the wrong one for a given change is a common review comment:
  • Unit tests for argument parsing, orchestration branches, domain logic, and individual error cases. Inject a stubClient (cmd/cloudstic/stub_client_test.go) to test CLI command flow without a real repository.
  • Golden-file tests for deterministic print*/render* presentation output, where the exact full text is the contract.
  • Testscript tests for whole-command behavior that crosses the process boundary: flag ordering, stdout/stderr separation, exit codes, filesystem effects.
  • E2E tests (e2e/, Testcontainers-backed) for behavior that needs a real backend (MinIO, SFTP) end to end.
Don’t reach for a golden file when a value is inherently unstable (timestamps, hashes), and don’t replace a focused unit test with a testscript when a direct assertion gives a clearer failure message.

Test Coverage

Aim for high test coverage, especially for:
  • Public API methods in client.go
  • Engine logic in internal/engine/
  • Store implementations in pkg/store/
  • Crypto operations in pkg/crypto/

Test Patterns

Unit Tests

Use mock stores for isolated engine testing:
For CLI command flow, inject a stubClient instead of a real repository — see TestRunStats in the command example above.

Golden-File Tests

Write deterministic output to a buffer and compare it against cmd/cloudstic/testdata/*.golden with assertGolden:
Regenerate an intentionally changed golden file with -update, and review the diff as part of the change:

Testscript Tests

Add a hermetic .txtar script under cmd/cloudstic/testdata/scripts/ for whole-process behavior. Prefer local stores/sources so the script needs no network access, credentials, or Docker:

Integration Tests

Use real stores with temporary directories:

E2E Tests

Use Testcontainers for hermetic E2E tests:

Before Committing

Always run the full check script:
This ensures:
  • Code is formatted correctly
  • No linting errors
  • All tests pass
  • Race conditions are detected
  • Coverage is adequate

Architecture Overview

Store Layering

Stores are composed as a decorator chain (from outermost to innermost):
  • CompressedStore - zstd compression on write, auto-detects zstd/gzip/raw on read.
  • EncryptedStore - AES-256-GCM. Passes through objects under keys/ prefix unencrypted.
  • MeteredStore - Tracks bytes written for reporting.
  • PackStore (optional) - Bundles small objects (less than 512KB) into 8MB packfiles to reduce API calls. Each packfile ends with a self-describing footer, so its catalog (index/packs) is a rebuildable cache rather than the sole source of truth — see RFC 0018 if you’re touching this layer.
  • KeyCacheStore - Caches key existence in a temporary bbolt database.
  • Backend - local.Store, s3.Store, b2.Store, or sftp.Store, each in its own subpackage under pkg/store/.
Do not assemble this chain yourself. PackStore sits below EncryptedStore, so its catalog and footers never pass through encryption and need a separately derived key; a hand-built chain missing it produces a repository whose pack index is plaintext, with no error at any layer. That is why these types live in internal/storelayer. To add your own wrapper, implement store.ObjectStore and pass it to NewClient — it layers the chain on top, exactly as the CLI does with DebugStore.

Repository Compatibility

This is the most important constraint in the codebase for anything that touches what gets written to a store. Read docs/compatibility.md in the repository before changing on-disk format — it’s normative and takes precedence over convenience.
  • Backward compatibility is permanent. A repository written by any released version must stay readable by every later version (list, ls, check, cat, diff, restore). There is no deprecation window for reads.
  • Forward compatibility isn’t guaranteed, but failure must be safe. An older build may be unable to read a newer repository, but it must never misread it as empty or valid — “cannot decode” must always surface as an error, never as “no entries,” because that’s exactly the condition that lets a garbage collector delete a live repository.
  • The version gate (core.RepoFormatVersion / core.MaxSupportedRepoFormat in internal/core/models.go) gates every repository open. Raise it only when a change would make a repository unreadable or misreadable by earlier builds.
  • Changing the on-disk format requires: keeping older layouts readable, upgrading only opportunistically (never requiring a migration to read), committing a fixture from the last release in the old format, deciding on the version gate, adding the baseline to docs/compatibility.md’s table, and stating in the PR what older builds do when they meet the new format — verified by running an old binary, not by reasoning about it. e2e/feature_legacy_repo_test.go enforces the fixture and doc-table requirements.

Backup Flow

  1. BackupManager acquires a shared lock, loads the previous snapshot (if any) for its source identity.
  2. Source is scanned via Walk() (full) or WalkChanges() (incremental).
  3. New/changed files are chunked using FastCDC, content-addressed, and uploaded.
  4. The HAMT tree is updated with new filemeta refs. TransactionalStore buffers all intermediate HAMT nodes and only flushes reachable ones from the final root.
  5. A new Snapshot object is written, and index/latest is updated.

Encryption Model

  • On init, a random 32-byte master key is generated and wrapped into key slots (password-based via scrypt, platform key, KMS-wrapped platform key, or BIP39 recovery key).
  • Key slots are stored under keys/ prefix, which the EncryptedStore passes through unencrypted.
  • An HMAC dedup key is derived from the encryption key via HKDF for content-addressing without exposing plaintext hashes.

Naming Conventions

Commits & PR titles use a Conventional Commit prefix — type: imperative summary, or type(scope): …. Lowercase the summary after the colon, no trailing period, keep it short (~72 chars) and specific about what changed. PRs are squash-merged, so the PR title becomes the commit subject — give both the same form.
  • Types: feat, fix, refactor, test, docs, chore, perf, ci.
  • An optional scope names the area: feat(tui):, fix(completion):.
Branch names are <type>/<kebab-slug> (e.g. feat/tui-profile-history), matching the commit type. Issue titles carry no conventional-commit prefix — the type lives in the label. Lead with an imperative verb (Add …, Convert …) or an Area: scanning prefix, no trailing period. RFCs: a substantial feature gets a design record under rfcs/NNNN-kebab-slug.md (zero-padded, next free number). The proposal PR/commit is rfc: <summary> (RFC NNNN); the implementation PR/commit is a standard type: prefix with a trailing (RFC NNNN) reference, e.g. feat: unified source identity (RFC 0009). See rfcs/README.md for the index and RFC 0010 as a template.

Pull Request Guidelines

Before Submitting

  1. Run tests: ./scripts/check.sh
  2. Update documentation: Add/update user guide, README, and code comments
  3. Add tests: Cover new functionality with unit/golden-file/testscript tests as appropriate
  4. Format code: go fmt ./...
  5. Lint code: golangci-lint run ./...

PR Description

Two sections:
Keep the Summary bullets high-signal — what changed and why, not a file-by-file diff. Under Verification, paste the exact commands you ran, e.g.:

Creating an Issue

Issues use four sections — ## Context (current state, with backtick paths to concrete files/functions), ## Goal (desired end state, one or two sentences), ## Scope (bullet list of concrete changes), and ## Acceptance Criteria (verifiable outcomes, always ending with the exact test/lint commands that must pass). Apply exactly one type label (bug, enhancement, refactor, tech debt, chore, test, documentation, rfc, tracking) plus one or more area/* labels.

Getting Help

If you have questions or need help:
  • Check AGENTS.md for architecture details
  • Review existing code for patterns
  • Open a GitHub issue for discussion
  • Join our community chat (if available)

License

By contributing, you agree that your contributions will be licensed under the same license as the project.