nirpa/granthi-sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(client): snapshot backups, restore points, scoped bulk pull

Browse Source 
Two modes per linked folder. 'mirror' keeps today's behaviour for a plain
folder that add turned into a repo. 'snapshot' is new and is for a folder
that already had a git history: nothing is ever committed on the user's
behalf, and instead each pass builds a commit object from the working tree
via a scratch index + commit-tree and pushes it to
refs/granthi-backup/<device>/<ts>. HEAD, the index and every file stay
exactly as the user left them, so uncommitted, unmerged, half-finished work
leaves the machine with a timestamp to restore from.

Verified on the beta forge (Gitea 1.27.2) that a custom ref namespace is
accepted, readable via ls-remote, and absent from the branch list.

Also: retention (all for 24h, hourly for 7d, daily beyond; unparseable
timestamps kept), snapshots/restore commands, restore never writing over the
working tree, get --all bounded by what the forge grants, list <pattern>,
.gitignore seeding, an add size guard, and a persisted device_id.

141 tests (was 108).
This commit is contained in:
claude
2026-08-23 11:22:20 -04:00
parent 1091aa61f2
commit 9e3201a296
3 changed files with 1275 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+177 -16
View File
@@ -1,4 +1,4 @@
# granthi-sync v1
# granthi-sync v1.2
The signup → download → link-folders → cloud product spine for the Granthi
forge, tested against the BETA forge (granthi-beta.shre.ai). Python 3 stdlib +
@@ -50,10 +50,17 @@ cd granthi-sync
# 3b. ...or push a local folder up. It becomes a private repo.
./bin/granthi-sync add ~/work/notes
# 3c. ...or pull down everything this account is allowed to see.
./bin/granthi-sync get --all --into ~/granthi
# 4. Keep everything synced. Autocommits, ff-pulls, pushes; skips anything
# that has diverged rather than merging or forcing.
./bin/granthi-sync watch # --once for a single pass
./bin/granthi-sync status # what is linked, last sync, divergence
./bin/granthi-sync status # what is linked, mode, last sync
# 5. Go back to how a folder looked at some point in time.
./bin/granthi-sync snapshots ~/work/notes
./bin/granthi-sync restore ~/work/notes --at 20260823T142530Z
```
Run `watch` as a background daemon on macOS with
@@ -67,6 +74,79 @@ no account, no token, no partial state. Just run `link` again.
to merge; when a folder shows `DIVERGED` in `status`, resolve it in git or on
the forge web UI. The client will never force or auto-merge your work.
## Two modes, because two very different folders ask for this
A folder people sync is either *their documents* or *their git project*, and
the correct behaviour is opposite in each case. Each linked folder therefore
carries a `mode`.
| | `mirror` | `snapshot` |
|---|---|---|
| chosen for | a plain folder `add` turned into a repo | a folder that was already a git repo, and anything `get` clones |
| commits on your behalf | yes, `sync: <ISO ts>` | **never** |
| where work lands | the branch | `refs/granthi-backup/<device>/<ts>` |
| a restore point is | every commit | every snapshot |
`snapshot` mode is what "the work may not be committed, but it is still
backed up" means in git terms. Each pass loads a scratch index from HEAD,
stages the working tree into *that* index, writes a tree, and commits it with
`commit-tree`. HEAD, your index, your stash and every file on disk are
untouched — you can be mid-rebase with a dirty tree and the backup still
records exactly what is on the disk right now. The user's history stays the
user's.
Why a custom ref namespace: verified on the beta forge (Gitea 1.27.2) that
`refs/granthi-backup/...` is accepted, is readable through `ls-remote`, and
does **not** appear in the branch list. Under `refs/heads` a machine taking a
backup every 30 seconds would bury the branches a person actually made.
Snapshots are parented on HEAD and deliberately **not** chained to the
previous snapshot: chaining would keep every old snapshot reachable from the
newest, so pruning a ref would free nothing and retention would be
decorative.
**Retention** (or 30-second backups become a disk leak nobody can navigate):
everything is kept for 24 h, then thinned to hourly for 7 days, then daily.
Pruning runs at most hourly, per device, and only over that device's own
refs. A ref whose timestamp this version cannot parse is **kept** — deleting
the unrecognised is how a backup system loses the one thing someone needed.
**Restore never writes over the working tree.** `restore` materialises a
restore point into a *new* directory and refuses a non-empty destination.
Someone restoring a backup is already having a bad day; overwriting the files
they still have would make the recovery tool the second disaster.
## Device identity
`link` mints a uuid on first run and persists it in `~/.granthi-sync/config.json`
as `device_id`, and sends it to `/v1/link`. Hostnames are neither stable
(people rename laptops) nor unique (every new mac is "Mac mini"), so a
hostname cannot key a backup ref or a device registry — two machines would
overwrite each other's snapshots. The service-side device registry is the
next phase; the client leads so the id already exists when it lands.
## What "add the computer to the network" means — and does not
The onboarding shape is: download → login → **the device is federated to the
account** → the device can reach its repos.
The middle step is a *device registration*, not a network membership. Those
sound like one step and must not be built as one: this estate's tailnet is a
single flat private network carrying the granthi VPS, aros-vps, the Shadow
box and the Mac. Putting a customer's laptop on it to let them sync a folder
would hand that laptop L3 reach to every piece of infrastructure we run.
So:
* **Our own machines** may join the tailnet — that is an operator action with
an operator's judgement behind it.
* **Customer devices never do.** Their transport is public HTTPS to
granthi-link and the forge through cloudflared. That is the same exposure
step already in the promotion window below, and it is what makes a genuinely
new computer able to onboard itself at all — today `link` only works from
inside the tailnet, which means "you can't set up a new computer without an
operator first" is the honest status.
## Components
### `server/granthi_link.py` — provisioning service (granthi VPS)
@@ -211,13 +291,29 @@ deleted it again, `DELETE …/tokens/{id}` returning 204 under basic auth):
(`authorization_pending`/`slow_down` handled), then calls `/v1/link`.
`--token` skips the device flow with a ready Zitadel token (headless/dev).
Result stored in `~/.granthi-sync/config.json` (0600).
* `list` — every repo the linked token can see, with the local folder each
is already synced to. Reads `GET /api/v1/user/repos` on the forge
* `list [pattern]` — every repo the linked token can see, with the local
folder each is already synced to. `pattern` narrows the table by name
(substring, or a glob like `work-*`), case-insensitively, against both
`owner/name` and the bare name. Filtering is display-only: the set already
came from the forge under this account's token. Reads
`GET /api/v1/user/repos` on the forge
**directly** with the scoped user token — no granthi-link round-trip, so
the read path needs no service change. Pagination is followed to a short
page; if the `FORGE_MAX_PAGES` guard trips, the output says the list is
incomplete rather than letting a bounded page read as the whole set.
* `get <repo|owner/repo> [--into DIR]` — the download half of `add`. Clones
* `get --all [--into DIR] [--mode M]` — clone every repo this account can
see, skipping the ones already linked here. **"Only the repos they are
granted" needs no client-side permission logic**: `/api/v1/user/repos` is
evaluated by the forge against this account's own scoped token, so the
list *is* the grant. A client-side filter would be a second opinion about
someone else's authorisation. One repo failing does not abandon the rest,
and a truncated listing is reported loudly — `--all` must never quietly
mean "the first 2000".
* `get <repo|owner/repo> [--into DIR] [--mode M]` — the download half of
`add`. Defaults to `snapshot` mode unless the repo carries a
`.granthi-sync.json` marker saying otherwise, so a plain synced folder
behaves the same on the second machine while someone's real project is
never autocommitted onto. Clones
with `--origin granthi` (the remote name `watch` looks for) and
`-c credential.helper=…` (the repo does not exist yet, so the helper
cannot be installed first; git also persists it into the new config), then
@@ -225,22 +321,55 @@ deleted it again, `DELETE …/tokens/{id}` returning 204 under basic auth):
so a cloned repo is picked up by `watch` immediately. Refuses a non-empty
destination. Branch is read with `symbolic-ref` (an empty repo has an
unborn HEAD) and falls back to `main`.
* `add <folder> [--name N] [--private|--public]` — `git init -b main` if
needed, creates the cloud repo via `/v1/repos`, adds remote `granthi`,
initial commit + push. The token is delivered by a **git credential
helper** (the client's hidden `git-credential` subcommand reading the 0600
config) — never embedded in the remote URL (estate rule).
* `watch [--interval 30] [--once]` — per folder: autocommit
(`sync: <ISO ts>`) → fetch → ff-pull if remote strictly ahead → push if
local strictly ahead. **DIVERGED → log + record + SKIP. Never force, never
merge** — the same policy as the mesh. SIGTERM-clean.
* `status` — table of linked folders, last sync, divergence flags.
* `add <folder> [--name N] [--private|--public] [--mode M] [--force]` —
`git init -b main` if needed, creates the cloud repo via `/v1/repos`, adds
remote `granthi`, initial commit + push. The token is delivered by a **git
credential helper** (the client's hidden `git-credential` subcommand
reading the 0600 config) — never embedded in the remote URL (estate rule).
Pushes the folder's **current** branch, not a hardcoded `main`: an existing
repo may sit on `master` or a feature branch, and publishing that work
under the wrong name is not a cosmetic error.
Two guards, because `add -A` takes whatever it is given: a starter
`.gitignore` is seeded when the folder has none (an existing one is never
touched — it is the user's), and a folder over 20 000 files / 512 MB is
refused unless `--force`. The seeded ignore file covers `.env`, `*.key`,
`*.pem`, `id_rsa` and friends, and it governs snapshots too — the scratch
index honours `.gitignore` exactly as a normal commit does.
* `watch [--interval 30] [--once]` — per folder, by mode. `mirror`:
autocommit (`sync: <ISO ts>`) → fetch → ff-pull if remote strictly ahead →
push if local strictly ahead. `snapshot`: fetch → push a snapshot of the
working tree to this device's backup ref → ff-pull only when the tree is
clean (local edits are already safe on the backup ref, so it reports and
leaves the tree alone rather than failing) → push the user's own commits
when they are strictly ahead. **DIVERGED → log + record + SKIP. Never
force, never merge** — the same policy as the mesh — **but the backup
still happens**, because divergence is when work is most at risk.
Retention pruning runs at most hourly. SIGTERM-clean.
* `snapshots <folder> [--limit 20]` — restore points, newest first. Read from
the **remote**, not a local cache: the feature exists for the case where
this machine is gone.
* `restore <folder> --at <ts|sha> [--into DIR]` — materialise one restore
point into a new directory; refuses a non-empty destination.
* `status` — table of linked folders, mode, last sync, divergence flags.
* Run as a daemon on macOS with `client/launchd/ai.granthi.sync.plist`
(edit the script path, then `launchctl bootstrap gui/$UID <plist>`).
## Tests
* `python3 -m unittest discover -s tests` — 65 tests: autocommit/ff/diverged
* `python3 -m unittest discover -s tests` — 141 tests. The v1.2 additions
cover: a snapshot capturing uncommitted work while HEAD, the index and the
working tree stay byte-identical; snapshots landing outside `refs/heads`;
an unchanged tree not being re-pushed; a diverged folder still being backed
up; a dirty tree blocking the ff-pull but not the backup; retention keeping
everything recent, thinning to hourly then daily, and **keeping**
unparseable timestamps; prune deleting only the thinned refs; `.gitignore`
seeding never overwriting an existing one and keeping `.env` out of
snapshots; the folder-size guard being bounded rather than walking the
disk; mode detection; `list` filtering; `get --all` skipping what is
already present, defaulting to snapshot mode, and shouting about
truncation; `restore` writing a new folder, refusing a non-empty
destination, and leaving the working tree alone.
* Earlier suite: autocommit/ff/diverged
logic against real temp git repos (including "diverged never touches the
remote"), config 0600 handling (including umask-proof creation and a
no-chmod guard), credential-helper quoting/injection, mocked device-flow
@@ -267,6 +396,38 @@ the provisioning service creates their forge account + scoped token on the
fly — the forge never sees a password and the user never sees the forge admin.
Every linked folder becomes a private repo under their account.
## Next phase — invites and per-repo access (designed, not built)
Today `/v1/link` creates an account and every folder becomes a private repo
under it. What is missing is the multi-person case: an existing account
inviting somebody, and that person's device waking up with access to *some*
repos and not others.
Shape this should take, so the next session does not re-litigate it:
* **Where grants live: granthi-link's own store, not Zitadel orgs.** The
estate's house pattern is app-side tenancy tables with the IdP only
providing identity (see the AROS `tenants` / `tenant_members` split).
Grants therefore sit beside `state.json`, and **Gitea is the enforcement
point** — a grant is materialised as a repo collaborator or an org team
membership, so the forge itself refuses unauthorised reads. Nothing in the
client decides access, which is why `get --all` needs no permission logic.
* **Token scope does not change.** `write:repository,write:user` stays; per
repo permission is collaborator/team state, not a token property.
* **`POST /v1/invite`** (account admin → new member): creates the shre-id
user (Zitadel admin PAT, on aros-vps at
`/opt/shre-id/deploy/secrets/shre_id_zitadel_pat`), records the intended
grants, and returns an invite the person redeems by running `link`. Until
they redeem it, nothing exists on the forge.
* **`POST /v1/grants`** (account admin): add/remove repo access for a member;
applies the change to Gitea and records it. Removing a grant must also
remove the collaborator — a grant store that drifts from the forge is
worse than no store.
* Both endpoints are account-admin-only and rate-limited like `/v1/link`.
* The grant store inherits the same fragility already noted for the rate
limiter: a flat JSON file behind an in-process lock, fine for one
`ThreadingHTTPServer` and **not** fine the day this runs multi-process.
## Promotion window (beta → prod)
1. **Expose :3042** behind cloudflared (granthi.shre.ai vhost or