granthi-sync/README.md

granthi-sync v1

The signup → download → link-folders → cloud product spine for the Granthi forge, tested against the BETA forge (granthi-beta.shre.ai). Python 3 stdlib + git CLI only — same portability heritage as the estate's gitea_sync.py mesh.

   ┌──────────────┐   device flow    ┌─────────────────┐
   │ granthi-sync │ ───────────────▶ │ shre-id Zitadel │
   │  (client,    │ ◀─────────────── │  id.shre.ai     │
   │  Mac/laptop) │   access token   └─────────────────┘
   │              │
   │              │  POST /v1/link {zitadel_access_token, device_name}
   │              │ ───────────────▶ ┌───────────────────────────────┐
   │              │ ◀─────────────── │ granthi-link  :3042           │
   │              │  {login, token}  │ (granthi VPS, tailnet-only)   │
   │              │                  │  · userinfo validation        │
   │              │  POST /v1/repos  │  · ensure Gitea user (admin)  │
   │              │ ───────────────▶ │  · mint scoped user token     │
   │              │                  └──────────────┬────────────────┘
   │              │   git push/fetch (user token    │ admin API
   │              │   via credential helper)        ▼
   │              │ ───────────────▶ ┌───────────────────────────────┐
   └──────────────┘                  │ BETA forge  :3041             │
                                     │ granthi-beta.shre.ai          │
                                     └───────────────────────────────┘

Quickstart (invited user)

You need a shre-id account — an operator creates it; there is no open signup (see "Invite-only story"). You also need to be on the tailnet: the provisioning service is not publicly exposed yet.

git clone https://granthi.shre.ai/nirpa/granthi-sync.git
cd granthi-sync

# 1. Link this device. Prints a URL + code; approve it in a browser within
#    5 minutes. Creates your forge account and stores a scoped token in
#    ~/.granthi-sync/config.json (0600). You never see a forge password.
./bin/granthi-sync link

# 2. See what is already yours on the forge.
./bin/granthi-sync list

# 3. Either pull an existing repo down...
./bin/granthi-sync get <repo>           # or <owner>/<repo>, --into DIR

# 3b. ...or push a local folder up. It becomes a private repo.
./bin/granthi-sync add ~/work/notes

# 4. Keep everything synced. Autocommits, ff-pulls, pushes; skips anything
#    that has diverged rather than merging or forcing.
./bin/granthi-sync watch                # --once for a single pass
./bin/granthi-sync status               # what is linked, last sync, divergence

Run watch as a background daemon on macOS with client/launchd/ai.granthi.sync.plist (edit the script path, then launchctl bootstrap gui/$UID <plist>).

If the device code expires (5 minutes, unapproved), nothing is created — no account, no token, no partial state. Just run link again.

Merging is a forge action, not a client one. watch deliberately refuses to merge; when a folder shows DIVERGED in status, resolve it in git or on the forge web UI. The client will never force or auto-merge your work.

Components

server/granthi_link.py — provisioning service (granthi VPS)

• GET /health
• POST /v1/link {zitadel_access_token, device_name} → validates the token against https://id.shre.ai/oidc/v1/userinfo, applies the identity binding rules (below), mints a token scoped write:repository,write:user, returns {gitea_base, login, token, token_name}. POST bodies are capped at 64 KB (413 beyond; missing Content-Length → 411, invalid → 400).
• POST /v1/repos {token, name, private} → creates the user repo with the USER token; clone/html URLs are rebased onto public_gitea_base because the container ROOT_URL (https://granthi-beta.shre.ai) does not resolve for tailnet-only clients.

Deployment: /opt/granthi-link/{granthi_link.py,config.json,state.json} + systemd unit granthi-link.service; binds 127.0.0.1:3042 and 100.111.127.127:3042 (tailnet). Not publicly exposed — see promotion window. The service refuses to start (exit 2) unless config.json is mode 0600/0400 and owned by the user it runs as — the config carries the forge admin password, so permissive perms fail closed, not open.

Rate limiting

Sliding-window, per (route, client), on by default — /v1/link creates accounts and mints tokens, so unlimited has to be a deliberate config act, never an omission. Defaults: /v1/link 5/hour, /v1/repos 60/hour. GET /health is never limited. Over the limit → 429 with a Retry-After header, decided before the body is read so an abusive caller costs nothing.

"rate_limit": {
  "enabled": true,
  "trust_forwarded_for": false,
  "rules": {"/v1/link": [5, 3600], "/v1/repos": [60, 3600]}
}

• A malformed rule refuses startup rather than silently meaning unlimited; [0, N] disables an endpoint outright.
• Denied requests are not recorded, so a client that keeps hammering cannot push its own window forward and lock itself out permanently.
• State is an in-process dict behind a lock — granthi-link is one ThreadingHTTPServer, so that is the entire store. If this ever runs multi-process or multi-host, the limiter must move with it. The key store is capped (MAX_RATE_KEYS); at capacity it drops least-recent windows and logs loudly, which is fail-open, chosen over an unbounded dict that is a memory DoS.
• trust_forwarded_for is off by default. Turn it on only behind cloudflared, where every request otherwise arrives from the tunnel and one abuser would starve everyone. A caller can prepend anything to X-Forwarded-For; a trusted proxy appends the peer it actually saw, so the service reads the last entry, never the first.

Identity binding (state.json)

/v1/link originally bound purely by preferred_username / email local-part — any Zitadel identity whose derived login collided with an existing account got a token for that account (account takeover). The service now persists a map of Zitadel sub → Gitea login in /opt/granthi-link/state.json (0600, atomic tmp+rename writes) and applies:

1. Mapped sub → always the mapped login, regardless of what the current userinfo claims. If the mapped login was deleted from the forge it is re-created only when the service created it originally; adopted accounts are refused (409).
2. Unmapped sub, login free → create the user, record the mapping (created_by_service: true). A concurrent-create 409 from Gitea is handled idempotently: the user is re-fetched and accepted only if its primary email is exactly the one this request would have set.
3. Unmapped sub, login taken → bind ONLY when the Gitea user's primary email equals the Zitadel userinfo email and email_verified is true (recorded with created_by_service: false); anything else → 409 login exists and is not linked to this identity.

A Gitea token is never minted before the binding rule passes, and a corrupt/unreadable state.json fails closed (500) instead of falling back to an empty map.

Migration (pre-1.1.0 accounts): identities whose userinfo carries no verified email (e.g. Zitadel machine users) cannot self-adopt an existing forge login under rule (c) — for them the first link after the upgrade would 409 forever. Any forge account the v1 service created before this change must be seeded into state.json once, as {"<sub>": {"login": "<login>", "created_by_service": true, ...}}, written 0600 atomically. As of the beta rollout the only such account is the E2E machine user granthi-sync-e2e (seeded); the forge's human admin nirpa is never provisioned through /v1/link, so nothing else needed seeding.

Empirically verified mechanics on Gitea 1.27.1, re-probed and still holding on 1.27.2 (beta forge, 2026-08-22 — all three results below matched; the probe minted a token on the granthi-sync-e2e machine user and deleted it again, DELETE …/tokens/{id} returning 204 under basic auth):

• Token minting: POST /api/v1/users/{login}/tokens returns 401 for token-authenticated sudo (both Sudo: header and ?sudo=); it only works with admin basic auth + Sudo: <login> header (201). The config therefore carries admin_login/admin_password (root-only, 0600) in addition to admin_token (minted via gitea admin user generate-access-token, used for all other admin calls).
• User creation: source_id is omitted → local user with a random 30-char password, must_change_password=false, visibility=private. Rationale: /api/v1/admin/identity-auth-sources 404s on 1.27.1; the shre-id OAuth2 source is ID 1 (via gitea admin auth list), but users attached to an OAuth2 source cannot basic-auth and admin-created users get no external_login_user row anyway — first OIDC web login links by email regardless of this choice.
• test_mode (config flag, never in production): allows /v1/link to accept test_userinfo in the body instead of a Zitadel round-trip, so E2E can exercise the ensure-user + mint path headlessly. It is honored only when the service environment also sets GRANTHI_LINK_ALLOW_TEST_MODE=1; a config flag without the env gate is logged as an ERROR and ignored.

client/granthi_sync_client.py (+ bin/granthi-sync) — client daemon

• link [--server URL] [--token TOK] — Zitadel device flow (native app granthi-sync-device, client_id 386909715541590022, project granthi-forge 386906525790109702; grants: device_code + refresh_token): prints the verification URL + user code, polls the token endpoint (authorization_pending/slow_down handled), then calls /v1/link. --token skips the device flow with a ready Zitadel token (headless/dev). Result stored in ~/.granthi-sync/config.json (0600).
• list — every repo the linked token can see, with the local folder each is already synced to. Reads GET /api/v1/user/repos on the forge directly with the scoped user token — no granthi-link round-trip, so the read path needs no service change. Pagination is followed to a short page; if the FORGE_MAX_PAGES guard trips, the output says the list is incomplete rather than letting a bounded page read as the whole set.
• get <repo|owner/repo> [--into DIR] — the download half of add. Clones with --origin granthi (the remote name watch looks for) and -c credential.helper=… (the repo does not exist yet, so the helper cannot be installed first; git also persists it into the new config), then registers the folder in config.json with the same shape add writes — so a cloned repo is picked up by watch immediately. Refuses a non-empty destination. Branch is read with symbolic-ref (an empty repo has an unborn HEAD) and falls back to main.
• add <folder> [--name N] [--private|--public] — git init -b main if needed, creates the cloud repo via /v1/repos, adds remote granthi, initial commit + push. The token is delivered by a git credential helper (the client's hidden git-credential subcommand reading the 0600 config) — never embedded in the remote URL (estate rule).
• watch [--interval 30] [--once] — per folder: autocommit (sync: <ISO ts>) → fetch → ff-pull if remote strictly ahead → push if local strictly ahead. DIVERGED → log + record + SKIP. Never force, never merge — the same policy as the mesh. SIGTERM-clean.
• status — table of linked folders, last sync, divergence flags.
• Run as a daemon on macOS with client/launchd/ai.granthi.sync.plist (edit the script path, then launchctl bootstrap gui/$UID <plist>).

Tests

• python3 -m unittest discover -s tests — 65 tests: autocommit/ff/diverged logic against real temp git repos (including "diverged never touches the remote"), config 0600 handling (including umask-proof creation and a no-chmod guard), credential-helper quoting/injection, mocked device-flow polling, the full /v1/link + /v1/repos service flows against an in-process stub playing Zitadel + Gitea, all identity-binding rules (collision 409, verified-email adoption, deleted-login re-create/refuse, concurrent-create race, corrupt-state fail-closed), the test_mode env gate, config-permission refusal, and the 64 KB body cap. The list/get set covers pagination-to-a-short-page, truncation being reported rather than hidden, HTTP errors being fatal instead of a silent empty list, non-empty-destination refusal, owner-qualified names, unborn-HEAD branch fallback, no token in the remote URL, and — the one that matters — that a get folder is actually picked up by a subsequent sync_folder pass.
• Live E2E against the beta forge is recorded in the delivery notes (link → add → watch ff/push → forced divergence → DIVERGED skip verified via API, remote sha untouched).

Invite-only story (v1)

There is no open signup. An operator invites a user by creating them in shre-id (Zitadel org). The user downloads the client, runs granthi-sync link, signs in at id.shre.ai with the printed device code, and the provisioning service creates their forge account + scoped token on the fly — the forge never sees a password and the user never sees the forge admin. Every linked folder becomes a private repo under their account.

Promotion window (beta → prod)

1. Expose :3042 behind cloudflared (granthi.shre.ai vhost or link.granthi.shre.ai) — today it is tailnet-only by design.
2. Swap forge base URLs in /opt/granthi-link/config.json: gitea_base → prod forge, public_gitea_base → https://granthi.shre.ai; the client default server URL moves to the public endpoint.
3. The granthi-web OIDC app already lists the prod callback; the device app is host-independent. Rotate the beta admin token/password out of the config when pointing at prod (prod forge is READ-ONLY to this estate — promotion is an operator action, not an agent action).
4. Add rate limiting / abuse controls before public exposure. DONE — see "Rate limiting" above (5 links/hour per client by default). When exposing behind cloudflared, set trust_forwarded_for: true in the same change, or every request will look like the tunnel and one abuser will throttle everybody.
5. Hardening checklist (must all hold before exposing):
   • config.json is 0600 (or 0400) and owned by the service user — the service refuses to start otherwise; verify with systemctl status granthi-link after any config edit.
   • test_mode is absent from the production config and GRANTHI_LINK_ALLOW_TEST_MODE is not set in the unit environment.
   • /opt/granthi-link/state.json exists, is 0600, and is included in VPS backups — losing it orphans sub→login bindings (existing users would need verified-email re-adoption).
   • GET /health returns 200 on both binds after restart.
   • Spot-check the identity map: a repeat link for a known sub returns the same login; a colliding username with a different sub gets 409.