granthi-sync/README.md

# granthi-sync v1

The signup → download → link-folders → cloud product spine for the Granthi
forge, tested against the BETA forge (granthi-beta.shre.ai). Python 3 stdlib +
git CLI only — same portability heritage as the estate's `gitea_sync.py` mesh.

```
   ┌──────────────┐   device flow    ┌─────────────────┐
   │ granthi-sync │ ───────────────▶ │ shre-id Zitadel │
   │  (client,    │ ◀─────────────── │  id.shre.ai     │
   │  Mac/laptop) │   access token   └─────────────────┘
   │              │
   │              │  POST /v1/link {zitadel_access_token, device_name}
   │              │ ───────────────▶ ┌───────────────────────────────┐
   │              │ ◀─────────────── │ granthi-link  :3042           │
   │              │  {login, token}  │ (granthi VPS, tailnet-only)   │
   │              │                  │  · userinfo validation        │
   │              │  POST /v1/repos  │  · ensure Gitea user (admin)  │
   │              │ ───────────────▶ │  · mint scoped user token     │
   │              │                  └──────────────┬────────────────┘
   │              │   git push/fetch (user token    │ admin API
   │              │   via credential helper)        ▼
   │              │ ───────────────▶ ┌───────────────────────────────┐
   └──────────────┘                  │ BETA forge  :3041             │
                                     │ granthi-beta.shre.ai          │
                                     └───────────────────────────────┘
```

## Quickstart (invited user)

You need a shre-id account — an operator creates it; there is no open signup
(see "Invite-only story"). You also need to be on the tailnet: the
provisioning service is not publicly exposed yet.

```sh
git clone https://granthi.shre.ai/nirpa/granthi-sync.git
cd granthi-sync

# 1. Link this device. Prints a URL + code; approve it in a browser within
#    5 minutes. Creates your forge account and stores a scoped token in
#    ~/.granthi-sync/config.json (0600). You never see a forge password.
./bin/granthi-sync link

# 2. See what is already yours on the forge.
./bin/granthi-sync list

# 3. Either pull an existing repo down...
./bin/granthi-sync get <repo>           # or <owner>/<repo>, --into DIR

# 3b. ...or push a local folder up. It becomes a private repo.
./bin/granthi-sync add ~/work/notes

# 4. Keep everything synced. Autocommits, ff-pulls, pushes; skips anything
#    that has diverged rather than merging or forcing.
./bin/granthi-sync watch                # --once for a single pass
./bin/granthi-sync status               # what is linked, last sync, divergence
```

Run `watch` as a background daemon on macOS with
`client/launchd/ai.granthi.sync.plist` (edit the script path, then
`launchctl bootstrap gui/$UID <plist>`).

**If the device code expires** (5 minutes, unapproved), nothing is created —
no account, no token, no partial state. Just run `link` again.

**Merging is a forge action, not a client one.** `watch` deliberately refuses
to merge; when a folder shows `DIVERGED` in `status`, resolve it in git or on
the forge web UI. The client will never force or auto-merge your work.

## Components

### `server/granthi_link.py` — provisioning service (granthi VPS)

* `GET /health`
* `POST /v1/link {zitadel_access_token, device_name}` → validates the token
  against `https://id.shre.ai/oidc/v1/userinfo`, applies the **identity
  binding rules** (below), mints a token scoped
  `write:repository,write:user`, returns
  `{gitea_base, login, token, token_name}`. POST bodies are capped at
  **64 KB** (413 beyond; missing `Content-Length` → 411, invalid → 400).
* `POST /v1/repos {token, name, private}` → creates the user repo with the
  USER token; clone/html URLs are rebased onto `public_gitea_base` because the
  container `ROOT_URL` (https://granthi-beta.shre.ai) does not resolve for
  tailnet-only clients.

Deployment: `/opt/granthi-link/{granthi_link.py,config.json,state.json}` +
systemd unit `granthi-link.service`; binds `127.0.0.1:3042` **and**
`100.111.127.127:3042` (tailnet). **Not publicly exposed** — see promotion
window. The service **refuses to start** (exit 2) unless `config.json` is
mode 0600/0400 and owned by the user it runs as — the config carries the
forge admin password, so permissive perms fail closed, not open.

#### Rate limiting

Sliding-window, per `(route, client)`, **on by default** — `/v1/link` creates
accounts and mints tokens, so unlimited has to be a deliberate config act,
never an omission. Defaults: `/v1/link` 5/hour, `/v1/repos` 60/hour. `GET
/health` is never limited. Over the limit → **429** with a `Retry-After`
header, decided *before* the body is read so an abusive caller costs nothing.

```json
"rate_limit": {
  "enabled": true,
  "trust_forwarded_for": false,
  "trusted_proxies": [],
  "rules": {"/v1/link": [5, 3600], "/v1/repos": [60, 3600]}
}
```

* Anything malformed **refuses startup** rather than silently meaning
  unlimited: a bad rule, a non-boolean `enabled`/`trust_forwarded_for` (JSON
  `null`, `0`, or the *string* `"false"` — every non-empty string is truthy),
  or a `rate_limit` that is not an object. `[0, N]` disables an endpoint
  outright.
* Denied requests are *not* recorded, so a client that keeps hammering cannot
  push its own window forward and lock itself out permanently.
* State is an in-process dict behind a lock — granthi-link is one
  `ThreadingHTTPServer`, so that is the entire store. **If this ever runs
  multi-process or multi-host, the limiter must move with it.** The clock is
  read *inside* the lock; taken outside, racing threads append out of order
  and both `Retry-After` and window reclamation silently go wrong.
* The key store is capped (`MAX_RATE_KEYS`) **per route**, not globally. At
  capacity it reclaims expired windows, and if every window is still live it
  **refuses the new key** — fail closed. Evicting a live window would let an
  attacker who can mint many distinct keys clear their *own* limit on demand.
  The per-route budget matters just as much: with one shared table, a flood
  of cheap `/v1/repos` keys would exhaust it and lock brand-new `/v1/link`
  clients out, turning the fail-closed guard into a cross-route DoS.
* Rule values must be real integers. `bool` subclasses `int` in Python, so
  `[5, true]` would otherwise pass as a **1-second** window — an hourly limit
  quietly becoming ~5/sec.
* `trust_forwarded_for` is **off** by default, and turning it on **requires a
  non-empty `trusted_proxies`** — the header is honored only when the socket
  peer is in that list. Without it, anyone reaching the origin directly (it
  also listens on the tailnet) could pick and rotate their own rate-limit key
  just by sending a header. Turn it on when exposing behind cloudflared,
  where every request otherwise arrives from the tunnel and one abuser would
  starve everyone. A caller can *prepend* anything to `X-Forwarded-For`; a
  trusted proxy *appends* the peer it actually saw, so the service reads the
  **last** entry, never the first, and requires it to parse as a real IP.
  `trusted_proxies` is validated at startup: it must be a list (a bare string
  would be iterated character by character), every entry a valid network, and
  wildcards (`0.0.0.0/0`, `::/0`) are refused outright — they would restore
  exactly the "trust anyone" hole the setting exists to close.

#### Identity binding (`state.json`)

`/v1/link` originally bound purely by `preferred_username` / email
local-part — any Zitadel identity whose derived login collided with an
existing account got a token for that account (account takeover). The
service now persists a map of Zitadel `sub` → Gitea login in
`/opt/granthi-link/state.json` (0600, atomic tmp+rename writes) and applies:

1. **Mapped sub** → always the mapped login, regardless of what the current
   userinfo claims. If the mapped login was deleted from the forge it is
   re-created only when the service created it originally; adopted accounts
   are refused (409).
2. **Unmapped sub, login free** → create the user, record the mapping
   (`created_by_service: true`). A concurrent-create 409 from Gitea is
   handled idempotently: the user is re-fetched and accepted only if its
   primary email is exactly the one this request would have set.
3. **Unmapped sub, login taken** → bind ONLY when the Gitea user's primary
   email equals the Zitadel userinfo `email` **and** `email_verified` is
   true (recorded with `created_by_service: false`); anything else →
   `409 login exists and is not linked to this identity`.

A Gitea token is **never minted before the binding rule passes**, and a
corrupt/unreadable `state.json` fails closed (500) instead of falling back
to an empty map.

**Migration (pre-1.1.0 accounts):** identities whose userinfo carries no
verified `email` (e.g. Zitadel *machine* users) cannot self-adopt an
existing forge login under rule (c) — for them the first link after the
upgrade would 409 forever. Any forge account the v1 service created before
this change must be seeded into `state.json` once, as
`{"<sub>": {"login": "<login>", "created_by_service": true, ...}}`, written
0600 atomically. As of the beta rollout the only such account is the E2E
machine user `granthi-sync-e2e` (seeded); the forge's human admin `nirpa`
is never provisioned through `/v1/link`, so nothing else needed seeding.

Empirically verified mechanics on Gitea **1.27.1**, re-probed and still
holding on **1.27.2** (beta forge, 2026-08-22 — all three results below
matched; the probe minted a token on the `granthi-sync-e2e` machine user and
deleted it again, `DELETE …/tokens/{id}` returning 204 under basic auth):

* Token minting: `POST /api/v1/users/{login}/tokens` returns **401 for
  token-authenticated sudo** (both `Sudo:` header and `?sudo=`); it only works
  with **admin basic auth + `Sudo: <login>` header** (201). The config
  therefore carries `admin_login`/`admin_password` (root-only, 0600) in
  addition to `admin_token` (minted via
  `gitea admin user generate-access-token`, used for all other admin calls).
* User creation: `source_id` is omitted → **local user** with a random
  30-char password, `must_change_password=false`, `visibility=private`.
  Rationale: `/api/v1/admin/identity-auth-sources` 404s on 1.27.1; the
  shre-id OAuth2 source is ID 1 (via `gitea admin auth list`), but users
  attached to an OAuth2 source cannot basic-auth and admin-created users get
  no `external_login_user` row anyway — first OIDC web login links by email
  regardless of this choice.
* `test_mode` (config flag, **never in production**): allows `/v1/link` to
  accept `test_userinfo` in the body instead of a Zitadel round-trip, so E2E
  can exercise the ensure-user + mint path headlessly. It is honored **only
  when the service environment also sets `GRANTHI_LINK_ALLOW_TEST_MODE=1`**;
  a config flag without the env gate is logged as an ERROR and ignored.

### `client/granthi_sync_client.py` (+ `bin/granthi-sync`) — client daemon

* `link [--server URL] [--token TOK]` — Zitadel **device flow** (native app
  `granthi-sync-device`, client_id `386909715541590022`, project
  granthi-forge `386906525790109702`; grants: device_code + refresh_token):
  prints the verification URL + user code, polls the token endpoint
  (`authorization_pending`/`slow_down` handled), then calls `/v1/link`.
  `--token` skips the device flow with a ready Zitadel token (headless/dev).
  Result stored in `~/.granthi-sync/config.json` (0600).
* `list` — every repo the linked token can see, with the local folder each
  is already synced to. Reads `GET /api/v1/user/repos` on the forge
  **directly** with the scoped user token — no granthi-link round-trip, so
  the read path needs no service change. Pagination is followed to a short
  page; if the `FORGE_MAX_PAGES` guard trips, the output says the list is
  incomplete rather than letting a bounded page read as the whole set.
* `get <repo|owner/repo> [--into DIR]` — the download half of `add`. Clones
  with `--origin granthi` (the remote name `watch` looks for) and
  `-c credential.helper=…` (the repo does not exist yet, so the helper
  cannot be installed first; git also persists it into the new config), then
  registers the folder in `config.json` with the same shape `add` writes —
  so a cloned repo is picked up by `watch` immediately. Refuses a non-empty
  destination. Branch is read with `symbolic-ref` (an empty repo has an
  unborn HEAD) and falls back to `main`.
* `add <folder> [--name N] [--private|--public]` — `git init -b main` if
  needed, creates the cloud repo via `/v1/repos`, adds remote `granthi`,
  initial commit + push. The token is delivered by a **git credential
  helper** (the client's hidden `git-credential` subcommand reading the 0600
  config) — never embedded in the remote URL (estate rule).
* `watch [--interval 30] [--once]` — per folder: autocommit
  (`sync: <ISO ts>`) → fetch → ff-pull if remote strictly ahead → push if
  local strictly ahead. **DIVERGED → log + record + SKIP. Never force, never
  merge** — the same policy as the mesh. SIGTERM-clean.
* `status` — table of linked folders, last sync, divergence flags.
* Run as a daemon on macOS with `client/launchd/ai.granthi.sync.plist`
  (edit the script path, then `launchctl bootstrap gui/$UID <plist>`).

## Tests

* `python3 -m unittest discover -s tests` — 65 tests: autocommit/ff/diverged
  logic against real temp git repos (including "diverged never touches the
  remote"), config 0600 handling (including umask-proof creation and a
  no-chmod guard), credential-helper quoting/injection, mocked device-flow
  polling, the full `/v1/link` + `/v1/repos` service flows against an
  in-process stub playing Zitadel + Gitea, all identity-binding rules
  (collision 409, verified-email adoption, deleted-login re-create/refuse,
  concurrent-create race, corrupt-state fail-closed), the test_mode env
  gate, config-permission refusal, and the 64 KB body cap. The `list`/`get`
  set covers pagination-to-a-short-page, truncation being reported rather
  than hidden, HTTP errors being fatal instead of a silent empty list,
  non-empty-destination refusal, owner-qualified names, unborn-HEAD branch
  fallback, no token in the remote URL, and — the one that matters — that a
  `get` folder is actually picked up by a subsequent `sync_folder` pass.
* Live E2E against the beta forge is recorded in the delivery notes
  (link → add → watch ff/push → forced divergence → DIVERGED skip verified
  via API, remote sha untouched).

## Invite-only story (v1)

There is no open signup. An operator invites a user by creating them in
shre-id (Zitadel org). The user downloads the client, runs
`granthi-sync link`, signs in at id.shre.ai with the printed device code, and
the provisioning service creates their forge account + scoped token on the
fly — the forge never sees a password and the user never sees the forge admin.
Every linked folder becomes a private repo under their account.

## Promotion window (beta → prod)

1. **Expose :3042** behind cloudflared (granthi.shre.ai vhost or
   link.granthi.shre.ai) — today it is tailnet-only by design.
2. **Swap forge base URLs** in `/opt/granthi-link/config.json`:
   `gitea_base` → prod forge, `public_gitea_base` →
   `https://granthi.shre.ai`; the client default server URL moves to the
   public endpoint.
3. The `granthi-web` OIDC app already lists the prod callback; the device
   app is host-independent. Rotate the beta admin token/password out of the
   config when pointing at prod (prod forge is READ-ONLY to this estate —
   promotion is an operator action, not an agent action).
4. ~~Add rate limiting / abuse controls before public exposure.~~ **DONE** —
   see "Rate limiting" above (5 links/hour per client by default). When
   exposing behind cloudflared, set `trust_forwarded_for: true` in the same
   change, or every request will look like the tunnel and one abuser will
   throttle everybody.
5. **Hardening checklist (must all hold before exposing):**
   - [ ] `config.json` is 0600 (or 0400) and owned by the service user —
         the service refuses to start otherwise; verify with
         `systemctl status granthi-link` after any config edit.
   - [ ] `test_mode` is absent from the production config **and**
         `GRANTHI_LINK_ALLOW_TEST_MODE` is not set in the unit environment.
   - [ ] `/opt/granthi-link/state.json` exists, is 0600, and is included in
         VPS backups — losing it orphans sub→login bindings (existing users
         would need verified-email re-adoption).
   - [ ] `GET /health` returns 200 on both binds after restart.
   - [ ] Spot-check the identity map: a repeat link for a known sub returns
         the same login; a colliding username with a different sub gets 409.

<!-- review-service live check 20260819T131430Z -->