nirpa/granthi-sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(client): snapshot backups, restore points, scoped bulk pull

Browse Source 
Two modes per linked folder. 'mirror' keeps today's behaviour for a plain
folder that add turned into a repo. 'snapshot' is new and is for a folder
that already had a git history: nothing is ever committed on the user's
behalf, and instead each pass builds a commit object from the working tree
via a scratch index + commit-tree and pushes it to
refs/granthi-backup/<device>/<ts>. HEAD, the index and every file stay
exactly as the user left them, so uncommitted, unmerged, half-finished work
leaves the machine with a timestamp to restore from.

Verified on the beta forge (Gitea 1.27.2) that a custom ref namespace is
accepted, readable via ls-remote, and absent from the branch list.

Also: retention (all for 24h, hourly for 7d, daily beyond; unparseable
timestamps kept), snapshots/restore commands, restore never writing over the
working tree, get --all bounded by what the forge grants, list <pattern>,
.gitignore seeding, an add size guard, and a persisted device_id.

141 tests (was 108).
This commit is contained in:
claude
2026-08-23 11:22:20 -04:00
parent 1091aa61f2
commit 9e3201a296
3 changed files with 1275 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
client/granthi_sync_client.py
+661 -40
View File
@@ -7,32 +7,62 @@ Commands:
the headless path), then POST /v1/link on the granthi-link service.
Stores {server, gitea_base, login, token, token_name} in
~/.granthi-sync/config.json (0600).
list
Table of every repo the linked token can see on the forge, with the
local folder each one is already synced to (if any).
get <repo|owner/repo> [--into DIR]
list [pattern]
Table of every repo the linked token can see on the forge -- which
is exactly the set the forge grants this account, never a
client-side filter over someone else's repos -- with the local
folder each one is already synced to (if any). `pattern` narrows
the table by name (substring, or a glob like 'work-*').
get <repo|owner/repo> [--into DIR] [--mode M] | --all [--into DIR]
Clone a forge repo and register it for `watch` -- the download half
of `add`. Remote is named 'granthi' at clone time and the token is
supplied by the credential helper, never embedded in the URL.
add <folder> [--name N] [--private/--public]
`--all` pulls every repo this account can see, skipping the ones
already linked on this machine.
add <folder> [--name N] [--private/--public] [--mode M] [--force]
git init (branch main) if needed, create the cloud repo via
/v1/repos, add remote 'granthi', initial commit + push. The token
is supplied via a git credential helper (this script's hidden
`git-credential` subcommand), never embedded in the remote URL.
Seeds a starter .gitignore and refuses an unexpectedly large folder
unless --force.
watch [--interval 30] [--once]
Poll linked folders: autocommit local changes, fetch, ff-pull when
the remote is strictly ahead, push when local is strictly ahead.
DIVERGED branches are logged + recorded and SKIPPED -- never force,
never merge (same policy as the estate's gitea_sync.py mesh).
SIGTERM-clean.
Poll linked folders. Per folder, by MODE (below): `mirror`
autocommits local changes and pushes them; `snapshot` never commits
for you and instead mirrors the working tree to a per-device backup
ref. Both ff-pull when the remote is strictly ahead. DIVERGED
branches are logged + recorded and SKIPPED -- never force, never
merge (same policy as the estate's gitea_sync.py mesh) -- but a
diverged folder is still backed up. SIGTERM-clean.
snapshots <folder>
List the restore points held for a folder, newest first.
restore <folder> --at <TIMESTAMP|SHA> [--into DIR]
Materialise one restore point into a NEW directory. Never writes
over the working tree.
status
Table of linked folders, last sync, divergence flags.
Table of linked folders, mode, last sync, divergence flags.
MODES
mirror The folder IS the repo. Local changes are autocommitted as
`sync: <ISO ts>` and pushed; every commit is a restore point.
Chosen automatically for a plain folder that `add` turned into
a repo -- the "my documents live in the cloud" case.
snapshot The folder has a real git history that belongs to the user.
NOTHING is committed on their behalf. Each pass builds a commit
object from the working tree without touching HEAD, the index
or any file, and pushes it to
refs/granthi-backup/<device>/<ts>. Uncommitted, unmerged,
half-finished work is therefore off the machine with a
timestamp to restore from, while the user's own history stays
exactly as they left it. Chosen automatically for a folder that
was already a git repo, and for anything `get` clones.
Stdlib + git CLI only.
"""
import argparse
import base64
import fnmatch
import json
import os
import re
@@ -40,13 +70,15 @@ import shlex
import signal
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timezone
import uuid
from datetime import datetime, timedelta, timezone
VERSION = "1.1.0"
VERSION = "1.2.0"
CONFIG_DIR = os.path.expanduser(os.environ.get("GRANTHI_SYNC_HOME", "~/.granthi-sync"))
CONFIG_PATH = os.path.join(CONFIG_DIR, "config.json")
@@ -61,6 +93,50 @@ DEVICE_SCOPE = "openid profile email"
FORGE_PAGE_LIMIT = 50
FORGE_MAX_PAGES = 40 # 2000 repos; a guard against an unbounded paging loop
# Backup refs live OUTSIDE refs/heads on purpose: verified on the beta forge
# (Gitea 1.27.2) that a custom namespace is accepted, is readable through
# ls-remote, and does NOT appear in the branch list or the branch dropdown.
# Under refs/heads a machine taking a snapshot every 30s would bury the
# user's real branches.
BACKUP_NS = "refs/granthi-backup"
SNAPSHOT_TS_FMT = "%Y%m%dT%H%M%SZ"
# Retention. Unbounded snapshots are a disk leak with no way to use them, so
# thin them by age: everything recent, then hourly, then daily.
KEEP_ALL_HOURS = 24
KEEP_HOURLY_DAYS = 7
PRUNE_EVERY_SECONDS = 3600
# `add` guard rails. A plain `git add -A` on the wrong folder commits every
# byte in it, so refuse the surprising cases unless the user insists.
ADD_MAX_BYTES = 512 * 1024 * 1024
ADD_MAX_FILES = 20000
MARKER_FILE = ".granthi-sync.json"
# Seeded only when the folder has no .gitignore of its own. The point is the
# secret-shaped entries: the rest is noise reduction.
DEFAULT_GITIGNORE = """\
# Seeded by granthi-sync. Edit freely -- it is yours now.
.DS_Store
Thumbs.db
*.swp
node_modules/
__pycache__/
.venv/
venv/
# Never sync credentials.
.env
.env.*
*.key
*.pem
*.p12
id_rsa
id_ed25519
credentials.json
secrets.json
"""
def log(msg):
ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@@ -109,6 +185,21 @@ def load_config():
return {}
def device_id(cfg):
"""Stable per-installation id, minted once and persisted.
The hostname is neither stable (people rename laptops) nor unique (every
fresh mac is 'Mac mini'), so it cannot key a backup ref or a device
registry -- two machines would write each other's snapshots. Callers get
the id from here; `save_config` is theirs to call.
"""
existing = cfg.get("device_id")
if existing:
return existing
cfg["device_id"] = uuid.uuid4().hex
return cfg["device_id"]
def save_config(cfg):
os.makedirs(CONFIG_DIR, mode=0o700, exist_ok=True)
tmp = CONFIG_PATH + ".tmp"
@@ -125,10 +216,14 @@ def save_config(cfg):
# git helpers
# --------------------------------------------------------------------------
def git(folder, *args, check=True):
def git(folder, *args, check=True, env=None):
"""Run git in folder. Returns (rc, stdout). Never uses --force."""
run_env = None
if env:
run_env = dict(os.environ)
run_env.update(env)
proc = subprocess.run(["git", "-C", folder] + list(args),
capture_output=True, text=True)
capture_output=True, text=True, env=run_env)
if check and proc.returncode != 0:
raise RuntimeError(
f"git {' '.join(args)} failed in {folder}: {proc.stderr.strip()}")
@@ -156,6 +251,181 @@ def autocommit(folder):
return True
# --------------------------------------------------------------------------
# snapshots -- back up work that was never committed
# --------------------------------------------------------------------------
def _head_sha(folder):
rc, sha = git(folder, "rev-parse", "--verify", "--quiet", "HEAD",
check=False)
return sha.strip() if rc == 0 and sha.strip() else None
def build_snapshot(folder):
"""Commit the working tree WITHOUT touching HEAD, the index or a file.
A scratch index is loaded from HEAD, `add -A` stages the working tree
into *that* index, and commit-tree writes a commit object parented on
HEAD. The user's own index and history are untouched -- they can be
mid-rebase with a dirty tree and this still records what is on disk.
Returns (commit_sha, tree_sha) or None when the tree is empty.
"""
head = _head_sha(folder)
git_dir = git(folder, "rev-parse", "--absolute-git-dir")[1].strip()
fd, tmp_index = tempfile.mkstemp(prefix="granthi-index-", dir=git_dir)
os.close(fd)
os.unlink(tmp_index) # git wants to create it itself
env = {"GIT_INDEX_FILE": tmp_index}
try:
if head:
git(folder, "read-tree", head, env=env)
# Honours .gitignore, so a seeded ignore file keeps secrets out of
# snapshots exactly as it keeps them out of commits.
git(folder, "add", "-A", env=env)
_, tree = git(folder, "write-tree", env=env)
tree = tree.strip()
if not tree:
return None
if head:
_, head_tree = git(folder, "rev-parse", f"{head}^{{tree}}")
if head_tree.strip() == tree:
# Nothing uncommitted: HEAD already holds this exact content.
return None
ts = datetime.now(timezone.utc).isoformat(timespec="seconds")
args = ["commit-tree", tree, "-m", f"granthi snapshot: {ts}"]
if head:
args += ["-p", head]
# Snapshots are parented on HEAD and nothing else -- deliberately NOT
# chained to the previous snapshot. Chaining would keep every old
# snapshot reachable from the newest one, so pruning a ref would free
# nothing and retention would be decorative.
_, commit = git(folder, *args,
env={"GIT_AUTHOR_NAME": "granthi-sync",
"GIT_AUTHOR_EMAIL": "[email protected]",
"GIT_COMMITTER_NAME": "granthi-sync",
"GIT_COMMITTER_EMAIL": "[email protected]"})
return commit.strip(), tree
finally:
if os.path.exists(tmp_index):
os.unlink(tmp_index)
def snapshot_ref(dev, ts=None):
ts = ts or datetime.now(timezone.utc).strftime(SNAPSHOT_TS_FMT)
return f"{BACKUP_NS}/{dev}/{ts}", ts
def push_snapshot(folder, dev, remote="granthi"):
"""Take a snapshot and push it to this device's backup namespace.
Returns the ref pushed, or None when there was nothing new to back up.
"""
built = build_snapshot(folder)
if not built:
return None
commit, tree = built
if tree == _last_snapshot_tree(folder, dev):
return None # working tree unchanged since the last backup
ref, _ = snapshot_ref(dev)
git(folder, "push", remote, f"{commit}:{ref}")
_remember_snapshot_tree(folder, dev, tree)
return ref
def _tree_marker_path(folder, dev):
git_dir = git(folder, "rev-parse", "--absolute-git-dir")[1].strip()
return os.path.join(git_dir, f"granthi-last-snapshot-{dev}")
def _last_snapshot_tree(folder, dev):
try:
with open(_tree_marker_path(folder, dev)) as f:
return f.read().strip()
except OSError:
return None
def _remember_snapshot_tree(folder, dev, tree):
try:
with open(_tree_marker_path(folder, dev), "w") as f:
f.write(tree)
except OSError:
pass # a lost marker costs one redundant push, never correctness
def list_snapshots(folder, dev, remote="granthi"):
"""Backup refs for this device on the remote, newest first.
Read from the REMOTE, not a local cache: the point of the feature is
surviving the loss of this machine, so what the server holds is the
answer that matters.
"""
rc, out = git(folder, "ls-remote", remote, f"{BACKUP_NS}/{dev}/*",
check=False)
if rc != 0:
return []
snaps = []
for line in out.splitlines():
sha, _, ref = line.partition("\t")
ref = ref.strip()
if not ref.startswith(f"{BACKUP_NS}/{dev}/"):
continue
snaps.append({"sha": sha.strip(), "ref": ref,
"ts": ref.rsplit("/", 1)[-1]})
return sorted(snaps, key=lambda s: s["ts"], reverse=True)
def _parse_ts(ts):
try:
return datetime.strptime(ts, SNAPSHOT_TS_FMT).replace(tzinfo=timezone.utc)
except ValueError:
return None
def snapshots_to_prune(snaps, now=None):
"""Retention: keep everything for KEEP_ALL_HOURS, then one per hour for
KEEP_HOURLY_DAYS, then one per day. Returns the refs to delete.
Unparseable timestamps are KEPT. A ref this version does not understand
is not evidence it is worthless, and deleting the unrecognised is how a
backup system loses the one thing someone needed.
"""
now = now or datetime.now(timezone.utc)
keep, drop, seen = [], [], set()
for snap in sorted(snaps, key=lambda s: s["ts"], reverse=True):
when = _parse_ts(snap["ts"])
if when is None:
keep.append(snap)
continue
age = now - when
if age <= timedelta(hours=KEEP_ALL_HOURS):
keep.append(snap)
continue
bucket = (when.strftime("%Y%m%dT%H")
if age <= timedelta(days=KEEP_HOURLY_DAYS)
else when.strftime("%Y%m%d"))
if bucket in seen:
drop.append(snap)
else:
seen.add(bucket)
keep.append(snap)
return [s["ref"] for s in drop]
def prune_snapshots(folder, dev, remote="granthi", now=None):
"""Delete thinned-out backup refs on the remote. Returns how many went."""
stale = snapshots_to_prune(list_snapshots(folder, dev, remote), now=now)
if not stale:
return 0
# Deleting a ref is not force-pushing over anyone's work: these refs are
# written by this device alone and hold no history but its own.
for batch_start in range(0, len(stale), 50):
batch = stale[batch_start:batch_start + 50]
git(folder, "push", remote, *[f":{ref}" for ref in batch], check=False)
return len(stale)
def sync_state(folder, remote="granthi", branch="main"):
"""Classify local vs remote after a fetch.
@@ -182,30 +452,60 @@ def sync_state(folder, remote="granthi", branch="main"):
return "diverged"
def sync_folder(folder, remote="granthi", branch="main"):
def worktree_dirty(folder):
_, status = git(folder, "status", "--porcelain")
return bool(status)
def sync_folder(folder, remote="granthi", branch="main", mode="mirror",
dev=None):
"""One sync pass for one folder. Returns (outcome, detail).
outcome in {'synced', 'pushed', 'clean', 'diverged', 'error'}.
Policy: ff-only pulls, plain pushes. NEVER --force, NEVER merge.
In `snapshot` mode nothing is committed on the user's behalf; their
uncommitted work is pushed to a backup ref instead, and that happens on
EVERY outcome -- including diverged and including a tree too dirty to
ff-pull. Those are exactly the moments work is most at risk, so they are
the last moments to skip a backup.
"""
try:
committed = autocommit(folder)
backed_up = None
committed = False
if mode == "mirror":
committed = autocommit(folder)
git(folder, "fetch", remote)
if mode == "snapshot" and dev:
backed_up = push_snapshot(folder, dev, remote)
def note(base):
if backed_up:
return f"{base}; backed up {backed_up.rsplit('/', 1)[-1]}"
return base
state = sync_state(folder, remote, branch)
ff_pulled = False
if state == "remote-ahead-ff":
if mode == "snapshot" and worktree_dirty(folder):
# A ff-pull would fail (or clobber) with local edits present.
# The edits are already safe on the backup ref, so report and
# leave the working tree exactly as the user left it.
return "clean", note("remote ahead; local edits present, "
"not pulling")
git(folder, "merge", "--ff-only", f"refs/remotes/{remote}/{branch}")
ff_pulled = True
state = sync_state(folder, remote, branch)
if state == "diverged":
return "diverged", "local and remote both advanced; skipping (no force, no merge)"
return "diverged", note("local and remote both advanced; "
"skipping (no force, no merge)")
if state in ("local-ahead", "no-remote-branch"):
git(folder, "push", "-u", remote, branch)
return "pushed", "committed+pushed" if committed else "pushed"
return "pushed", note("committed+pushed" if committed else "pushed")
# state == "in-sync"
if ff_pulled:
return "synced", "ff-pulled"
return "clean", "in-sync"
return "synced", note("ff-pulled")
return "clean", note("in-sync")
except RuntimeError as e:
return "error", str(e)
@@ -283,13 +583,19 @@ def device_flow():
def cmd_link(args):
zitadel_token = args.token or device_flow()
cfg = load_config()
dev = device_id(cfg)
status, resp = http_json(
"POST", f"{args.server.rstrip('/')}/v1/link",
body={"zitadel_access_token": zitadel_token,
"device_name": args.device})
"device_name": args.device,
# Sent so the service can keep a device registry (and so a
# device can later be listed or revoked by something other
# than a hostname). A service that does not know the field
# ignores it, so the client can lead.
"device_id": dev})
if status != 200:
raise SystemExit(f"link failed (HTTP {status}): {resp}")
cfg = load_config()
cfg.update({"server": args.server.rstrip("/"),
"gitea_base": resp["gitea_base"],
"login": resp["login"],
@@ -298,7 +604,8 @@ def cmd_link(args):
cfg.setdefault("folders", {})
save_config(cfg)
log(f"linked as {resp['login']} on {resp['gitea_base']} "
f"(token {resp['token_name']}); config: {CONFIG_PATH}")
f"(token {resp['token_name']}, device {dev[:8]}); "
f"config: {CONFIG_PATH}")
return 0
@@ -346,9 +653,35 @@ def list_repos(cfg):
return repos, bool(fetch(FORGE_MAX_PAGES + 1))
def match_repo(repo, pattern):
"""Case-insensitive: glob if the pattern has glob syntax, else substring.
Matched against full_name AND the bare name, so 'notes' finds
'alice/notes' without anyone having to think about owners.
"""
if not pattern:
return True
full = (repo.get("full_name") or "").lower()
name = (repo.get("name") or full.rsplit("/", 1)[-1]).lower()
pat = pattern.lower()
if any(c in pat for c in "*?["):
return fnmatch.fnmatch(full, pat) or fnmatch.fnmatch(name, pat)
return pat in full or pat in name
def cmd_list(args):
cfg = require_linked(load_config())
repos, truncated = list_repos(cfg)
pattern = getattr(args, "pattern", None)
if pattern:
# Filter for display only. The set came from the forge under this
# account's own token, so it is already the permitted set -- this
# narrows what is shown, it does not widen what is reachable.
repos = [r for r in repos if match_repo(r, pattern)]
if not repos:
print(f"no repos matching {pattern!r}"
+ (" (and the listing was truncated)" if truncated else ""))
return 0
if not repos:
print("no repos on the forge for this account")
return 0
@@ -396,11 +729,25 @@ def parse_repo_arg(repo, login):
return "/".join(parts)
def cmd_get(args):
cfg = require_linked(load_config())
full_name = parse_repo_arg(args.repo, cfg["login"])
def read_marker(folder):
"""Intent `add` left in the repo, if any: {'mode': ...}.
Carrying the mode in the repo is what makes a plain synced folder behave
the same on the second machine. Without it, `get` would have to guess,
and guessing 'mirror' on someone's real project is the destructive
direction.
"""
try:
with open(os.path.join(folder, MARKER_FILE)) as f:
data = json.load(f)
return data if isinstance(data, dict) else {}
except (OSError, ValueError):
return {}
def clone_one(cfg, full_name, dest, mode=None):
"""Clone one repo and register it. Returns the folder meta written."""
name = full_name.rsplit("/", 1)[-1]
dest = os.path.abspath(args.into or name)
if os.path.exists(dest) and os.listdir(dest):
raise SystemExit(f"refusing to clone into a non-empty path: {dest}")
clone_url = f"{cfg['gitea_base'].rstrip('/')}/{full_name}.git"
@@ -420,23 +767,151 @@ def cmd_get(args):
rc, branch = git(dest, "symbolic-ref", "--short", "HEAD", check=False)
if rc != 0 or not branch:
branch = "main"
cfg.setdefault("folders", {})[dest] = {
# Default to `snapshot`: a cloned repo carries history that is the
# user's, and autocommitting onto it is the one mistake that cannot be
# undone quietly. `add` marks the folders that genuinely want mirroring.
chosen = mode or read_marker(dest).get("mode") or "snapshot"
meta = {
"name": name, "full_name": full_name, "branch": branch,
"mode": chosen,
"last_sync": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"diverged": False}
cfg.setdefault("folders", {})[dest] = meta
save_config(cfg)
log(f"cloned {full_name} -> {dest} (branch {branch}); "
log(f"cloned {full_name} -> {dest} (branch {branch}, mode {chosen}); "
f"`granthi-sync watch` will keep it synced")
return meta
def cmd_get(args):
cfg = require_linked(load_config())
device_id(cfg)
if getattr(args, "all", False):
return _get_all(cfg, args)
if not args.repo:
raise SystemExit("give a repo name, or --all")
full_name = parse_repo_arg(args.repo, cfg["login"])
name = full_name.rsplit("/", 1)[-1]
clone_one(cfg, full_name, os.path.abspath(args.into or name), args.mode)
return 0
def _get_all(cfg, args):
"""Pull every repo this account can see.
'Only the repos they are granted' needs no client-side permission logic:
/api/v1/user/repos is evaluated by the forge against this account's own
scoped token, so the list IS the grant. Anything else would be a second
opinion about someone else's authorisation.
"""
repos, truncated = list_repos(cfg)
base = os.path.abspath(args.into or ".")
linked = set(cfg.get("folders", {}))
cloned = skipped = failed = 0
for repo in sorted(repos, key=lambda r: r.get("full_name") or ""):
full_name = repo.get("full_name")
if not full_name:
continue
dest = os.path.join(base, full_name.rsplit("/", 1)[-1])
if dest in linked or (os.path.exists(dest) and os.listdir(dest)):
log(f"skip {full_name}: already present at {dest}")
skipped += 1
continue
try:
clone_one(cfg, full_name, dest, args.mode)
cloned += 1
except SystemExit as e: # one bad repo must not abandon the rest
log(f"FAILED {full_name}: {e}")
failed += 1
log(f"get --all: {cloned} cloned, {skipped} already here, {failed} failed")
if truncated:
log(f"WARNING: the forge listing stopped after {FORGE_MAX_PAGES} "
f"pages of {FORGE_PAGE_LIMIT} -- this was NOT every repo you "
f"have access to.")
return 1 if failed else 0
def measure_folder(folder, max_files=ADD_MAX_FILES):
"""(files, bytes) below .git, stopping once max_files is exceeded.
Bounded on purpose: the caller only needs to know whether the folder is
surprisingly big, and walking a 2 TB drive to answer that would be the
same mistake as syncing it.
"""
files = total = 0
for root, dirs, names in os.walk(folder):
dirs[:] = [d for d in dirs if d != ".git"]
for n in names:
path = os.path.join(root, n)
try:
if os.path.islink(path):
continue
total += os.path.getsize(path)
except OSError:
continue
files += 1
if files > max_files:
return files, total
return files, total
def seed_gitignore(folder):
"""Write a starter .gitignore when the folder has none. Returns bool.
`add -A` takes everything it is not told to leave, and the folders
people sync are exactly the ones holding a stray .env or id_rsa. An
existing .gitignore is never touched -- it is the user's.
"""
path = os.path.join(folder, ".gitignore")
if os.path.exists(path):
return False
with open(path, "w") as f:
f.write(DEFAULT_GITIGNORE)
return True
def detect_mode(folder, had_git):
"""`mirror` for a plain folder we turned into a repo; `snapshot` for one
that already had a history worth protecting."""
if not had_git:
return "mirror"
return "snapshot" if _head_sha(folder) else "mirror"
def write_marker(folder, mode):
with open(os.path.join(folder, MARKER_FILE), "w") as f:
json.dump({"mode": mode, "created_by": f"granthi-sync/{VERSION}"},
f, indent=2)
f.write("\n")
def cmd_add(args):
cfg = require_linked(load_config())
folder = os.path.abspath(args.folder)
if not os.path.isdir(folder):
raise SystemExit(f"no such folder: {folder}")
name = args.name or re.sub(r"[^a-zA-Z0-9._-]+", "-", os.path.basename(folder))
had_git = os.path.isdir(os.path.join(folder, ".git"))
if not args.force:
files, size = measure_folder(folder)
if files > ADD_MAX_FILES or size > ADD_MAX_BYTES:
raise SystemExit(
f"{folder} holds {files} files / {size // (1024 * 1024)} MB, "
f"over the {ADD_MAX_FILES} file / "
f"{ADD_MAX_BYTES // (1024 * 1024)} MB guard.\n"
f"Add a .gitignore for what should not sync, point at a "
f"narrower folder, or re-run with --force if this is really "
f"what you want.")
ensure_repo(folder)
mode = args.mode or detect_mode(folder, had_git)
if seed_gitignore(folder):
log(f"seeded a starter .gitignore in {folder} -- review it; "
f"anything listed there will NOT be synced")
if mode == "mirror":
# Only mirror folders carry the marker: it tells the next machine's
# `get` that autocommitting here is wanted. A snapshot folder is
# someone's real project and gets no file dropped into it.
write_marker(folder, mode)
install_credential_helper(folder)
status, resp = http_json(
"POST", f"{cfg['server']}/v1/repos",
@@ -453,14 +928,30 @@ def cmd_add(args):
git(folder, "remote", "set-url", "granthi", clone_url)
else:
git(folder, "remote", "add", "granthi", clone_url)
autocommit(folder)
git(folder, "push", "-u", "granthi", "main")
# An existing repo may sit on master, or on a feature branch. Pushing
# its work to a hardcoded 'main' would publish the wrong thing under the
# right name.
rc, branch = git(folder, "symbolic-ref", "--short", "HEAD", check=False)
branch = branch.strip() if rc == 0 and branch.strip() else "main"
if mode == "mirror":
autocommit(folder)
git(folder, "push", "-u", "granthi", branch)
dev = device_id(cfg)
if mode == "snapshot":
ref = push_snapshot(folder, dev)
if ref:
log(f"backed up uncommitted work to {ref}")
cfg.setdefault("folders", {})[folder] = {
"name": name, "full_name": f"{cfg['login']}/{name}", "branch": "main",
"name": name, "full_name": f"{cfg['login']}/{name}", "branch": branch,
"mode": mode,
"last_sync": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"diverged": False}
save_config(cfg)
log(f"linked folder {folder} -> {clone_url}")
log(f"linked folder {folder} -> {clone_url} (mode {mode}, branch {branch})")
if mode == "snapshot":
log("mode snapshot: your commits stay yours -- granthi-sync will "
"never commit for you here, and backs up uncommitted work to "
f"{BACKUP_NS}/{dev[:8]}…/<timestamp>")
return 0
@@ -473,13 +964,23 @@ def _sigterm(signum, frame):
log(f"signal {signum} received; finishing current pass then exiting")
def watch_pass():
_LAST_PRUNE = 0.0
def watch_pass(now=None):
global _LAST_PRUNE
cfg = load_config()
dev = device_id(cfg)
# Pruning talks to the remote for every folder, so it runs on a slow
# clock of its own rather than on every 30s pass.
due = (now or time.time()) - _LAST_PRUNE >= PRUNE_EVERY_SECONDS
for folder, meta in sorted(cfg.get("folders", {}).items()):
if not os.path.isdir(folder):
log(f"SKIP {folder}: missing")
continue
outcome, detail = sync_folder(folder, branch=meta.get("branch", "main"))
mode = meta.get("mode", "mirror")
outcome, detail = sync_folder(folder, branch=meta.get("branch", "main"),
mode=mode, dev=dev)
meta["diverged"] = outcome == "diverged"
if outcome in ("pushed", "synced"):
meta["last_sync"] = datetime.now(timezone.utc).isoformat(timespec="seconds")
@@ -489,6 +990,16 @@ def watch_pass():
log(f"ERROR {folder}: {detail}")
else:
log(f"{outcome} {folder}: {detail}")
if due and mode == "snapshot" and outcome != "error":
try:
gone = prune_snapshots(folder, dev)
if gone:
log(f"pruned {gone} old restore points in {folder}")
except RuntimeError as e:
# Retention failing must never stop the backup itself.
log(f"prune failed in {folder} (backups unaffected): {e}")
if due:
_LAST_PRUNE = now or time.time()
save_config(cfg)
@@ -509,6 +1020,88 @@ def cmd_watch(args):
return 0
def _folder_meta(cfg, folder):
path = os.path.abspath(folder)
meta = cfg.get("folders", {}).get(path)
if not meta:
raise SystemExit(f"{path} is not a linked folder "
f"(see: granthi-sync status)")
return path, meta
def cmd_snapshots(args):
"""Restore points, newest first. Both modes answer the same question."""
cfg = require_linked(load_config())
folder, meta = _folder_meta(cfg, args.folder)
dev = device_id(cfg)
if meta.get("mode", "mirror") == "mirror":
rc, out = git(folder, "log", "--format=%H\t%cI\t%s", "-n",
str(args.limit), check=False)
if rc != 0 or not out:
print("no restore points yet")
return 0
print(f"restore points for {folder} (mode mirror: every commit is one)")
for line in out.splitlines():
sha, ts, subject = (line.split("\t") + ["", ""])[:3]
print(f" {ts} {sha[:12]} {subject}")
return 0
snaps = list_snapshots(folder, dev)
if not snaps:
print("no restore points yet (nothing uncommitted has been backed up)")
return 0
print(f"restore points for {folder} (mode snapshot, device {dev[:8]}):")
for snap in snaps[:args.limit]:
print(f" {snap['ts']} {snap['sha'][:12]}")
if len(snaps) > args.limit:
print(f" ... {len(snaps) - args.limit} older (use --limit)")
return 0
def cmd_restore(args):
"""Materialise a restore point into a NEW directory.
Never into the working tree. Someone restoring a backup is already
having a bad day; overwriting the files they still have would be how a
recovery tool becomes the second disaster.
"""
cfg = require_linked(load_config())
folder, meta = _folder_meta(cfg, args.folder)
dev = device_id(cfg)
target = args.at
sha = None
if meta.get("mode", "mirror") == "snapshot":
for snap in list_snapshots(folder, dev):
if snap["ts"] == target or snap["sha"].startswith(target):
sha = snap["sha"]
git(folder, "fetch", "granthi", f"{snap['ref']}")
break
if sha is None:
rc, resolved = git(folder, "rev-parse", "--verify", "--quiet",
f"{target}^{{commit}}", check=False)
if rc != 0 or not resolved.strip():
raise SystemExit(
f"no restore point {target!r} for {folder} "
f"(see: granthi-sync snapshots {folder})")
sha = resolved.strip()
dest = os.path.abspath(
args.into or f"{folder.rstrip(os.sep)}-restore-{target}")
if os.path.exists(dest) and os.listdir(dest):
raise SystemExit(f"refusing to restore over a non-empty path: {dest}")
os.makedirs(dest, exist_ok=True)
proc = subprocess.run(["git", "-C", folder, "archive", "--format=tar", sha],
capture_output=True)
if proc.returncode != 0:
raise SystemExit(
f"could not read {sha[:12]}: {proc.stderr.decode(errors='replace').strip()}")
untar = subprocess.run(["tar", "-x", "-C", dest], input=proc.stdout,
capture_output=True)
if untar.returncode != 0:
raise SystemExit(
f"could not write {dest}: {untar.stderr.decode(errors='replace').strip()}")
log(f"restored {sha[:12]} -> {dest} (your working tree was not touched)")
return 0
def cmd_status(args):
cfg = load_config()
if "login" in cfg:
@@ -520,12 +1113,13 @@ def cmd_status(args):
if not folders:
print("no folders linked")
return 0
rows = [("FOLDER", "REPO", "LAST SYNC", "DIVERGED")]
rows = [("FOLDER", "REPO", "MODE", "LAST SYNC", "DIVERGED")]
for folder, meta in sorted(folders.items()):
rows.append((folder, meta.get("name", "?"),
meta.get("mode", "mirror"),
meta.get("last_sync", "never"),
"YES" if meta.get("diverged") else "no"))
widths = [max(len(r[i]) for r in rows) for i in range(4)]
widths = [max(len(r[i]) for r in rows) for i in range(len(rows[0]))]
for r in rows:
print(" ".join(c.ljust(w) for c, w in zip(r, widths)))
return 0
@@ -544,21 +1138,48 @@ def main(argv=None):
sp.set_defaults(fn=cmd_link)
sp = sub.add_parser("list", help="list forge repos this account can see")
sp.add_argument("pattern", nargs="?",
help="filter by name: substring, or a glob like 'work-*'")
sp.set_defaults(fn=cmd_list)
sp = sub.add_parser("get", help="clone a forge repo and keep it synced")
sp.add_argument("repo", help="repo name, or owner/repo")
sp.add_argument("--into", help="target folder (default: ./<repo>)")
sp.add_argument("repo", nargs="?", help="repo name, or owner/repo")
sp.add_argument("--all", action="store_true",
help="clone every repo this account can see")
sp.add_argument("--into",
help="target folder (default: ./<repo>; with --all, the "
"directory to clone into)")
sp.add_argument("--mode", choices=("mirror", "snapshot"),
help="override the sync mode (default: snapshot, or "
"whatever the repo was marked with)")
sp.set_defaults(fn=cmd_get)
sp = sub.add_parser("add", help="link a folder and push it to the cloud")
sp.add_argument("folder")
sp.add_argument("--name")
sp.add_argument("--mode", choices=("mirror", "snapshot"),
help="override the sync mode (default: mirror for a plain "
"folder, snapshot for an existing git repo)")
sp.add_argument("--force", action="store_true",
help="skip the folder-size guard")
grp = sp.add_mutually_exclusive_group()
grp.add_argument("--private", dest="private", action="store_true", default=True)
grp.add_argument("--public", dest="private", action="store_false")
sp.set_defaults(fn=cmd_add)
sp = sub.add_parser("snapshots", help="list restore points for a folder")
sp.add_argument("folder")
sp.add_argument("--limit", type=int, default=20)
sp.set_defaults(fn=cmd_snapshots)
sp = sub.add_parser("restore",
help="materialise a restore point into a NEW folder")
sp.add_argument("folder")
sp.add_argument("--at", required=True,
help="timestamp or sha from `granthi-sync snapshots`")
sp.add_argument("--into", help="destination (default: <folder>-restore-<at>)")
sp.set_defaults(fn=cmd_restore)
sp = sub.add_parser("watch", help="sync loop over linked folders")
sp.add_argument("--interval", type=int, default=30)
sp.add_argument("--once", action="store_true", help="single pass then exit")