# shre-embed Shared embedding + semantic search service for the estate. Plain Node.js + Express, ESM, no build step. Embeds text via a local **Ollama** instance and stores/searches vectors in **Qdrant**. ## Design principle: three-valued observability Every probe and every upstream-dependent answer in this service distinguishes **three** states, never two: | state | meaning | |---|---| | `ok` / present | the probe saw the thing and it works / exists | | `absent` | the probe **saw the surface** and the thing is genuinely missing (e.g. Ollama reachable but the embed model is not installed) | | `unobservable` | the probe **could not see**: dependency down, timeout, auth failure, endpoint missing, HTTP 5xx from the surface | `unobservable` is **never** collapsed into "absent", "zero results" or "healthy". Every result carries its `measurement_surface` (the exact URL that was measured) and `observed_at` (ISO timestamp). This exists to prevent a recorded estate incident class: healthchecks probing `/` and reporting healthy while `/api` was 502, and an MCP tool answering `0` for data that was never collectable. Concretely: - `/health` returns overall `"ok"` only when **all** dependencies are `ok`; otherwise `"degraded"` with HTTP 503. It never reports plain healthy while a dependency is unobservable. - `/v1/embed`, `/v1/index`, `/v1/search` return HTTP 502 with `{"status":"unobservable", "measurement_surface": ..., "observed_at": ...}` when an upstream call fails — a failed search is never presented as an empty result set, and a failed index is never presented as `indexed: 0`. ## Configuration (env, with defaults) | var | default | notes | |---|---|---| | `PORT` | `5499` | HTTP listen port (binds 127.0.0.1) | | `QDRANT_URL` | `http://127.0.0.1:6333` | Qdrant REST endpoint | | `OLLAMA_URL` | `http://127.0.0.1:11436` | **11436, not 11434.** On this Mac, `127.0.0.1:11434` is a known-broken Colima SSH-mux forward that accepts connections (tags respond) but hangs on generation/embedding. Native Ollama listens on `11436`. | | `EMBED_MODEL` | `nomic-embed-text` | Ollama embedding model | | `PROBE_TIMEOUT_MS` | `2500` | timeout for `/health` dependency probes | | `UPSTREAM_TIMEOUT_MS` | `30000` | timeout for embed/index/search upstream calls | ## Run ```sh npm install npm start # listens on 127.0.0.1:5499 npm test # node:test unit tests; no Qdrant/Ollama required ``` ## Endpoints ### `GET /health` Three-valued, per-dependency health with exact probed surfaces. ```json { "status": "degraded", "service": "shre-embed", "observed_at": "2026-08-22T15:04:05.000Z", "dependencies": { "qdrant": { "status": "ok", "measurement_surface": "http://127.0.0.1:6333/readyz", "observed_at": "2026-08-22T15:04:05.000Z", "http_status": 200, "latency_ms": 3 }, "ollama": { "status": "unobservable", "measurement_surface": "http://127.0.0.1:11436/api/tags", "observed_at": "2026-08-22T15:04:05.000Z", "reason": "fetch failed: ECONNREFUSED: fetch failed", "latency_ms": 1 }, "embed_model": { "status": "unobservable", "measurement_surface": "http://127.0.0.1:11436/api/tags", "observed_at": "2026-08-22T15:04:05.000Z", "model": "nomic-embed-text", "reason": "ollama unobservable, model presence not measurable" } } } ``` - HTTP 200 when overall `"ok"`, HTTP 503 when `"degraded"`. - `embed_model` demonstrates all three values: `ok` (installed), `absent` (Ollama reachable but model not pulled — a true gap), `unobservable` (Ollama itself could not be seen, so the model's presence is unknowable). ### `POST /v1/embed` ```json { "texts": ["hello", "world"] } ``` → `200 {"vectors": [[...],[...]], "model": "nomic-embed-text", "measurement_surface": "http://127.0.0.1:11436/api/embeddings", "observed_at": "..."}` Each text is embedded via `POST {OLLAMA_URL}/api/embeddings` with `{"model": EMBED_MODEL, "prompt": text}`. ### `POST /v1/index` ```json { "collection": "notes", "items": [ { "id": 1, "text": "alpha", "payload": { "src": "obsidian" } }, { "id": 2, "text": "beta" } ] } ``` Embeds every `text`, auto-creates the Qdrant collection if missing (Cosine distance, vector size taken from the first embedding), and upserts points with `payload` (the original `text` is folded into the payload as `text`). → `200 {"indexed": 2, "collection": "notes", "qdrant": {...}, "measurement_surface": ..., "observed_at": ...}` ### `POST /v1/search` ```json { "collection": "notes", "query": "find alpha", "limit": 5 } ``` Embeds the query, then `POST {QDRANT_URL}/collections/{collection}/points/search` with `with_payload: true`. `limit` defaults to 10, capped at 100. → `200 {"results": [{"id": "a", "score": 0.91, "payload": {...}}, ...], "collection": ..., "limit": ..., "measurement_surface": ..., "observed_at": ...}` ### Error shape for all `/v1/*` endpoints - `400 {"error": "..."}` — invalid request body. - `502 {"error": "...", "status": "unobservable", "measurement_surface": "", "observed_at": "...", "upstream_http_status": 500}` — the upstream could not be observed; the truthful answer is "unknown", never "empty" or "zero". ## Example launchd plist (documentation only — do NOT install blindly) The estate convention is `ai.shre.` labels. This is a reference template only; deployment goes through the normal ops-from-git flow, not by hand-installing this file. ```xml Labelai.shre.embed ProgramArguments /usr/local/bin/node /opt/shre/shre-embed/src/server.js WorkingDirectory/opt/shre/shre-embed EnvironmentVariables PORT5499 QDRANT_URLhttp://127.0.0.1:6333 OLLAMA_URLhttp://127.0.0.1:11436 EMBED_MODELnomic-embed-text RunAtLoad KeepAlive StandardOutPath/tmp/ai.shre.embed.out.log StandardErrorPath/tmp/ai.shre.embed.err.log ``` ## Layout ``` src/config.js env config with estate-safe defaults src/probe.js three-valued probe primitives (ok | absent | unobservable) src/upstreams.js Ollama + Qdrant clients (injectable fetch) src/app.js express app factory (createApp(config, {fetchImpl})) src/server.js entrypoint test/app.test.js node:test unit tests, run with zero live dependencies ```