shre-embed/README.md

shre-embed

Shared embedding + semantic search service for the estate. Plain Node.js + Express, ESM, no build step. Embeds text via a local Ollama instance and stores/searches vectors in Qdrant.

Design principle: three-valued observability

Every probe and every upstream-dependent answer in this service distinguishes three states, never two:

state	meaning
ok / present	the probe saw the thing and it works / exists
absent	the probe saw the surface and the thing is genuinely missing (e.g. Ollama reachable but the embed model is not installed)
unobservable	the probe could not see: dependency down, timeout, auth failure, endpoint missing, HTTP 5xx from the surface

unobservable is never collapsed into "absent", "zero results" or "healthy". Every result carries its measurement_surface (the exact URL that was measured) and observed_at (ISO timestamp). This exists to prevent a recorded estate incident class: healthchecks probing / and reporting healthy while /api was 502, and an MCP tool answering 0 for data that was never collectable.

Concretely:

• /health returns overall "ok" only when all dependencies are ok; otherwise "degraded" with HTTP 503. It never reports plain healthy while a dependency is unobservable.
• /v1/embed, /v1/index, /v1/search return HTTP 502 with {"status":"unobservable", "measurement_surface": ..., "observed_at": ...} when an upstream call fails — a failed search is never presented as an empty result set, and a failed index is never presented as indexed: 0.

Configuration (env, with defaults)

var	default	notes
PORT	5499	HTTP listen port (binds 127.0.0.1)
QDRANT_URL	http://127.0.0.1:6333	Qdrant REST endpoint
OLLAMA_URL	http://127.0.0.1:11436	11436, not 11434. On this Mac, 127.0.0.1:11434 is a known-broken Colima SSH-mux forward that accepts connections (tags respond) but hangs on generation/embedding. Native Ollama listens on 11436.
EMBED_MODEL	nomic-embed-text	Ollama embedding model
PROBE_TIMEOUT_MS	2500	timeout for /health dependency probes
UPSTREAM_TIMEOUT_MS	30000	timeout for embed/index/search upstream calls

Run

npm install
npm start          # listens on 127.0.0.1:5499
npm test           # node:test unit tests; no Qdrant/Ollama required

Endpoints

GET /health

Three-valued, per-dependency health with exact probed surfaces.

{
  "status": "degraded",
  "service": "shre-embed",
  "observed_at": "2026-08-22T15:04:05.000Z",
  "dependencies": {
    "qdrant": {
      "status": "ok",
      "measurement_surface": "http://127.0.0.1:6333/readyz",
      "observed_at": "2026-08-22T15:04:05.000Z",
      "http_status": 200,
      "latency_ms": 3
    },
    "ollama": {
      "status": "unobservable",
      "measurement_surface": "http://127.0.0.1:11436/api/tags",
      "observed_at": "2026-08-22T15:04:05.000Z",
      "reason": "fetch failed: ECONNREFUSED: fetch failed",
      "latency_ms": 1
    },
    "embed_model": {
      "status": "unobservable",
      "measurement_surface": "http://127.0.0.1:11436/api/tags",
      "observed_at": "2026-08-22T15:04:05.000Z",
      "model": "nomic-embed-text",
      "reason": "ollama unobservable, model presence not measurable"
    }
  }
}

• HTTP 200 when overall "ok", HTTP 503 when "degraded".
• embed_model demonstrates all three values: ok (installed), absent (Ollama reachable but model not pulled — a true gap), unobservable (Ollama itself could not be seen, so the model's presence is unknowable).

POST /v1/embed

{ "texts": ["hello", "world"] }

→ 200 {"vectors": [[...],[...]], "model": "nomic-embed-text", "measurement_surface": "http://127.0.0.1:11436/api/embeddings", "observed_at": "..."}

Each text is embedded via POST {OLLAMA_URL}/api/embeddings with {"model": EMBED_MODEL, "prompt": text}.

POST /v1/index

{
  "collection": "notes",
  "items": [
    { "id": 1, "text": "alpha", "payload": { "src": "obsidian" } },
    { "id": 2, "text": "beta" }
  ]
}

Embeds every text, auto-creates the Qdrant collection if missing (Cosine distance, vector size taken from the first embedding), and upserts points with payload (the original text is folded into the payload as text).

→ 200 {"indexed": 2, "collection": "notes", "qdrant": {...}, "measurement_surface": ..., "observed_at": ...}

POST /v1/search

{ "collection": "notes", "query": "find alpha", "limit": 5 }

Embeds the query, then POST {QDRANT_URL}/collections/{collection}/points/search with with_payload: true. limit defaults to 10, capped at 100.

→ 200 {"results": [{"id": "a", "score": 0.91, "payload": {...}}, ...], "collection": ..., "limit": ..., "measurement_surface": ..., "observed_at": ...}

Error shape for all /v1/* endpoints

• 400 {"error": "..."} — invalid request body.
• 502 {"error": "...", "status": "unobservable", "measurement_surface": "<exact upstream URL>", "observed_at": "...", "upstream_http_status": 500} — the upstream could not be observed; the truthful answer is "unknown", never "empty" or "zero".

Example launchd plist (documentation only — do NOT install blindly)

The estate convention is ai.shre.<service> labels. This is a reference template only; deployment goes through the normal ops-from-git flow, not by hand-installing this file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>ai.shre.embed</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/node</string>
    <string>/opt/shre/shre-embed/src/server.js</string>
  </array>
  <key>WorkingDirectory</key><string>/opt/shre/shre-embed</string>
  <key>EnvironmentVariables</key>
  <dict>
    <key>PORT</key><string>5499</string>
    <key>QDRANT_URL</key><string>http://127.0.0.1:6333</string>
    <!-- 11436 = native Ollama. NEVER 11434 (broken Colima forward on this Mac). -->
    <key>OLLAMA_URL</key><string>http://127.0.0.1:11436</string>
    <key>EMBED_MODEL</key><string>nomic-embed-text</string>
  </dict>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>StandardOutPath</key><string>/tmp/ai.shre.embed.out.log</string>
  <key>StandardErrorPath</key><string>/tmp/ai.shre.embed.err.log</string>
</dict>
</plist>

Layout

src/config.js     env config with estate-safe defaults
src/probe.js      three-valued probe primitives (ok | absent | unobservable)
src/upstreams.js  Ollama + Qdrant clients (injectable fetch)
src/app.js        express app factory (createApp(config, {fetchImpl}))
src/server.js     entrypoint
test/app.test.js  node:test unit tests, run with zero live dependencies