shre-embed/README.md

# shre-embed

Shared embedding + semantic search service for the estate. Plain Node.js +
Express, ESM, no build step. Embeds text via a local **Ollama** instance and
stores/searches vectors in **Qdrant**.

## Design principle: three-valued observability

Every probe and every upstream-dependent answer in this service distinguishes
**three** states, never two:

| state | meaning |
|---|---|
| `ok` / present | the probe saw the thing and it works / exists |
| `absent` | the probe **saw the surface** and the thing is genuinely missing (e.g. Ollama reachable but the embed model is not installed) |
| `unobservable` | the probe **could not see**: dependency down, timeout, auth failure, endpoint missing, HTTP 5xx from the surface |

`unobservable` is **never** collapsed into "absent", "zero results" or
"healthy". Every result carries its `measurement_surface` (the exact URL that
was measured) and `observed_at` (ISO timestamp). This exists to prevent a
recorded estate incident class: healthchecks probing `/` and reporting healthy
while `/api` was 502, and an MCP tool answering `0` for data that was never
collectable.

Concretely:

- `/health` returns overall `"ok"` only when **all** dependencies are `ok`;
  otherwise `"degraded"` with HTTP 503. It never reports plain healthy while a
  dependency is unobservable.
- `/v1/embed`, `/v1/index`, `/v1/search` return HTTP 502 with
  `{"status":"unobservable", "measurement_surface": ..., "observed_at": ...}`
  when an upstream call fails — a failed search is never presented as an empty
  result set, and a failed index is never presented as `indexed: 0`.

## Configuration (env, with defaults)

| var | default | notes |
|---|---|---|
| `PORT` | `5499` | HTTP listen port (binds 127.0.0.1) |
| `QDRANT_URL` | `http://127.0.0.1:6333` | Qdrant REST endpoint |
| `OLLAMA_URL` | `http://127.0.0.1:11436` | **11436, not 11434.** On this Mac, `127.0.0.1:11434` is a known-broken Colima SSH-mux forward that accepts connections (tags respond) but hangs on generation/embedding. Native Ollama listens on `11436`. |
| `EMBED_MODEL` | `nomic-embed-text` | Ollama embedding model |
| `PROBE_TIMEOUT_MS` | `2500` | timeout for `/health` dependency probes |
| `UPSTREAM_TIMEOUT_MS` | `30000` | timeout for embed/index/search upstream calls |

## Run

```sh
npm install
npm start          # listens on 127.0.0.1:5499
npm test           # node:test unit tests; no Qdrant/Ollama required
```

## Endpoints

### `GET /health`

Three-valued, per-dependency health with exact probed surfaces.

```json
{
  "status": "degraded",
  "service": "shre-embed",
  "observed_at": "2026-08-22T15:04:05.000Z",
  "dependencies": {
    "qdrant": {
      "status": "ok",
      "measurement_surface": "http://127.0.0.1:6333/readyz",
      "observed_at": "2026-08-22T15:04:05.000Z",
      "http_status": 200,
      "latency_ms": 3
    },
    "ollama": {
      "status": "unobservable",
      "measurement_surface": "http://127.0.0.1:11436/api/tags",
      "observed_at": "2026-08-22T15:04:05.000Z",
      "reason": "fetch failed: ECONNREFUSED: fetch failed",
      "latency_ms": 1
    },
    "embed_model": {
      "status": "unobservable",
      "measurement_surface": "http://127.0.0.1:11436/api/tags",
      "observed_at": "2026-08-22T15:04:05.000Z",
      "model": "nomic-embed-text",
      "reason": "ollama unobservable, model presence not measurable"
    }
  }
}
```

- HTTP 200 when overall `"ok"`, HTTP 503 when `"degraded"`.
- `embed_model` demonstrates all three values: `ok` (installed), `absent`
  (Ollama reachable but model not pulled — a true gap), `unobservable`
  (Ollama itself could not be seen, so the model's presence is unknowable).

### `POST /v1/embed`

```json
{ "texts": ["hello", "world"] }
```

→ `200 {"vectors": [[...],[...]], "model": "nomic-embed-text", "measurement_surface": "http://127.0.0.1:11436/api/embeddings", "observed_at": "..."}`

Each text is embedded via `POST {OLLAMA_URL}/api/embeddings` with
`{"model": EMBED_MODEL, "prompt": text}`.

### `POST /v1/index`

```json
{
  "collection": "notes",
  "items": [
    { "id": 1, "text": "alpha", "payload": { "src": "obsidian" } },
    { "id": 2, "text": "beta" }
  ]
}
```

Embeds every `text`, auto-creates the Qdrant collection if missing (Cosine
distance, vector size taken from the first embedding), and upserts points with
`payload` (the original `text` is folded into the payload as `text`).

→ `200 {"indexed": 2, "collection": "notes", "qdrant": {...}, "measurement_surface": ..., "observed_at": ...}`

### `POST /v1/search`

```json
{ "collection": "notes", "query": "find alpha", "limit": 5 }
```

Embeds the query, then `POST {QDRANT_URL}/collections/{collection}/points/search`
with `with_payload: true`. `limit` defaults to 10, capped at 100.

→ `200 {"results": [{"id": "a", "score": 0.91, "payload": {...}}, ...], "collection": ..., "limit": ..., "measurement_surface": ..., "observed_at": ...}`

### Error shape for all `/v1/*` endpoints

- `400 {"error": "..."}` — invalid request body.
- `502 {"error": "...", "status": "unobservable", "measurement_surface": "<exact upstream URL>", "observed_at": "...", "upstream_http_status": 500}` —
  the upstream could not be observed; the truthful answer is "unknown", never
  "empty" or "zero".

## Example launchd plist (documentation only — do NOT install blindly)

The estate convention is `ai.shre.<service>` labels. This is a reference
template only; deployment goes through the normal ops-from-git flow, not by
hand-installing this file.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>ai.shre.embed</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/node</string>
    <string>/opt/shre/shre-embed/src/server.js</string>
  </array>
  <key>WorkingDirectory</key><string>/opt/shre/shre-embed</string>
  <key>EnvironmentVariables</key>
  <dict>
    <key>PORT</key><string>5499</string>
    <key>QDRANT_URL</key><string>http://127.0.0.1:6333</string>
    <!-- 11436 = native Ollama. NEVER 11434 (broken Colima forward on this Mac). -->
    <key>OLLAMA_URL</key><string>http://127.0.0.1:11436</string>
    <key>EMBED_MODEL</key><string>nomic-embed-text</string>
  </dict>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>StandardOutPath</key><string>/tmp/ai.shre.embed.out.log</string>
  <key>StandardErrorPath</key><string>/tmp/ai.shre.embed.err.log</string>
</dict>
</plist>
```

## Layout

```
src/config.js     env config with estate-safe defaults
src/probe.js      three-valued probe primitives (ok | absent | unobservable)
src/upstreams.js  Ollama + Qdrant clients (injectable fetch)
src/app.js        express app factory (createApp(config, {fetchImpl}))
src/server.js     entrypoint
test/app.test.js  node:test unit tests, run with zero live dependencies
```