nirpa/shre-embed
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Scaffold shre-embed: embedding + semantic search with three-valued health

Browse Source 
Express (ESM, no build step) service wrapping Ollama embeddings and Qdrant:
- POST /v1/embed, /v1/index, /v1/search
- GET /health reports each dependency as ok | absent | unobservable with the
  exact measurement_surface URL and observed_at; overall ok only when all
  deps are ok, 503 degraded otherwise
- upstream failures on /v1/* return 502 marked unobservable — never an empty
  result set or indexed:0
- defaults PORT=5499, QDRANT_URL=127.0.0.1:6333, OLLAMA_URL=127.0.0.1:11436
  (11436 native Ollama; 11434 is the broken Colima forward on this Mac)
- node:test unit tests with injected fetch; run with zero live dependencies
- README documents endpoints and a reference launchd plist (not installed)

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
Claude
2026-08-22 02:05:11 -04:00
parent d3db1701af
commit a707c05dcc
10 changed files with 1751 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+185 -1
View File
@@ -1,3 +1,187 @@
# shre-embed
Shared embedding + semantic search service for the estate (Qdrant + local Ollama)
Shared embedding + semantic search service for the estate. Plain Node.js +
Express, ESM, no build step. Embeds text via a local **Ollama** instance and
stores/searches vectors in **Qdrant**.
## Design principle: three-valued observability
Every probe and every upstream-dependent answer in this service distinguishes
**three** states, never two:
| state | meaning |
|---|---|
| `ok` / present | the probe saw the thing and it works / exists |
| `absent` | the probe **saw the surface** and the thing is genuinely missing (e.g. Ollama reachable but the embed model is not installed) |
| `unobservable` | the probe **could not see**: dependency down, timeout, auth failure, endpoint missing, HTTP 5xx from the surface |
`unobservable` is **never** collapsed into "absent", "zero results" or
"healthy". Every result carries its `measurement_surface` (the exact URL that
was measured) and `observed_at` (ISO timestamp). This exists to prevent a
recorded estate incident class: healthchecks probing `/` and reporting healthy
while `/api` was 502, and an MCP tool answering `0` for data that was never
collectable.
Concretely:
- `/health` returns overall `"ok"` only when **all** dependencies are `ok`;
otherwise `"degraded"` with HTTP 503. It never reports plain healthy while a
dependency is unobservable.
- `/v1/embed`, `/v1/index`, `/v1/search` return HTTP 502 with
`{"status":"unobservable", "measurement_surface": ..., "observed_at": ...}`
when an upstream call fails — a failed search is never presented as an empty
result set, and a failed index is never presented as `indexed: 0`.
## Configuration (env, with defaults)
| var | default | notes |
|---|---|---|
| `PORT` | `5499` | HTTP listen port (binds 127.0.0.1) |
| `QDRANT_URL` | `http://127.0.0.1:6333` | Qdrant REST endpoint |
| `OLLAMA_URL` | `http://127.0.0.1:11436` | **11436, not 11434.** On this Mac, `127.0.0.1:11434` is a known-broken Colima SSH-mux forward that accepts connections (tags respond) but hangs on generation/embedding. Native Ollama listens on `11436`. |
| `EMBED_MODEL` | `nomic-embed-text` | Ollama embedding model |
| `PROBE_TIMEOUT_MS` | `2500` | timeout for `/health` dependency probes |
| `UPSTREAM_TIMEOUT_MS` | `30000` | timeout for embed/index/search upstream calls |
## Run
```sh
npm install
npm start # listens on 127.0.0.1:5499
npm test # node:test unit tests; no Qdrant/Ollama required
```
## Endpoints
### `GET /health`
Three-valued, per-dependency health with exact probed surfaces.
```json
{
"status": "degraded",
"service": "shre-embed",
"observed_at": "2026-08-22T15:04:05.000Z",
"dependencies": {
"qdrant": {
"status": "ok",
"measurement_surface": "http://127.0.0.1:6333/readyz",
"observed_at": "2026-08-22T15:04:05.000Z",
"http_status": 200,
"latency_ms": 3
},
"ollama": {
"status": "unobservable",
"measurement_surface": "http://127.0.0.1:11436/api/tags",
"observed_at": "2026-08-22T15:04:05.000Z",
"reason": "fetch failed: ECONNREFUSED: fetch failed",
"latency_ms": 1
},
"embed_model": {
"status": "unobservable",
"measurement_surface": "http://127.0.0.1:11436/api/tags",
"observed_at": "2026-08-22T15:04:05.000Z",
"model": "nomic-embed-text",
"reason": "ollama unobservable, model presence not measurable"
}
}
}
```
- HTTP 200 when overall `"ok"`, HTTP 503 when `"degraded"`.
- `embed_model` demonstrates all three values: `ok` (installed), `absent`
(Ollama reachable but model not pulled — a true gap), `unobservable`
(Ollama itself could not be seen, so the model's presence is unknowable).
### `POST /v1/embed`
```json
{ "texts": ["hello", "world"] }
```
→ `200 {"vectors": [[...],[...]], "model": "nomic-embed-text", "measurement_surface": "http://127.0.0.1:11436/api/embeddings", "observed_at": "..."}`
Each text is embedded via `POST {OLLAMA_URL}/api/embeddings` with
`{"model": EMBED_MODEL, "prompt": text}`.
### `POST /v1/index`
```json
{
"collection": "notes",
"items": [
{ "id": 1, "text": "alpha", "payload": { "src": "obsidian" } },
{ "id": 2, "text": "beta" }
]
}
```
Embeds every `text`, auto-creates the Qdrant collection if missing (Cosine
distance, vector size taken from the first embedding), and upserts points with
`payload` (the original `text` is folded into the payload as `text`).
→ `200 {"indexed": 2, "collection": "notes", "qdrant": {...}, "measurement_surface": ..., "observed_at": ...}`
### `POST /v1/search`
```json
{ "collection": "notes", "query": "find alpha", "limit": 5 }
```
Embeds the query, then `POST {QDRANT_URL}/collections/{collection}/points/search`
with `with_payload: true`. `limit` defaults to 10, capped at 100.
→ `200 {"results": [{"id": "a", "score": 0.91, "payload": {...}}, ...], "collection": ..., "limit": ..., "measurement_surface": ..., "observed_at": ...}`
### Error shape for all `/v1/*` endpoints
- `400 {"error": "..."}` — invalid request body.
- `502 {"error": "...", "status": "unobservable", "measurement_surface": "<exact upstream URL>", "observed_at": "...", "upstream_http_status": 500}` —
the upstream could not be observed; the truthful answer is "unknown", never
"empty" or "zero".
## Example launchd plist (documentation only — do NOT install blindly)
The estate convention is `ai.shre.<service>` labels. This is a reference
template only; deployment goes through the normal ops-from-git flow, not by
hand-installing this file.
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>ai.shre.embed</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/node</string>
<string>/opt/shre/shre-embed/src/server.js</string>
</array>
<key>WorkingDirectory</key><string>/opt/shre/shre-embed</string>
<key>EnvironmentVariables</key>
<dict>
<key>PORT</key><string>5499</string>
<key>QDRANT_URL</key><string>http://127.0.0.1:6333</string>
<!-- 11436 = native Ollama. NEVER 11434 (broken Colima forward on this Mac). -->
<key>OLLAMA_URL</key><string>http://127.0.0.1:11436</string>
<key>EMBED_MODEL</key><string>nomic-embed-text</string>
</dict>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
<key>StandardOutPath</key><string>/tmp/ai.shre.embed.out.log</string>
<key>StandardErrorPath</key><string>/tmp/ai.shre.embed.err.log</string>
</dict>
</plist>
```
## Layout
```
src/config.js env config with estate-safe defaults
src/probe.js three-valued probe primitives (ok | absent | unobservable)
src/upstreams.js Ollama + Qdrant clients (injectable fetch)
src/app.js express app factory (createApp(config, {fetchImpl}))
src/server.js entrypoint
test/app.test.js node:test unit tests, run with zero live dependencies
```