memoturn

What it is

The layer that answers whether an LLM application is working. Traces, spans, generations and scores over OpenTelemetry with GenAI semantic conventions, so instrumentation is not proprietary. Cost, token and latency metrics at p50 and p95, aggregated by day and by model.

Evaluation runs three ways rather than one: offline against datasets and experiments, online against sampled production traces through the worker, and human through review queues. All three write scores into the same store, and the score shows on the trace it came from.

The case for it came from tracing adventure-agent (a LangGraph system with seventeen-plus specialised agents under an orchestrator) and finding that the interesting failures were invisible without it.

Architecture

The ingest contract

import { Memoturn, wrapOpenAI } from "@memoturn/sdk";

const mt = new Memoturn();
const trace = mt.trace({ name: "chat", userId: "u1" });

trace
  .generation({ name: "answer", model: "claude-sonnet-4-6", input: messages })
  .end({ output, usage });

await mt.shutdown(); # flushes before the process exits

The decisions worth defending

• The ack is early and that is the point. The API validates, writes the raw event log to blob, enqueues, and returns 207 without waiting for the analytical write. An observability tool that rejects traffic when its own warehouse is slow has inverted its job: a Doris stall should cost freshness, not the caller’s data.
• The raw log is written before the queue. Blob holds a replayable record of what actually arrived, so a bad merge is recoverable by replay rather than by asking customers to re-send telemetry they no longer have.
• PII masking happens at ingest, not at read. Masking on the way out means the unmasked value was already stored, which is the wrong side of the boundary for anything a compliance reviewer will ask about.
• Prompts are versioned with deployment channels. Production, latest, or a custom channel, resolved by the SDK at call time. Prompt changes stop being code deploys, which is what makes online evaluation of a prompt change meaningful at all.

Surface

SDKs in TypeScript, Python and Go. An MCP server exposing prompts, datasets and review queues to agent IDEs. Better Auth with organisations, projects, RBAC and SSO over OIDC and SAML. Per-project rate limiting, audit logs, retention policy and scheduled NDJSON exports.

Apache-2.0, at v0.5.0, with a live demo and published packages on npm and PyPI.

Ask about it

The assistant on the home page answers from this same record, and will tell you what is not on it.