27. Full-content chunks in CouchDB (per-turn content, not just byte ranges)

Date: 2026-07-22

Status

Accepted — implemented on both the write and read paths.

The hook + backfill embed entries[] when couchFullContentChunks is on, validated at the webapi. Every consuming view has landed:

• speaker_split/by_role (v4) → GET /api/sessions/{id}/turns, with the webui's per-speaker toggle on the session detail;
• speaker_split/by_role_time (v5) → GET /api/turns, cross-session in time order;
• chunks/entries_by_session (v6) → the transcript in reading order across speakers, which makes chunks the default source for GET /api/sessions/{id}/transcript (S3 is the fallback) — so a live or crashed session is readable. This narrows ADR 0014.

Per-turn search over entries[] is live in Meilisearch's turns index (ADR 0009).

Content chunks are the default for both writers (the hook and backfill), so history adopted from here on needs nothing extra.

Remaining, and only where history was adopted before this landed or with --no-content: those sessions carry byte-range-only chunks, fall back to S3 on read, and contribute nothing to content search. backfill can't redo them — it skips any session that already has a summary doc — so closing this needs either a re-process flag on backfill or a migration that rebuilds chunks from the S3 transcript. Not built, because no known deployment is in that state.

Context

A session is stored two ways. The byte-faithful transcript lives in S3 as a single transcript.jsonl object (ADR 0014), and CouchDB holds append-only metadata: event markers, a summary doc at SessionEnd, and — added since — chunk docs written mid-flight for crash resilience (mid-flight-chunking.md).

The mid-flight chunking mechanism is already built: the hook flushes chunk docs as the session grows, ingest is idempotent with stable ids (chunk:<session>:<byteStart>), and backfill reconstructs the same chunks for adopted history. But a chunk doc today records only a byte-range slice — byte_start, byte_end, entry_count — a pointer into the S3 transcript. It does not contain the messages.

The consequence: CouchDB cannot see an individual turn. Anything that needs per-turn structure — speaker-split views (user vs Claude), per-turn search indexing, map-reduce feature extraction, prompt/instruction provenance — is blocked, because the only place the turns exist in parsed form is the S3 blob, and map-reduce can't run over S3. The couchFullContentChunks feature flag was added in anticipation of this but nothing populates content yet.

This is the remaining half of the logging rework (roadmap #4).

Decision

Promote chunk docs from byte-range pointers to full-content chunks: parse the transcript entries and store their content in CouchDB, so map-reduce views operate directly on turns.

• A chunk carries its parsed entries. In addition to the existing byte-range fields (kept, so a chunk still maps 1:1 to a transcript slice and stays reconstructable), a chunk doc gains the parsed entries it covers: for each, the role/type (user | assistant | tool result | system), a stable per-entry index, timestamps, and the content (text + tool-use structure). The exact per-entry projection is defined in couchdb-documents.md and the shared validators, not here.
• Append-only + immutable, like every other doc (ADR 0016): content chunks are written once by the writer (hook or backfill), keyed by byte offset, never edited in place.
• Schema-versioned + migrated. The new fields and the design views that read them ship through the self-built migrations (ADR 0021), never ad-hoc — chunk docs carry schema_version, and a migration can re-derive content chunks for existing sessions from their S3 transcript.
• S3 stays the source of truth (ADR 0014). Content chunks are a projection of the transcript for query, not a second authority; they are byte-attributable back to the S3 object and rebuildable from it. The transcript is never reconstructed from chunks.
• Byte-identical writer/shared code. The parse + chunk-content builder is written once in @claude-transcripts/shared and (at the time) copied byte-identically into the hook (which can't resolve the workspace), exactly as sumTranscriptTokens and sliceIntoChunks already are.
• Guarded by couchFullContentChunks. The flag gates population and the content-reading views, so the byte-range-only behaviour remains a fallback.

Consequences

• Unlocks speaker-split views, per-turn search (Meilisearch indexing over turns), map-reduce feature/analytics views, and prompt/instruction provenance — all of which become straightforward map functions over role/content.
• CouchDB storage grows: the corpus is effectively stored twice (S3 blob + parsed content in Couch). Accepted for Tier-1/2 volumes; chunk granularity (maxEntriesPerChunk) bounds per-doc size, and content chunks can be pruned/ re-derived from S3 since S3 remains authoritative.
• The content projection must track Claude Code's transcript entry shape; a drift check (cf. ADR 0025) guards it, and the schema_version + migration path absorbs format changes.
• A migration backfills content chunks for already-recorded sessions from their S3 transcripts, so history is not left behind.
• Redaction/secrets-masking (app-logging.md, #11) becomes more load-bearing: parsed content in Couch is more directly queryable than an opaque blob, so masking-on-write matters more here.