Add cmd/ingotctl with four subcommands: blocks (list with stats),
inspect (series/postings dump), chunks (decode raw samples by ref), and
fsck (CRC and index integrity validation across all blocks).
Add cmd/ingothttp with JSON query endpoints: /api/v1/query_range
(Prometheus-style matrix response), /api/v1/read (matcher-based read
requests), and /epi/v1/status (DB stats snapshot). Uses JSON instead of
protobuf to maintain zero dependencies.
Add self-instrumentation via the normal Appender path so metrics are
queryable with the same API: ingot_head_series,
ingot_head_chunks_active, ingot_blocks_total, ingot_compactions_total,
ingot_wal_fsync_duration_seconds.
Supporting changes: block.Validate() and block.ReadMeta() exports,
Head.Stats() for series/chunk counts, WAL.LastSyncDuration() with timed
fsync tracking, DB.Stats() for the HTTP status endpoint.
Block refcounting (atomic refs + condemned flag) lets live queries
survive concurrent compaction and retention. Levelled compactor merges
2h->8h->32h blocks via raw chunk passthrough. Retention drops blocks
older than the configured window. Background goroutine drives
flush/compact/retain cycles; exported RunCompaction/ApplyRetention allow
deterministic test control via injectable clock.
Soak test: 10k series x 48h simulated at 15s intervals (115M samples).
Validates flat memory (158 MiB peak), bounded disk (13 blocks), and zero
query errors during compaction.
All existing tests refactored to table-driven with uniform assertions.
Label matchers (equal, not-equal, regexp, not-regexp) in labels/,
postings intersection/union/difference in internal/postings/,
LabelValues and AllPostings on index, block, and head readers.
ingot.go wired to real DB backed by head + block readers. Querier
snapshots overlapping blocks, Select resolves matchers to postings,
merged iterator chains blocks then head with block-wins dedup.
Oracle tests compare every query against a naive []sample reference
across head-only, block-only, head+block merge seam, multiple blocks,
all four matcher types, and time range filtering.
Flush sealed head chunks to ULID-named block directories on disk. Each
block contains CRC'd chunk segment files (mmap'd for reads), a binary
index (symbol table, series, postings with TOC), and a meta.json written
last as the immutability gate. WAL is truncated after block fsync,
preserving the crash-safety ording invariant.
david