Add cmd/ingotctl with four subcommands: blocks (list with stats),
inspect (series/postings dump), chunks (decode raw samples by ref), and
fsck (CRC and index integrity validation across all blocks).
Add cmd/ingothttp with JSON query endpoints: /api/v1/query_range
(Prometheus-style matrix response), /api/v1/read (matcher-based read
requests), and /epi/v1/status (DB stats snapshot). Uses JSON instead of
protobuf to maintain zero dependencies.
Add self-instrumentation via the normal Appender path so metrics are
queryable with the same API: ingot_head_series,
ingot_head_chunks_active, ingot_blocks_total, ingot_compactions_total,
ingot_wal_fsync_duration_seconds.
Supporting changes: block.Validate() and block.ReadMeta() exports,
Head.Stats() for series/chunk counts, WAL.LastSyncDuration() with timed
fsync tracking, DB.Stats() for the HTTP status endpoint.
Block refcounting (atomic refs + condemned flag) lets live queries
survive concurrent compaction and retention. Levelled compactor merges
2h->8h->32h blocks via raw chunk passthrough. Retention drops blocks
older than the configured window. Background goroutine drives
flush/compact/retain cycles; exported RunCompaction/ApplyRetention allow
deterministic test control via injectable clock.
Soak test: 10k series x 48h simulated at 15s intervals (115M samples).
Validates flat memory (158 MiB peak), bounded disk (13 blocks), and zero
query errors during compaction.
All existing tests refactored to table-driven with uniform assertions.
Label matchers (equal, not-equal, regexp, not-regexp) in labels/,
postings intersection/union/difference in internal/postings/,
LabelValues and AllPostings on index, block, and head readers.
ingot.go wired to real DB backed by head + block readers. Querier
snapshots overlapping blocks, Select resolves matchers to postings,
merged iterator chains blocks then head with block-wins dedup.
Oracle tests compare every query against a naive []sample reference
across head-only, block-only, head+block merge seam, multiple blocks,
all four matcher types, and time range filtering.
Flush sealed head chunks to ULID-named block directories on disk. Each
block contains CRC'd chunk segment files (mmap'd for reads), a binary
index (symbol table, series, postings with TOC), and a meta.json written
last as the immutability gate. WAL is truncated after block fsync,
preserving the crash-safety ording invariant.
Head connects chunkenc and WAL: label-to-ref resolution with striped
concurrent maps, active Gorilla chunk per series sealing at 120 samples,
and an Appender that buffers samples, writes WAL on commit, then applies
to head. OOO rejection checks both committed and batch state. WAL replay
on Open rebuilds the full in-memory state. Exported
ChunkAppender/ChunkIterator interfaces from chunkenc.
Append-only WAL with CRC32-validated records, automatic segment rotation
at 128 MiB, background periodic fsync, and crash recovery by truncation
at the first corrupt or incomplete record.
Record format: type(1) | len(4) | payload | crc32c(4). Two payload
types: series (ref + labels) and samples (batch of ref/t/v). Fixed-width
encoding througout.
Torn-write harness truncates at every byte offset and asserts recovery
produces a valid prefix of the original sequnce, both single-segment and
multi-segment.
DESIGN.md, NOTICE.md, and README.md.
Bitstream primitives (bstream) and XOR encoder/decoder following the
Prometheus TSDB variant of the Gorilla scheme. Delta-of-delta timestamps
with 14/17/20/64-bit buckets, XOR-compressed float64 values. Decoder is
total: arbitrary bytes produce values or ErrShortStream.
david