diff --git a/README.md b/README.md index 48cd781..3f513f4 100644 --- a/README.md +++ b/README.md @@ -1,41 +1,85 @@ # ingot -An embedded time-series database for Go. SQLite for metrics: a library you import, not a server you deploy. +An embedded time-series database for Go. SQLite for metrics. ```go -db, _ := ingot.Open("./data", ingot.Options{Retention: 30 * 24 * time.Hour}) +db, _ := ingot.Open("./data", ingot.Options{ + Retention: 30 * 24 * time.Hour, + BlockDuration: 2 * time.Hour, +}) +// Write app := db.Appender() -app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3) +ref, _ := app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3) +app.Append(ref, nil, ts+15000, 71.4) // ref fast-path skips label hashing app.Commit() +// Read q, _ := db.Querier(mint, maxt) ss := q.Select(labels.MustNewMatcher(labels.MatchEqual, "room", "office")) +for ss.Next() { + it := ss.At().Iterator() + for it.Next() { + t, v := it.At() + _ = t; _ = v + } +} +q.Close() + +db.Close() ``` ## Status -**Pre-alpha. Not usable yet.** Built bottom-up; the public API above is the design target, not the current state. +**Pre-alpha.** The API is frozen (M4) and the system survives a 48h soak test under sustained load (M5), but this hasn't seen production use yet. | Milestone | State | |---|---| | M1 — Gorilla chunk encoding, fuzzed, benchmarked | Done | -| M2 — Head + WAL, kill -9 safe | In progress | -| M3 — Immutable blocks, mmap reads | — | -| M4 — Query path, API freeze, shippable alpha | — | -| M5 — Compaction + retention, 48h soak | — | -| M6 — ingotctl, HTTP layer, Grafana | — | +| M2 — Head + WAL, kill -9 safe | Done | +| M3 — Immutable blocks, mmap reads | Done | +| M4 — Query path, API freeze, shippable alpha | Done | +| M5 — Compaction + retention, 48h soak | Done | +| M6 — ingotctl, HTTP layer, self-instrumentation | Done | -See [DESIGN.md](DESIGN.md) for architecture, on-disk format, and the non-goals table (no replication, no PromQL, no deletes, no out-of-order writes — each one deliberate). +See [DESIGN.md](DESIGN.md) for architecture, on-disk format, and the non-goals table. See [ROADMAP.md](ROADMAP.md) for what's next. -## Why +## Features -Go programs that need local metrics storage have two options: run a Prometheus-shaped server next to your process, or hand-roll encoding on top of a key-value store. The embedded middle ground — common in the SQLite world — doesn't exist for time series in Go. Target users: +- **Gorilla XOR compression** — ~1 byte/sample on regular metric data (see benchmarks below) +- **Crash-safe** — WAL with CRC32C records; kill -9 at any point loses at most uncommitted samples +- **Query by label matchers** — equality, negation, regex, negative regex; merged across head and blocks +- **Levelled compaction** — 2h → 8h → 32h blocks, background merging, retention-based expiry +- **Self-instrumentation** — the DB records its own metrics (series/chunk counts, compactions, WAL fsync duration) through the normal write path, queryable like any other series +- **Zero external dependencies** (except testify for tests) -- Edge/IoT devices buffering sensor data locally -- Go binaries recording their own operational history -- Homelab sidecars for sensor firehoses (Home Assistant recorder, but purpose-built) -- Sampling agents and CLIs currently writing CSV +## Tools + +### ingotctl + +CLI for block inspection and diagnostics: + +```sh +ingotctl blocks ./data # list blocks with ULID, time range, stats +ingotctl inspect ./data/01HXYZ.../ # series labels, chunk metadata, postings stats +ingotctl chunks ./data/01HXYZ.../ 42 # decode and print raw samples for series ref 42 +ingotctl fsck ./data # CRC + index integrity check on all blocks +``` + +### ingothttp + +Minimal HTTP query server for Grafana integration: + +```sh +ingothttp -data ./data -addr :9001 +``` + +Endpoints: +- `GET /api/v1/query_range?query=&start=&end=` — Prometheus-style JSON matrix response +- `POST /api/v1/read` — JSON read request with label matchers +- `GET /api/v1/status` — DB stats snapshot + +This is a demo/bridge, not a full PromQL engine. The query parameter matches `__name__` by equality. ## Compression @@ -58,32 +102,48 @@ Two things worth knowing about XOR compression that the headline numbers hide: ## Layout ``` -ingot/ public API (target: Open, Appender, Querier) +ingot/ public API: Open, Appender, Querier ├── internal/ -│ ├── chunkenc/ Gorilla encoder/decoder, bitstream [done] -│ ├── wal/ segmented write-ahead log [next] +│ ├── chunkenc/ Gorilla encoder/decoder, bitstream +│ ├── wal/ segmented write-ahead log │ ├── head/ in-memory series, active chunks │ ├── index/ symbols, postings, matchers -│ ├── block/ immutable block read/write -│ └── compact/ merge + retention -├── cmd/ingotctl/ block inspection, fsck +│ ├── block/ immutable block read/write, validation +│ ├── compact/ levelled merge + retention +│ └── postings/ sorted posting list operations +├── cmd/ +│ ├── ingotctl/ block inspection, fsck +│ └── ingothttp/ HTTP query server └── labels/ label types ``` ## Development ```sh -go test -race ./... -go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/ -go test -bench=. ./internal/chunkenc/ +go test -race -short ./... # all tests (skip soak) +go test -race ./... # all tests including soak (~5 min) +go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/ # fuzz the decoder +go test -bench=. ./internal/chunkenc/ # benchmarks +go build ./cmd/ingotctl/ # build CLI tool +go build ./cmd/ingothttp/ # build HTTP server ``` -The decoder is total: arbitrary bytes produce values or `ErrShortStream`, never a panic. Fuzzing gates every change to `chunkenc`. +The decoder is total: arbitrary bytes produce values or `ErrShortStream` and never panics. Fuzzing gates every change to `chunkenc`. + +## Inspiration + +Most of the design is lifted from Prometheus TSDB — chunk encoding, index format, block/compaction model, label data model. The WAL is simpler (no page-level framing). The difference is that Prometheus TSDB is a storage engine inside a server; ingot is a library. See [NOTICE.md](NOTICE.md). + +The chunk encoding comes from the Gorilla paper (Pelkonen et al., VLDB 2015) via Prometheus, which adapted the bit-width buckets for millisecond timestamps. ingot uses the Prometheus variant. + +[tstorage](https://github.com/nakabonne/tstorage) is the closest existing embedded TSDB for Go. It doesn't do Gorilla compression or label-based indexing, which is most of why ingot exists. + +I needed a library to store time-series data locally and the options out there didn't quite fit. Prometheus was too heavy for what I needed but I wanted that level of compression. tstorage was close but it didn't have the compression or label indexing. ## Non-goals -Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows. The reasoning for each is in DESIGN.md §3 — they're decisions, not omissions. +Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows (sorry not my thing). Check DESIGN.md for reasoning. ## License -TBD \ No newline at end of file +Apache 2.0 diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..8b822ff --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,23 @@ +# Roadmap + +Post-M6 work, roughly in priority order. + +## Protobuf remote-read + +Replace the JSON `/api/v1/read` endpoint with Prometheus-compatible protobuf+snappy encoding so ingothttp works as a native remote-read target for Prometheus and Grafana. Adds protobuf and snappy as dependencies to the cmd binary; the library stays zero-dep. + +## Grafana demo + +Docker-compose setup with ingothttp and Grafana pre-configured. Include a data generator that writes a few series so the dashboard isn't empty on first load. `docker compose up`, open localhost:3000. + +## Query path benchmarks + +`Benchmark*` tests for `Querier`, `Select`, and iteration across varying series/chunk/block counts. Track ns/query and allocations. The write path has benchmarks already; the read path has none. + +## CI + +At minimum: +- `go test -race -short ./...` on every push +- `go test -race -run TestSoak ./...` with a 10-minute timeout on main/nightly +- `go vet ./...` +- A linter (staticcheck or golangci-lint)