Update README.md and ROADMAP.md
This commit is contained in:
@@ -1,41 +1,85 @@
|
||||
# ingot
|
||||
|
||||
An embedded time-series database for Go. SQLite for metrics: a library you import, not a server you deploy.
|
||||
An embedded time-series database for Go. SQLite for metrics.
|
||||
|
||||
```go
|
||||
db, _ := ingot.Open("./data", ingot.Options{Retention: 30 * 24 * time.Hour})
|
||||
db, _ := ingot.Open("./data", ingot.Options{
|
||||
Retention: 30 * 24 * time.Hour,
|
||||
BlockDuration: 2 * time.Hour,
|
||||
})
|
||||
|
||||
// Write
|
||||
app := db.Appender()
|
||||
app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3)
|
||||
ref, _ := app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3)
|
||||
app.Append(ref, nil, ts+15000, 71.4) // ref fast-path skips label hashing
|
||||
app.Commit()
|
||||
|
||||
// Read
|
||||
q, _ := db.Querier(mint, maxt)
|
||||
ss := q.Select(labels.MustNewMatcher(labels.MatchEqual, "room", "office"))
|
||||
for ss.Next() {
|
||||
it := ss.At().Iterator()
|
||||
for it.Next() {
|
||||
t, v := it.At()
|
||||
_ = t; _ = v
|
||||
}
|
||||
}
|
||||
q.Close()
|
||||
|
||||
db.Close()
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
**Pre-alpha. Not usable yet.** Built bottom-up; the public API above is the design target, not the current state.
|
||||
**Pre-alpha.** The API is frozen (M4) and the system survives a 48h soak test under sustained load (M5), but this hasn't seen production use yet.
|
||||
|
||||
| Milestone | State |
|
||||
|---|---|
|
||||
| M1 — Gorilla chunk encoding, fuzzed, benchmarked | Done |
|
||||
| M2 — Head + WAL, kill -9 safe | In progress |
|
||||
| M3 — Immutable blocks, mmap reads | — |
|
||||
| M4 — Query path, API freeze, shippable alpha | — |
|
||||
| M5 — Compaction + retention, 48h soak | — |
|
||||
| M6 — ingotctl, HTTP layer, Grafana | — |
|
||||
| M2 — Head + WAL, kill -9 safe | Done |
|
||||
| M3 — Immutable blocks, mmap reads | Done |
|
||||
| M4 — Query path, API freeze, shippable alpha | Done |
|
||||
| M5 — Compaction + retention, 48h soak | Done |
|
||||
| M6 — ingotctl, HTTP layer, self-instrumentation | Done |
|
||||
|
||||
See [DESIGN.md](DESIGN.md) for architecture, on-disk format, and the non-goals table (no replication, no PromQL, no deletes, no out-of-order writes — each one deliberate).
|
||||
See [DESIGN.md](DESIGN.md) for architecture, on-disk format, and the non-goals table. See [ROADMAP.md](ROADMAP.md) for what's next.
|
||||
|
||||
## Why
|
||||
## Features
|
||||
|
||||
Go programs that need local metrics storage have two options: run a Prometheus-shaped server next to your process, or hand-roll encoding on top of a key-value store. The embedded middle ground — common in the SQLite world — doesn't exist for time series in Go. Target users:
|
||||
- **Gorilla XOR compression** — ~1 byte/sample on regular metric data (see benchmarks below)
|
||||
- **Crash-safe** — WAL with CRC32C records; kill -9 at any point loses at most uncommitted samples
|
||||
- **Query by label matchers** — equality, negation, regex, negative regex; merged across head and blocks
|
||||
- **Levelled compaction** — 2h → 8h → 32h blocks, background merging, retention-based expiry
|
||||
- **Self-instrumentation** — the DB records its own metrics (series/chunk counts, compactions, WAL fsync duration) through the normal write path, queryable like any other series
|
||||
- **Zero external dependencies** (except testify for tests)
|
||||
|
||||
- Edge/IoT devices buffering sensor data locally
|
||||
- Go binaries recording their own operational history
|
||||
- Homelab sidecars for sensor firehoses (Home Assistant recorder, but purpose-built)
|
||||
- Sampling agents and CLIs currently writing CSV
|
||||
## Tools
|
||||
|
||||
### ingotctl
|
||||
|
||||
CLI for block inspection and diagnostics:
|
||||
|
||||
```sh
|
||||
ingotctl blocks ./data # list blocks with ULID, time range, stats
|
||||
ingotctl inspect ./data/01HXYZ.../ # series labels, chunk metadata, postings stats
|
||||
ingotctl chunks ./data/01HXYZ.../ 42 # decode and print raw samples for series ref 42
|
||||
ingotctl fsck ./data # CRC + index integrity check on all blocks
|
||||
```
|
||||
|
||||
### ingothttp
|
||||
|
||||
Minimal HTTP query server for Grafana integration:
|
||||
|
||||
```sh
|
||||
ingothttp -data ./data -addr :9001
|
||||
```
|
||||
|
||||
Endpoints:
|
||||
- `GET /api/v1/query_range?query=<name>&start=<ms>&end=<ms>` — Prometheus-style JSON matrix response
|
||||
- `POST /api/v1/read` — JSON read request with label matchers
|
||||
- `GET /api/v1/status` — DB stats snapshot
|
||||
|
||||
This is a demo/bridge, not a full PromQL engine. The query parameter matches `__name__` by equality.
|
||||
|
||||
## Compression
|
||||
|
||||
@@ -58,32 +102,48 @@ Two things worth knowing about XOR compression that the headline numbers hide:
|
||||
## Layout
|
||||
|
||||
```
|
||||
ingot/ public API (target: Open, Appender, Querier)
|
||||
ingot/ public API: Open, Appender, Querier
|
||||
├── internal/
|
||||
│ ├── chunkenc/ Gorilla encoder/decoder, bitstream [done]
|
||||
│ ├── wal/ segmented write-ahead log [next]
|
||||
│ ├── chunkenc/ Gorilla encoder/decoder, bitstream
|
||||
│ ├── wal/ segmented write-ahead log
|
||||
│ ├── head/ in-memory series, active chunks
|
||||
│ ├── index/ symbols, postings, matchers
|
||||
│ ├── block/ immutable block read/write
|
||||
│ └── compact/ merge + retention
|
||||
├── cmd/ingotctl/ block inspection, fsck
|
||||
│ ├── block/ immutable block read/write, validation
|
||||
│ ├── compact/ levelled merge + retention
|
||||
│ └── postings/ sorted posting list operations
|
||||
├── cmd/
|
||||
│ ├── ingotctl/ block inspection, fsck
|
||||
│ └── ingothttp/ HTTP query server
|
||||
└── labels/ label types
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
go test -race ./...
|
||||
go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/
|
||||
go test -bench=. ./internal/chunkenc/
|
||||
go test -race -short ./... # all tests (skip soak)
|
||||
go test -race ./... # all tests including soak (~5 min)
|
||||
go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/ # fuzz the decoder
|
||||
go test -bench=. ./internal/chunkenc/ # benchmarks
|
||||
go build ./cmd/ingotctl/ # build CLI tool
|
||||
go build ./cmd/ingothttp/ # build HTTP server
|
||||
```
|
||||
|
||||
The decoder is total: arbitrary bytes produce values or `ErrShortStream`, never a panic. Fuzzing gates every change to `chunkenc`.
|
||||
The decoder is total: arbitrary bytes produce values or `ErrShortStream` and never panics. Fuzzing gates every change to `chunkenc`.
|
||||
|
||||
## Inspiration
|
||||
|
||||
Most of the design is lifted from Prometheus TSDB — chunk encoding, index format, block/compaction model, label data model. The WAL is simpler (no page-level framing). The difference is that Prometheus TSDB is a storage engine inside a server; ingot is a library. See [NOTICE.md](NOTICE.md).
|
||||
|
||||
The chunk encoding comes from the Gorilla paper (Pelkonen et al., VLDB 2015) via Prometheus, which adapted the bit-width buckets for millisecond timestamps. ingot uses the Prometheus variant.
|
||||
|
||||
[tstorage](https://github.com/nakabonne/tstorage) is the closest existing embedded TSDB for Go. It doesn't do Gorilla compression or label-based indexing, which is most of why ingot exists.
|
||||
|
||||
I needed a library to store time-series data locally and the options out there didn't quite fit. Prometheus was too heavy for what I needed but I wanted that level of compression. tstorage was close but it didn't have the compression or label indexing.
|
||||
|
||||
## Non-goals
|
||||
|
||||
Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows. The reasoning for each is in DESIGN.md §3 — they're decisions, not omissions.
|
||||
Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows (sorry not my thing). Check DESIGN.md for reasoning.
|
||||
|
||||
## License
|
||||
|
||||
TBD
|
||||
Apache 2.0
|
||||
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
# Roadmap
|
||||
|
||||
Post-M6 work, roughly in priority order.
|
||||
|
||||
## Protobuf remote-read
|
||||
|
||||
Replace the JSON `/api/v1/read` endpoint with Prometheus-compatible protobuf+snappy encoding so ingothttp works as a native remote-read target for Prometheus and Grafana. Adds protobuf and snappy as dependencies to the cmd binary; the library stays zero-dep.
|
||||
|
||||
## Grafana demo
|
||||
|
||||
Docker-compose setup with ingothttp and Grafana pre-configured. Include a data generator that writes a few series so the dashboard isn't empty on first load. `docker compose up`, open localhost:3000.
|
||||
|
||||
## Query path benchmarks
|
||||
|
||||
`Benchmark*` tests for `Querier`, `Select`, and iteration across varying series/chunk/block counts. Track ns/query and allocations. The write path has benchmarks already; the read path has none.
|
||||
|
||||
## CI
|
||||
|
||||
At minimum:
|
||||
- `go test -race -short ./...` on every push
|
||||
- `go test -race -run TestSoak ./...` with a 10-minute timeout on main/nightly
|
||||
- `go vet ./...`
|
||||
- A linter (staticcheck or golangci-lint)
|
||||
Reference in New Issue
Block a user