david/ingot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md and ROADMAP.md

Browse Source
This commit is contained in:
david
2026-07-04 18:16:23 -04:00
parent 30a93a868e
commit 199e555e3d
2 changed files with 111 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+88 -28
View File
@@ -1,41 +1,85 @@
# ingot
An embedded time-series database for Go. SQLite for metrics: a library you import, not a server you deploy.
An embedded time-series database for Go. SQLite for metrics.
```go
db, _ := ingot.Open("./data", ingot.Options{Retention: 30 * 24 * time.Hour})
db, _ := ingot.Open("./data", ingot.Options{
Retention: 30 * 24 * time.Hour,
BlockDuration: 2 * time.Hour,
})
// Write
app := db.Appender()
app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3)
ref, _ := app.Append(0, labels.FromStrings("__name__", "temp", "room", "office"), ts, 71.3)
app.Append(ref, nil, ts+15000, 71.4) // ref fast-path skips label hashing
app.Commit()
// Read
q, _ := db.Querier(mint, maxt)
ss := q.Select(labels.MustNewMatcher(labels.MatchEqual, "room", "office"))
for ss.Next() {
it := ss.At().Iterator()
for it.Next() {
t, v := it.At()
_ = t; _ = v
}
}
q.Close()
db.Close()
```
## Status
**Pre-alpha. Not usable yet.** Built bottom-up; the public API above is the design target, not the current state.
**Pre-alpha.** The API is frozen (M4) and the system survives a 48h soak test under sustained load (M5), but this hasn't seen production use yet.
| Milestone | State |
|---|---|
| M1 — Gorilla chunk encoding, fuzzed, benchmarked | Done |
| M2 — Head + WAL, kill -9 safe | In progress |
| M3 — Immutable blocks, mmap reads | — |
| M4 — Query path, API freeze, shippable alpha | — |
| M5 — Compaction + retention, 48h soak | — |
| M6 — ingotctl, HTTP layer, Grafana | — |
| M2 — Head + WAL, kill -9 safe | Done |
| M3 — Immutable blocks, mmap reads | Done |
| M4 — Query path, API freeze, shippable alpha | Done |
| M5 — Compaction + retention, 48h soak | Done |
| M6 — ingotctl, HTTP layer, self-instrumentation | Done |
See [DESIGN.md](DESIGN.md) for architecture, on-disk format, and the non-goals table (no replication, no PromQL, no deletes, no out-of-order writes — each one deliberate).
See [DESIGN.md](DESIGN.md) for architecture, on-disk format, and the non-goals table. See [ROADMAP.md](ROADMAP.md) for what's next.
## Why
## Features
Go programs that need local metrics storage have two options: run a Prometheus-shaped server next to your process, or hand-roll encoding on top of a key-value store. The embedded middle ground — common in the SQLite world — doesn't exist for time series in Go. Target users:
- **Gorilla XOR compression** — ~1 byte/sample on regular metric data (see benchmarks below)
- **Crash-safe** — WAL with CRC32C records; kill -9 at any point loses at most uncommitted samples
- **Query by label matchers** — equality, negation, regex, negative regex; merged across head and blocks
- **Levelled compaction** — 2h → 8h → 32h blocks, background merging, retention-based expiry
- **Self-instrumentation** — the DB records its own metrics (series/chunk counts, compactions, WAL fsync duration) through the normal write path, queryable like any other series
- **Zero external dependencies** (except testify for tests)
- Edge/IoT devices buffering sensor data locally
- Go binaries recording their own operational history
- Homelab sidecars for sensor firehoses (Home Assistant recorder, but purpose-built)
- Sampling agents and CLIs currently writing CSV
## Tools
### ingotctl
CLI for block inspection and diagnostics:
```sh
ingotctl blocks ./data # list blocks with ULID, time range, stats
ingotctl inspect ./data/01HXYZ.../ # series labels, chunk metadata, postings stats
ingotctl chunks ./data/01HXYZ.../ 42 # decode and print raw samples for series ref 42
ingotctl fsck ./data # CRC + index integrity check on all blocks
```
### ingothttp
Minimal HTTP query server for Grafana integration:
```sh
ingothttp -data ./data -addr :9001
```
Endpoints:
- `GET /api/v1/query_range?query=<name>&start=<ms>&end=<ms>` — Prometheus-style JSON matrix response
- `POST /api/v1/read` — JSON read request with label matchers
- `GET /api/v1/status` — DB stats snapshot
This is a demo/bridge, not a full PromQL engine. The query parameter matches `__name__` by equality.
## Compression
@@ -58,32 +102,48 @@ Two things worth knowing about XOR compression that the headline numbers hide:
## Layout
```
ingot/ public API (target: Open, Appender, Querier)
ingot/ public API: Open, Appender, Querier
├── internal/
│ ├── chunkenc/ Gorilla encoder/decoder, bitstream [done]
│ ├── wal/ segmented write-ahead log [next]
│ ├── chunkenc/ Gorilla encoder/decoder, bitstream
│ ├── wal/ segmented write-ahead log
│ ├── head/ in-memory series, active chunks
│ ├── index/ symbols, postings, matchers
│ ├── block/ immutable block read/write
│ └── compact/ merge + retention
├── cmd/ingotctl/ block inspection, fsck
│ ├── block/ immutable block read/write, validation
│ ├── compact/ levelled merge + retention
│ └── postings/ sorted posting list operations
├── cmd/
│ ├── ingotctl/ block inspection, fsck
│ └── ingothttp/ HTTP query server
└── labels/ label types
```
## Development
```sh
go test -race ./...
go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/
go test -bench=. ./internal/chunkenc/
go test -race -short ./... # all tests (skip soak)
go test -race ./... # all tests including soak (~5 min)
go test -fuzz=FuzzXORIterator -fuzztime=60s ./internal/chunkenc/ # fuzz the decoder
go test -bench=. ./internal/chunkenc/ # benchmarks
go build ./cmd/ingotctl/ # build CLI tool
go build ./cmd/ingothttp/ # build HTTP server
```
The decoder is total: arbitrary bytes produce values or `ErrShortStream`, never a panic. Fuzzing gates every change to `chunkenc`.
The decoder is total: arbitrary bytes produce values or `ErrShortStream` and never panics. Fuzzing gates every change to `chunkenc`.
## Inspiration
Most of the design is lifted from Prometheus TSDB — chunk encoding, index format, block/compaction model, label data model. The WAL is simpler (no page-level framing). The difference is that Prometheus TSDB is a storage engine inside a server; ingot is a library. See [NOTICE.md](NOTICE.md).
The chunk encoding comes from the Gorilla paper (Pelkonen et al., VLDB 2015) via Prometheus, which adapted the bit-width buckets for millisecond timestamps. ingot uses the Prometheus variant.
[tstorage](https://github.com/nakabonne/tstorage) is the closest existing embedded TSDB for Go. It doesn't do Gorilla compression or label-based indexing, which is most of why ingot exists.
I needed a library to store time-series data locally and the options out there didn't quite fit. Prometheus was too heavy for what I needed but I wanted that level of compression. tstorage was close but it didn't have the compression or label indexing.
## Non-goals
Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows. The reasoning for each is in DESIGN.md §3 — they're decisions, not omissions.
Replication, query languages, non-float64 values, deletes, multi-process access, out-of-order ingestion, Windows (sorry not my thing). Check DESIGN.md for reasoning.
## License
TBD
Apache 2.0