Open Commons ObservatoryEnglish·Русский

Research

What was looked at before acting: the question, what was found, and what it rests on.

Should tad embed DuckDB, or call its pinned standalone binary?

Measured on the lab, 2026-11-12: embedding (duckdb crate, bundled) adds 358 s to a release build and makes the binary 66 MB, and it cannot be built as tad's releases are (fully static, musl): the musl tools ship no C++ compiler, and DuckDB calls cross-compiling best effort. DuckDB's standalone binary is 60 MB, published for x86_64 and arm64, linked against the system C library (fine on the lab, CI runners and desktops), and answered a query over the phone project's data in 15 ms; the phone project's feeds already run it, pinned by fingerprint.

2026-11-12

How should tad reach the repositories that use it?

From 2026-09-22 each repository vendored the engine as a git subrepo; by 2026-11-11 three of four carried an older copy (185 and 194 lines against today's 478) - copies drift silently. Since then tad is called, not copied: release-please versions it from its commits, the package conveyor builds binaries with checksums and provenance in the same run, each repository installs its pinned release after checking the checksum, and the lab moves the pins weekly through each repository's own CI (a pull request opened by GitHub's own token would start no CI).

  • git-subrepo - Ingy döt Net, 2013 Vendoring one repository inside another with one commit per sync - tad's way in from 2026-09-22 until it was called, not copied.
  • release-please - Google, 2019 A release pull request kept from conventional commits; merging it tags the release.
  • The package conveyor - linux-package-distribution-oco, 2026 Binaries for every release, for each architecture, with checksums and signed provenance.

2026-11-11

What must tad in Rust be, and how should it declare its tables?

tad today: a 127-line runner (check, canonical form, SQL statements on DuckDB) and a 478-line renderer of Markdown documents, vendored into four repositories (the phone project, the canon, the playbook, Rhizoid). Their schemas hold 41 tables, and every constraint is an id pattern, a list of allowed values, a required field, or one list-of-words column - no foreign keys, no SQL logic. So tables can be declared as data (Frictionless Table Schema, CSV on the Web), checked without a database engine; SQL stays a possible view onto the data (as Datasette is), not its source of truth. Dolt keeps a database in a separate store; tad keeps text in plain git, which is its point. Measured 2026-11-11: Python's check only compares field names with columns; the real validation is canon (DuckDB enforces types and rules on import), and checks/*.sql holds 13 cross-table invariants of four kinds - ids unique across tables, references resolving, everything cited, no mirrored relations - all declarable as data. Measured 2026-11-11: three of the four repositories carry an older vendored copy of the renderer (185 and 194 lines against today's 478) - copies drift silently. On the switch the canon and Rhizoid get today's renderer for the first time; their documents change (rule lines between log entries, reStructuredText if asked).

  • Table Schema (Frictionless Data) - Open Knowledge Foundation, 2016 Tables declared as data: each field's type, required, unique, allowed values, pattern, minimum and maximum; primary and foreign keys. The shape of tad's declarations.
  • CSV on the Web: metadata for tabular data - W3C, 2015 Plain tabular files with a small description of their columns, kept beside them - text first, metadata as data.
  • JSON Lines - Ian Ward, 2013 One record per line: diffs a line per change, readable, streamable. tad's file format already.
  • Dolt - DoltHub, 2019 Weighed and rejected in 2026-09: a SQL database with branches - but its history sits where a pull request cannot show it.
  • Datasette - Simon Willison, 2017 Explore tabular data with SQL when wanted - SQL as a view onto text data, not its source of truth.

2026-11-11

Why tad exists: what does it offer that nothing else did?

Researched and decided 2026-09-21 (text-as-data-template's session log; the DuckDB spike, design-canon PR 1): one dataset that is at once files (JSON lines, one per table, readable and diffable in a pull request), text (jq, Miller, any editor for field surgery), and a real database (SQL with joins, edits round-tripped back to canonical files), so everything stays queryable, versioned and diffable in the same place. Dolt and Doltgres hide history where a pull request cannot show it; gitsheets has no SQL or joins; GlueSQL reinvents an engine without ALTER TABLE; SQLite with sqlite-diffable was weighed and DuckDB chosen - its EXPORT and IMPORT DATABASE write and read exactly this layout, and the spike measured a zero-diff round trip at 200,000 rows. The 2026-11-11 Rust rewrite dropped SQL because this research was not read: the engine README that pointed to it was deleted with the Python engine, and the research lived in another repository's log and an unmerged spike.

  • Spike: DuckDB + Miller + git as an open-file database (SPIKE.md) - grenudi, with Claude, 2026 Measured: a no-op import and export leaves zero diff at 200,000 rows; one updated row is one changed line; rich types round-trip; BLOB does not (use base64); keep insertion order; constraints fail loudly on import; a renamed or unknown key is silent - hence the guard.
  • text-as-data-template: session log, 2026-09-21 - Chose DuckDB + JSONL as the storage format - creation-guidelines, 2026 The decision and what it weighed: Doltgres, Dolt, gitsheets, GlueSQL, SQLite with sqlite-diffable - so everything stays queryable, versioned and diffable in one place.
  • DuckDB: EXPORT DATABASE and IMPORT DATABASE - DuckDB, 2019 A whole database written out as plain files and read back: the layout tad keeps.
  • Dolt - DoltHub, 2019 Weighed and rejected in 2026-09: a SQL database with branches - but its history sits where a pull request cannot show it.
  • gitsheets - Jarvus Innovations, 2019 Weighed and rejected in 2026-09: data as files in git - but no SQL and no joins.
  • sqlite-diffable - Simon Willison, 2020 Weighed in 2026-09: dumping SQLite to diffable files - DuckDB was chosen over it.

2026-11-11

Other sources