Research
What was looked at before acting: the question, what was found, and what it rests on.
What does everything we run cost from GitHub's hourly budget?
Measured 2026-11-12: 759 of 5,000 used in an hour of heavy work. The budget is one for the whole account: lab-agent's daemon, the client's submits and waits, land waiting on CI, the pin updater, the probes and oco-agent all draw on it. lab-agent's daemon, wait and board already ask conditionally (ETag), so idle polling every 5 s is free; oco-agent's first GitHub adapter did not, and would have spent about 30 a minute listening (about 480 an hour at its default pace). With conditional listings its cost while nothing changes is nothing but its first two answers: a minute of listening every 4 s counted 2, against 5 for a baseline minute in which only the measuring job's own log streaming ran - oco-agent's share is lost in that noise. A folder that does not exist answers 404 every time, charged and with no fingerprint, so the asked-work folder keeps a README and a missing listing is not asked again for two minutes. Content creation (commits through the API, pull requests) has its own lower limits, worth minding in rollouts.
- Rate limits for the REST API - GitHub, 2025 5,000 requests an hour for an authenticated user, shared by every token of that user; a conditional request answered 304 Not Modified does not count; content creation has its own, lower limits (per minute and per hour).
2026-11-12
Why did the watcher's schedule not run?
It ran once by schedule, at 16:54 UTC on 2026-11-12 - nearly nine hours after it was put in place - successfully, and not again in the two hours after. Nothing of ours: GitHub documents schedules as best-effort, delayed under load and sometimes dropped without failing, and community reports describe delays of 8 to 14 hours for new repositories. A watcher on that clock is silent exactly when it should speak. The pulse, meanwhile, reached Healthchecks every five minutes all day: outside the lab, depending on nothing the lab runs.
- Events that trigger workflows: schedule - GitHub, 2026 Scheduled runs are best-effort: delayed under load (the start of each hour most of all), and when load is high enough some are dropped - a dropped run does not fail, it does not happen. Community reports put delays on new repositories at hours.
2026-11-12
What let the 2026-11-11 incident happen, and how do proven systems prevent each part of it?
Measured: after the reboot the agent replayed the afternoon's jobs from the oldest (102 started, 40 ok, 62 failed - mostly guards refusing), fifty at a time after a reconnect, on shared clones; it had inferred 'pending' from a missing result. Before that it stalled at 22:29 CET with nothing logged until the reboot. No alert could reach the author - every channel went through the agent. Harm: none - four replayed pull requests, all the legitimate tad 0.4.0 pin bump; no direct pushes, tags, releases or phone actions; tools and clones restored. The root: a job is a script with no record of its life. Proven answers: durable execution's event history and heartbeats (Temporal), leases with fencing tokens (Kleppmann; Patterns of Distributed Systems), a dead man's switch watched from elsewhere (Healthchecks), crash-only recovery (Candea and Fox), run-once jobs with deadlines (Kubernetes), idempotency keys (Stripe), a working tree per job (git worktree). Replayed 2026-11-12: the old agent's real log of 2026-11-11, read line by line through oco-agent's rules (oco-agent/replay): of the 357 jobs the old agent started over six runs, oco-agent would have started 255 and refused exactly 102, each because it had already finished - the first at 23:16:24, the moment the replay began. The journal it kept (764 rows of tad data) checks out and answers SQL. 2026-11-12, from the whole system's log: the machine did not sleep and its network stayed up through the stall; a USB device was disconnected at 22:25:28 with nothing after it going wrong, so the phone looks innocent. The stall was inside lab-agent's daemon. Its GitHub client is built with no limit on how long a request may take (src/github.rs: ureq::AgentBuilder::new().build()), so one connection that hangs holds the polling loop for ever - which fits every fact, though nobody watched it hang. oco-agent's client limits every request to 20 s. The desktop is set to suspend after an hour without input, and lab-agent keeps it awake only while a job runs. 2026-11-12, proven: the replays come from lab-agent's reading of what is done. already_handled lists its results folder through GitHub's contents API, which returns at most 1,000 entries and says nothing of the rest: the folder held 1,135; the first replayed job of the second replay (1794436981-after-reconnect) had its result, but was not among the 1,000 listed. Every job past the thousandth looked unfinished. Both replays - 102 jobs on 2026-11-11 and 25 after the boot on 2026-11-12 - share this one cause. The 2026-11-12 replay changed no repository (no commits since the boot) but put tad 0.4.0 back on the lab (restored to 0.5.0). 2026-11-12, later - the stall is proven too, and it has the same cause as the replays. lab-agent sees new jobs through the same contents listing of its jobs folder, cut silently at 1,000 entries. The folder holds a placeholder, .keep, which sorts before every job; when lab-agent started its last job before the stall (22:29:23), the folder held 999 jobs - with .keep, exactly 1,000. Every job submitted after that lay past the end of the listing: the agent was alive, polling, and blind. The earlier note here that a request with no time limit was the probable cause was wrong as the main cause; time limits are added anyway, since a hang should never be silent either. One cause - a listing that stops without a word - made the stall, the replay of 102 jobs, and the replay of 25 after the boot.
- Temporal: durable execution - Temporal Technologies, 2020 Every step recorded in an append-only event history; after a crash, resume from the last recorded step without repeating finished ones.
- Long-running activity: progress and cancellation with heartbeats - Temporal Technologies, 2024 A long step heartbeats its progress; a lost worker is noticed within the heartbeat timeout, and the retry resumes from the last checkpoint.
- How to do distributed locking - Martin Kleppmann, 2016 A lease can expire while its holder is paused; a fencing token that only grows lets the resource refuse the stale holder.
- Patterns of Distributed Systems: Lease - Unmesh Joshi, 2020 A lease is a lock with a time limit: acquire, renew, release, expire.
- Healthchecks: a dead man's switch - Healthchecks.io, 2015 Alert on absence: silent while pings arrive, alerting when one is late; the watcher lives elsewhere.
- Crash-Only Software - George Candea, Armando Fox, 2003 A system that can be killed at any moment and recovers by reading its own record, the same way every time.
- Kubernetes: Jobs - The Kubernetes project, 2015 A finished job is never started again; deadlines and time-to-live end what waits too long.
- Designing robust and predictable APIs with idempotency - Brandur Leach (Stripe), 2017 An action that changes the world carries a key, so doing it twice finds it done.
- git worktree - Git project, 2015 Several working trees from one repository: each job its own, none shared.
2026-11-11
What should the redesigned lab-agent be called?
The author proposed oco-agent (2026-11-12): it is built on ocoeco by design. Prior art names this role - a long-running program on a machine that takes jobs and runs them - an agent (Jenkins, Buildkite, TeamCity) or a runner (GitHub, GitLab). 'agent' also means an AI agent; the design already separates them: the AI driving the lab is the driver, the program is the machine's hands and voice. 'lab' said where it runs, not what it belongs to; 'oco-' says its family and the direction of dependency (built on ocoeco, which knows nothing of it).
- Using Jenkins agents - The Jenkins project, 2011 An agent: a machine that connects to the controller and runs the jobs it is given.
- The Buildkite agent - Buildkite, 2014 A small program on your own machine that polls for jobs and runs them.
- TeamCity build agent - JetBrains, 2006 A build agent: the piece that runs builds, on any machine.
- Self-hosted runners - GitHub, 2019 The same role, called a runner.
2026-11-11
How should waiting be counted, and how can work go on while a long job runs?
Waiting is one of the seven wastes of the Toyota Production System; Kanban measures flow efficiency - time worked over total time - so manual work (effort) and waiting (idle time) are two halves of one measure. Little's law: shorter waits or more work in flight raise throughput. Make and CI run independent jobs side by side and serialise only those sharing a resource. So: lab-agent records every wait itself, with what is waited on and for which project; and jobs declare the resources they hold, so those that do not conflict run in parallel.
- Toyota Production System: the seven wastes - Taiichi Ohno, 1988 Waiting is one of the seven wastes: time in which nothing is made.
- A Proof for the Queuing Formula: L = λW - John D. C. Little, 1961 Work in flight equals throughput times time in the system: shorter waits, or more work in flight, raise throughput.
- Kanban: flow efficiency - David J. Anderson, 2010 Time worked divided by total time, waits included - the share of a job's life spent on it.
- GNU make: parallel execution (-j) - GNU Project, 1988 Run what does not depend on each other side by side; declare what depends on what.
- GitHub Actions: concurrency groups - GitHub, 2021 Jobs that share a resource take turns; the rest run freely.
2026-11-11
Other sources
- Monitoring Distributed Systems (Site Reliability Engineering, chapter 6) - Rob Ewaschuk, Google, 2016 Black-box monitoring tests behaviour as a user sees it - it catches what is broken now, symptoms rather than causes.
- Hexagonal architecture (ports and adapters), in the software-engineering canon - creation-guidelines, 2026 The core talks to the world only through ports it defines; adapters plug technologies in; in-memory adapters make tests trivial. Paired with functional core, imperative shell, and testing at the seams.
- REST API endpoints for repository contents - GitHub, 2025 A file is written with one request that makes one commit; writing over an existing file without its sha is refused.
- REST API endpoints for GitHub Actions variables - GitHub, 2025 Repository variables are read and written without a commit.
- Healthchecks.io: cron job monitoring - Pēteris Caune, 2015 A dead man's switch: a check expects a ping every period; when one is late beyond a grace time it alerts through integrations, ntfy among them.
- Jobs - Kubernetes, 2024 Work run to completion once, with a deadline after which it is no longer started, and everything it started ended with it.
- The ideal-project playbook: names explain the program; guards - creation-guidelines, 2026 Areas named for what the program is for, not for patterns; tool words only in the adapter that drives them; purity by role, vocabulary and findability checked on every change.
- Most widely deployed SQL database engine - SQLite, 2024 Embedded, everywhere, and stable for decades: a store and query engine that needs no server.