CodeAFRun a nightly test routine

Run a nightly test routine

Every night the suite runs, failures become issues, easy bugs get fixed, and a report waits for you in the morning.

The best time to find a failing test is while you sleep.
  • every night at 2am
  • before the Monday standup
  • after every release tag

This playbook sets up a routine you would otherwise run by hand each morning. Every night, CodeAF runs the tests several times to catch the ones that fail only some of the time, writes an issue file for every test that failed, fixes the obvious bugs, leaves the flaky ones for you, and writes a short morning report. You set it up with one sentence and one key.

Steps

  1. In the repository, run codeaf.
  2. Say what you want, when, in plain words, and press Enter:
Every night at 2am: run go test -count=5 ./... here. For every test that failed at least once, write one issue file issues/<TestName>.md (what failed, how often, the error lines, a guessed cause). Then fix the easy ones (a real bug with an obvious fix; leave a flaky or racy test as an issue only), make sure go test passes for what you fixed, and write a short morning report to MORNING.md: what failed, what you fixed, what is left for me.
  1. A card comes back with the routine's name, when it runs and what it may spend: wants to set up a repeating check · nightly go test sweep · every night at …, shares the day's $500.00 allowance, where · for this project. Press 1 (Set it up).
  2. In the morning, open codeaf. Home lists the firing under since you left: fired just now · it was the time you asked for.

What you see

CodeAF turns your sentence into a standing order: a nightly schedule, the full brief with your rule never to paper over a flaky test, and limits of $5 per firing and 10 firings a day. Standing orders are checked every 5 minutes, so a firing can start up to 5 minutes after the time you asked for.

After our firing, issues/ held three files: two tests that failed 5 times out of 5 and one that failed 2 out of 5. Both real bugs were fixed and the tests passed. The racy test stayed an issue, ending with a decision for us: make the ledger safe for concurrent use, or document that it is not. MORNING.md had three sections: what failed, what was fixed, and what is left for you.

The fixes are left as uncommitted edits in the working folder, so the morning review is a plain git diff.

In our run

Setting it up took 38 seconds and $0.02. The firing itself took 31 seconds and cost $0.006.

Make it yours

  • Change the hour or the cadence. "Every weekday at 6am" or "every Monday at 9" work the same way; the card shows the cadence it understood before anything is set up.
  • Stop it. Say "stop the nightly sweep" in any chat of the project, and the routine shows retired at once.
  • Work on another machine. Run the routine on a dev box that stays on, and read the report from anywhere. See Remote machines.

Go further

Standing orders: Say once what should stay true, and CodeAF keeps it without being asked again.

Coming soon: describe the org you want and CodeAF builds it: teams, managers, budgets, standing orders.